queueing

From IndieWeb
(Redirected from task-queue)


A queue is a method used to distribute work across machines or threads by defining a FIFO queue of work to be done.

Queue Systems

  • Amazon Simple Queue Service
  • Celery (in the Python world)
    • easy to use, but quite large.
    • can use Redis or RabbitMQ as the backend
  • beanstalkd – lightweight protocol based on memcached [1]
  • gearman – historically been fairly commonly used in the PHP community
  • Google App Engine Task Queue
  • PostgreSQL – supports pub-sub broadcasting through LISTEN/NOTIFY
  • RabbitMQ
    • Highly scalable message queue system written in Erlang
    • Implements the AMQP standard.
    • Supports broadcasting as well as 1-to-1 messaging etc.
  • Redis – a general key-value store that can store various data structures, including a FIFO queue.
  • uWSGI spooler
    • part of uWSGI
    • Simple file-based spool, similar in concept to mail spools
    • Processes 1 task every n-seconds, which means there is an unnecessary lag between queueing and processing. Probably more suitable for large background tasks.
    • lacking documentation

IndieWeb Examples

IndieWeb users using this approach

Alternatives

  • Make an entry somewhere for the task and then have a cron do it. You could also use a web bug.

Queues and Brokers

Many message queues are also message brokers as they have to translate the incoming messages into other formats or protocols to allow the clients/consumers to receive them. RabbitMQ is a very common example of a Message Broker that is often called a Queueing System. At it's core RabbitMQ implements AMQP and then has quite a few delivery and filtering methods that enable consumers to receive messages.

Another example of a broker that has queue like behaviour is Google PubSub. This Wikipedia page on brokers has quite a good definition of the differences.

See also