queueing

This article is a stub. You can help the IndieWeb wiki by expanding it.

A queue is a method used to distribute work across machines or threads by defining a FIFO queue of work to be done.

Queue Systems

• Amazon Simple Queue Service
• Celery (in the Python world)
  • easy to use, but quite large.
  • can use Redis or RabbitMQ as the backend
• beanstalkd – lightweight protocol based on memcached [1]
• gearman – historically been fairly commonly used in the PHP community
• Google App Engine Task Queue
• PostgreSQL – supports pub-sub broadcasting through LISTEN/NOTIFY
• RabbitMQ
  • Highly scalable message queue system written in Erlang
  • Implements the AMQP standard.
  • Supports broadcasting as well as 1-to-1 messaging etc.
• Redis – a general key-value store that can store various data structures, including a FIFO queue.
• uWSGI spooler
  • part of uWSGI
  • Simple file-based spool, similar in concept to mail spools
  • Processes 1 task every n-seconds, which means there is an unnecessary lag between queueing and processing. Probably more suitable for large background tasks.
  • lacking documentation

IndieWeb Examples

IndieWeb users using this approach

• Aaron Parecki likes beanstalk, uses it in p3k
• Kyle Mahan uses Redis with this Python/Flask snippet, lightly modified.
  • uwsgi spooler previously, polling every 30 seconds. It was simple and solid, but I didn't like that mentions weren't sent out right away, nor incoming mentions processed right away. Also no obvious way to query queued tasks for their status.
• Pelle Wessman uses PostgreSQL Pubsub in webmention.herokuapp.com as described in this blog post.
• bear uses Redis PubSub to deliver events and a list to implement a FIFO queue.

Alternatives

• Make an entry somewhere for the task and then have a cron do it. You could also use a web bug.

Queues and Brokers

Many message queues are also message brokers as they have to translate the incoming messages into other formats or protocols to allow the clients/consumers to receive them. RabbitMQ is a very common example of a Message Broker that is often called a Queueing System. At it's core RabbitMQ implements AMQP and then has quite a few delivery and filtering methods that enable consumers to receive messages.

Another example of a broker that has queue like behaviour is Google PubSub. This Wikipedia page on brokers has quite a good definition of the differences.

See also

• Celery Task Queue definition
• 2024-07-30 Anthony Accomazzo: Build your own SQS or Kafka with Postgres (archived) walks through how to build your own Simple Queue Service with standard SQL tables in Postgres