queueing
This article is a stub. You can help the IndieWeb wiki by expanding it.
A queue is a method used to distribute work across machines or threads by defining a FIFO queue of work to be done.
Queue Systems
- Amazon Simple Queue Service
- Celery (in the Python world)
- easy to use, but quite large.
- can use Redis or RabbitMQ as the backend
- beanstalkd β lightweight protocol based on memcached [1]
- gearman β historically been fairly commonly used in the PHP community
- Google App Engine Task Queue
- PostgreSQL β supports pub-sub broadcasting through LISTEN/NOTIFY
- RabbitMQ
- Highly scalable message queue system written in Erlang
- Implements the AMQP standard.
- Supports broadcasting as well as 1-to-1 messaging etc.
- Redis β a general key-value store that can store various data structures, including a FIFO queue.
- uWSGI spooler
- part of uWSGI
- Simple file-based spool, similar in concept to mail spools
- Processes 1 task every n-seconds, which means there is an unnecessary lag between queueing and processing. Probably more suitable for large background tasks.
- lacking documentation
IndieWeb Examples
IndieWeb users using this approach
- Aaron Parecki likes beanstalk, uses it in p3k
- Kyle Mahan uses Redis with this Python/Flask snippet, lightly modified.
- uwsgi spooler previously, polling every 30 seconds. It was simple and solid, but I didn't like that mentions weren't sent out right away, nor incoming mentions processed right away. Also no obvious way to query queued tasks for their status.
- Pelle Wessman uses PostgreSQL Pubsub in webmention.herokuapp.com as described in this blog post.
- bear uses Redis PubSub to deliver events and a list to implement a FIFO queue.
Alternatives
Queues and Brokers
Many message queues are also message brokers as they have to translate the incoming messages into other formats or protocols to allow the clients/consumers to receive them. RabbitMQ is a very common example of a Message Broker that is often called a Queueing System. At it's core RabbitMQ implements AMQP and then has quite a few delivery and filtering methods that enable consumers to receive messages.
Another example of a broker that has queue like behaviour is Google PubSub. This Wikipedia page on brokers has quite a good definition of the differences.
See also
- Celery Task Queue definition
- 2024-07-30 : Build your own SQS or Kafka with Postgres (archived) walks through how to build your own Simple Queue Service with standard SQL tables in Postgres