The broker is the mediator between task producers, such as the manager, and workers, the task consumers. To initiate a task, a producer sends a message to the broker's queue, which the broker then delivers to a worker.
We use RabbitMQ because it allows for multi-tenancy via virtual hosts. This allows accounts to provide their own workers by connecting to their own vhost on the broker.
The stencila/hub-broker Docker image is currently just a simple FROM rabbitmq:3.8-management image. We use the management image, so that we can access the Management HTTP API to add new users and vhosts for accounts. This is available at :15672/api.
The image also includes the rabbitmq_prometheus plugin to enable monitoring metrics. These are available at :15692/metrics.
Instead of self hosting stencila/hub-broker a RabbitMQ-as-a-service service, such as https://www.cloudamqp.com/ could be used.
Celery supports several brokers. Initially we used Redis because of it's ease of setup.