Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use amqp for broker/worker communication #29

Closed
wants to merge 2 commits into from
Closed

Use amqp for broker/worker communication #29

wants to merge 2 commits into from

Conversation

mweinelt
Copy link
Contributor

@mweinelt mweinelt commented Nov 16, 2020

Creates the WorkerRegistry object to
- track alive workers and their metrics
- submit client requests to workers

Creates the Worker object to
- hold a workers metric
- wrap queue interaction for the worker

Creates the Broker object to
- connect the HTTP API call to WorkerRegistry functions

Library wise these changes are included:
- Adds pika as the amqp connection library and bottle to replace the
  comparatively heavy flask and Werkzeug libraries.
  Bottle can be used with less global state and inside a class, which
  allows to track state within class members.

- Introduces structlog, for log messages that can also expose key/value
  pairs for structered logging.

@mweinelt mweinelt force-pushed the amqp branch 2 times, most recently from c17e739 to 09cee4e Compare November 16, 2020 15:30
@mweinelt
Copy link
Contributor Author

Why AMQP?

  • We can track delivery and acknowledgement of submitted jobs
  • We can requeue them should they fail
  • We can select which worker receives which jobs
  • We can create a high-available setup with multiple wgkex-broker instances and multiple, clustered rabbitmq daemons (https://www.rabbitmq.com/ha.html)
    • gateways push their registration and metrics into the brokers queue, and the message gets replicated for all brokers

@awlx
Copy link
Member

awlx commented Nov 17, 2020

Merged the netlink branch. So we can rebase on that :). Maybe we can also reuse some of the ideas from mqtt and move on to amqp.

Creates the WorkerRegistry object to
- track alive workers and their metrics
- submit client requests to workers

Creates the Worker object to
- hold a workers metric
- wrap queue interaction for the worker

Creates the Broker object to
- connect the HTTP API call to WorkerRegistry functions

Library wise these changes are included:
- Adds pika as the amqp connection library and bottle to replace the
  comparatively heavy flask and Werkzeug libraries.
  Bottle can be used with less global state and inside a class, which
  allows to track state within class members.

- Introduces structlog, for log messages that can also expose key/value
  pairs for structered logging.
@awlx
Copy link
Member

awlx commented Feb 12, 2023

Replaced by #87

@awlx awlx closed this Feb 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants