Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add round robin partitioner #1088

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Arseniy-Popov
Copy link

@Arseniy-Popov Arseniy-Popov commented Jan 27, 2025

Changes

This adds a round robin partitioner. When message ordering is not a concern, this can increase consumption throughput by ensuring a more unform temporal spacing between messages in a single partition.

As for production throughput, the modified benchmark shows that it is increased by switching to the RoundRobinPartitioner (partition count was manually set to 4):

$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 141488 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Produced 139712 messages in 1 second(s).
Produced 138528 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 140092.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "round-robin" --partition -1
Produced 140896 messages in 1 second(s).
Produced 136160 messages in 1 second(s).
Produced 139120 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Produced 141488 messages in 1 second(s).
Produced 142672 messages in 1 second(s).
Produced 140304 messages in 1 second(s).
Total produced 1000000 messages in 7.14 second(s). Avg 139996.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 96425 messages in 1 second(s).
Produced 96715 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96570 messages in 1 second(s).
Produced 96860 messages in 1 second(s).
Produced 96280 messages in 1 second(s).
Produced 95700 messages in 1 second(s).
Produced 96135 messages in 1 second(s).
Produced 95555 messages in 1 second(s).
Produced 92075 messages in 1 second(s).
Total produced 1000000 messages in 10.43 second(s). Avg 95880.0 m/s
$ python benchmark/simple_produce_bench.py -n 1000000 --partitioner "default" --partition -1 --key
Produced 93235 messages in 1 second(s).
Produced 88450 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 92365 messages in 1 second(s).
Produced 92655 messages in 1 second(s).
Produced 88015 messages in 1 second(s).
Produced 93090 messages in 1 second(s).
Produced 91060 messages in 1 second(s).
Produced 92945 messages in 1 second(s).
Produced 88885 messages in 1 second(s).
Produced 90415 messages in 1 second(s).
Total produced 1000000 messages in 11.00 second(s). Avg 90889.0 m/s

Checklist

  • I think the code is well written
  • Unit tests for the changes exist
  • Documentation reflects the changes
  • Add a new news fragment into the CHANGES folder
    • name it <issue_id>.<type> (e.g. 588.bugfix)
    • if you don't have an issue_id change it to the pr id after creating the PR
    • ensure type is one of the following:
      • .feature: Signifying a new feature.
      • .bugfix: Signifying a bug fix.
      • .doc: Signifying a documentation improvement.
      • .removal: Signifying a deprecation or removal of public API.
      • .misc: A ticket has been closed, but it is not of interest to users.
    • Make sure to use full sentences with correct case and punctuation, for example: Fix issue with non-ascii contents in doctest text files.

@Arseniy-Popov Arseniy-Popov force-pushed the feat/add-round-robin-partitioner branch from 36bdcfb to 750ed4b Compare January 28, 2025 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant