Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement of seek functionality in kafka consumer #645

Closed
a1shadows opened this issue Mar 14, 2024 · 1 comment · Fixed by #646
Closed

Improvement of seek functionality in kafka consumer #645

a1shadows opened this issue Mar 14, 2024 · 1 comment · Fixed by #646
Assignees

Comments

@a1shadows
Copy link
Collaborator

a1shadows commented Mar 14, 2024

For the newly added seek semantics in the kafka consume step, there are many things that are not ideal. I think we should address those.

  1. Upon retry on a consume step with seek, it reseeks to the timestamp and consumes duplicate records. Ideally it should only seek when the current offset is less than the offset to seek to. There is really no real reason I can see for a consumer to seek to an already consumed records
  2. Insufficient CI tests to cover edge scenarios around seek.
  3. Currently, the consumer only assigns to the partitions that have a valid offset > the seek offset for timestamp. Ideally, it should assign to all the partitions, seeking to end for partitions that don't have messages yet with offset > seekOffset
@a1shadows a1shadows changed the title Improvement iof seek functionality in kafka consumer Improvement of seek functionality in kafka consumer Mar 14, 2024
@a1shadows a1shadows self-assigned this Mar 14, 2024
@a1shadows
Copy link
Collaborator Author

@authorjapps Let me know if you can think of any other cases I should cover

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants