You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the newly added seek semantics in the kafka consume step, there are many things that are not ideal. I think we should address those.
Upon retry on a consume step with seek, it reseeks to the timestamp and consumes duplicate records. Ideally it should only seek when the current offset is less than the offset to seek to. There is really no real reason I can see for a consumer to seek to an already consumed records
Insufficient CI tests to cover edge scenarios around seek.
Currently, the consumer only assigns to the partitions that have a valid offset > the seek offset for timestamp. Ideally, it should assign to all the partitions, seeking to end for partitions that don't have messages yet with offset > seekOffset
The text was updated successfully, but these errors were encountered:
a1shadows
changed the title
Improvement iof seek functionality in kafka consumer
Improvement of seek functionality in kafka consumer
Mar 14, 2024
For the newly added seek semantics in the kafka consume step, there are many things that are not ideal. I think we should address those.
The text was updated successfully, but these errors were encountered: