Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for seekTo feature #630

Merged
merged 7 commits into from
Mar 2, 2024
Merged

Fix for seekTo feature #630

merged 7 commits into from
Mar 2, 2024

Conversation

a1shadows
Copy link
Collaborator

@a1shadows a1shadows commented Feb 26, 2024

Fixes #631

PR Branch
#ADD LINK TO THE PR BRANCH

Motivation and Context

If there are partitions for which there is no valid offset corresponding to the timestamp to seek to, the offsetForTimestamp method still includes it in the Map returned, but with a NULL value. We need to skip all partitions that have offset as NULL when subscribing to avoid an error when we try to seek to the offset at a later step.

Checklist:

  • Unit tests added

  • Integration tests added

  • Test names are meaningful

  • Feature manually tested

  • Branch build passed

  • No 'package.*' in the imports

  • Relevant Wiki page updated with clear instruction for the end user

    • Not applicable. This was only a refactor change, no functional or behaviour changes were introduced
  • Http test added to http-testing module(if applicable) ?

    • Not applicable. The changes did not affect HTTP automation flow
  • Kafka test added to kafka-testing module(if applicable) ?

    • Not applicable. The changes did not affect Kafka automation flow

@a1shadows a1shadows marked this pull request as ready for review February 26, 2024 01:21
@a1shadows
Copy link
Collaborator Author

@authorjapps There was a bug in the seekTo feature . In situations where there are multiple partitions on the topic and some of the partitions do not have a valid offset corresponding the timestamp to seek to, the seek operation will give an NPE.

Can you review the fix?

@authorjapps
Copy link
Owner

@authorjapps There was a bug in the seekTo feature . In situations where there are multiple partitions on the topic and some of the partitions do not have a valid offset corresponding the timestamp to seek to, the seek operation will give an NPE.

Can you review the fix?

Got you @a1shadows . That scenario causing NPE.
Is there anyway to reproduce this in a single node Kafka in our docker setup?

e.g.
by creating a topic with two or three partitions, then sending records to 1 and 3 based on their key, then not sending any record to 2.
Then try to consume with seekTimestamp, it should fail with NPE

Then with the fix, it should pass as expected(without NPE)

@a1shadows
Copy link
Collaborator Author

e.g.
by creating a topic with two or three partitions, then sending records to 1 and 3 based on their key, then not sending any record to 2.
Then try to consume with seekTimestamp, it should fail with NPE

Then with the fix, it should pass as expected(without NPE)

@authorjapps I'll look into this

…rtition topics, changed test to have sorted messages so that assertion does not fail
@a1shadows
Copy link
Collaborator Author

@authorjapps I've added a multi partition topic test to the CI. For this, I added an init container to the docker compose file, since by default the cp-kafka docker image does not accept num.partitions as an env variable for auto created topics. I ran it locally and it ran fine. However, for the CI the docker compose files are picked from the master branch so can't get a proper test on here

Do you have anything in mind for this? Can we run the CI using the docker compose files on the branch we are trying to merge?

@a1shadows a1shadows merged commit 5f3f664 into authorjapps:master Mar 2, 2024
1 check passed
@a1shadows a1shadows deleted the seek-fix branch March 25, 2024 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] seekTo throws NPE when no offset found for a timestamp
2 participants