fix(concurrent-partition-cursor): update cursor comparison to leverage ab_datetime #297
Conversation
📝 WalkthroughWalkthroughThe pull request introduces a modification to the Changes
Possibly related PRs
Suggested labels
Suggested reviewers
Hey there! I noticed you've made some interesting changes to the datetime comparison logic. Would you mind sharing a bit more context about what specific datetime parsing challenges you encountered that led to this modification? Wdyt about adding a quick comment explaining the rationale? This could help future maintainers understand the reasoning behind the change. Cheers! 👍 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (8)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py (1)
150-155: Consider adding error handling for datetime parsing.The cursor comparison enhancement looks good! Using
ab_datetime_parseto convert to epoch milliseconds will handle different datetime formats consistently. However, what do you think about adding error handling for cases where the cursor values might not be valid datetime strings? 🤔Example approach:
if ( self._new_global_cursor is None - or ab_datetime_parse( - self._new_global_cursor[self.cursor_field.cursor_field_key] - ).to_epoch_millis() - < ab_datetime_parse( - cursor.state[self.cursor_field.cursor_field_key] - ).to_epoch_millis() + or self._compare_cursor_values( + self._new_global_cursor[self.cursor_field.cursor_field_key], + cursor.state[self.cursor_field.cursor_field_key] + ) ): self._new_global_cursor = copy.deepcopy(cursor.state) +def _compare_cursor_values(self, value1: str, value2: str) -> bool: + try: + return (ab_datetime_parse(value1).to_epoch_millis() < + ab_datetime_parse(value2).to_epoch_millis()) + except ValueError as e: + logger.error(f"Failed to parse cursor values: {e}") + # Fall back to string comparison or raise based on your preference + return value1 < value2wdyt? This would make the code more robust against invalid datetime formats. 🛡️
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py(2 hunks)
🧰 Additional context used
🪛 GitHub Actions: Linters
airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py
[error] 5-28: Import block is un-sorted or un-formatted. This can be fixed automatically using the --fix option with ruff.
⏰ Context from checks skipped due to timeout of 90000ms (9)
- GitHub Check: Check: 'source-pokeapi' (skip=false)
- GitHub Check: Check: 'source-the-guardian-api' (skip=false)
- GitHub Check: Check: 'source-shopify' (skip=false)
- GitHub Check: Check: 'source-hardcoded-records' (skip=false)
- GitHub Check: Pytest (All, Python 3.12, Ubuntu)
- GitHub Check: Pytest (All, Python 3.11, Ubuntu)
- GitHub Check: Pytest (Fast)
- GitHub Check: Pytest (All, Python 3.10, Ubuntu)
- GitHub Check: Analyze (python)
🔇 Additional comments (1)
airbyte_cdk/sources/declarative/incremental/concurrent_partition_cursor.py (1)
26-26: Fix import block sorting.The import block needs to be sorted. This can be fixed automatically using the
--fixoption with ruff.Would you like me to generate a script to fix the import sorting? 🔧
natikgadzhi
changed the title
|
Closing as alternative PR merged: #298 |
What
AirbyteDatetimeto parse and compare cursor values that may have different formats.Summary by CodeRabbit