Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support writing state tables to Snowflake #928

Open
henneberger opened this issue Oct 18, 2024 · 0 comments
Open

Support writing state tables to Snowflake #928

henneberger opened this issue Oct 18, 2024 · 0 comments
Labels
enhancement New feature or request snowflake-engine
Milestone

Comments

@henneberger
Copy link
Contributor

henneberger commented Oct 18, 2024

Add this to track external limitations of Snowflake / flink iceberg connector.

Currently snowflake does not support equality or positional deletes but support is planned.

The flink iceberg connector does not support copy-on-write or table maintenance so there currently no path to compact a table. Some people suggest using spark solely to compact a table, which is a possible workaround. To get it to work in snowflake, all iceberg tables should annotated which the iceberg metadata file, and would need to be manually updated as new tables are compacted. https://github.com/orgs/apache/projects/358
https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA

@henneberger henneberger converted this from a draft issue Oct 18, 2024
@github-project-automation github-project-automation bot moved this to Backlog in DataSQRL Oct 18, 2024
@henneberger henneberger added enhancement New feature or request snowflake-engine labels Oct 18, 2024
@henneberger henneberger added this to the uncategorized milestone Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request snowflake-engine
Projects
Status: Ready
Status: Ready
Development

No branches or pull requests

1 participant