Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add fallback TableProvider to FederatedTableProviderAdaptor #39

Merged

Conversation

phillipleblanc
Copy link
Collaborator

Adds a fallback TableProvider that can be used in DataFusion contexts without the federation optimizer in-use, which provides a graceful fallback mechanism.

@phillipleblanc phillipleblanc changed the title Add fallback TableProvider to FederatedTableProviderAdaptor feat: add fallback TableProvider to FederatedTableProviderAdaptor Jun 18, 2024
Copy link
Collaborator

@backkem backkem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
One thing I'd like to explore is if we can disable the fallback if the optimiser is registered. Just to avoid confusion where one (wrongly) assumes the federation is done but in reality the fallback is being used. But this can be picked up in a follow-up.

@phillipleblanc
Copy link
Collaborator Author

LGTM. One thing I'd like to explore is if we can disable the fallback if the optimiser is registered. Just to avoid confusion where one (wrongly) assumes the federation is done but in reality the fallback is being used. But this can be picked up in a follow-up.

This should be possible, since we are passed the SessionState into scan and insert_into - and SessionState has the optimizer rules. Unfortunately there isn't an accessor method to get the currently loaded optimizer rules (there is for physical_optimizers, but not the logical ones). I'm guessing a PR into DataFusion would be accepted to add those methods, and then we could check here if the federation rule is present - and return the error regardless of a fallback table provider.

@backkem backkem merged commit ce87bf4 into datafusion-contrib:main Jun 18, 2024
9 of 10 checks passed
@phillipleblanc phillipleblanc deleted the phillip/upstream-fallback branch June 19, 2024 03:14
phillipleblanc added a commit that referenced this pull request Jan 17, 2025
#39)

* feat: fix OuterReferenceColumns not being rewritten correctly (#38)

* Fix OuterReferenceColumns not being rewritten correctly

* wip

* Handle collecting rewrites from expressions

* Add more unit tests for collect rewrite

* Fix comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants