feat: initial subquery support #37
Conversation
backkem
changed the title
|
|
||
| // NopFederationProvider is used to represent tables that are not federated, but | ||
| // are resolved by DataFusion. This simplifies the logic of the optimizer rule. | ||
| struct NopFederationProvider {} |
There was a problem hiding this comment.
This really represents TableScans handled by DataFusion - so in some sense it "federates" back to DF itself? 🤔 I wonder if we should call it something like TableFederationProvider ?
There was a problem hiding this comment.
Although Nop has the connotation that it isn't doing anything, so I think its also fine to leave as is.
| mod table_provider; | ||
| pub use table_provider::*; | ||
|
|
||
| mod plan_node; | ||
| pub use plan_node::*; | ||
|
|
||
| pub fn default_session_state() -> SessionState { |
| panic!("Could not locate ScalarSubqueryToJoin"); | ||
| }; | ||
|
|
||
| // TODO: check if we should allow other optimizers to run before the federation rule. |
There was a problem hiding this comment.
ScalarSubqueryToJoin is fairly early on in the optimizer list. I wonder if we could just add this to the end? PushDownLimit and PushDownFilter seem beneficial for the federation case.
There was a problem hiding this comment.
I would suggest that we take the time to think through the implications of having the optimizers run first and land it in separate PRs. Aside from that there are also federation-specific optimizations we can do (hinted at in #23). We may want to make different choices in those than in the local execution case.
Co-authored-by: Suriya Kandaswamy <[email protected]> Co-authored-by: Suriya Kandaswamy <[email protected]>
Ok, let's try this again.
This is a continuation of #33. cc @sardination
The PR includes initial support for subqueries. I also tried to re-write the
analyzeroptimizer logic in an attempt to make it more readable.PS: It's also becoming clear we'll need to look into a good way to get tests setup. For example by federating to DataFusion itself.