completeness of implementation?

[adriangb]

hi @suremarc, I was wondering if the implementation here is complete? I was trying to grok it and came across e.g. https://github.com/datafusion-contrib/datafusion-materialized-views/blob/main/src/rewrite.rs which seems to be an empty file. I wanted to confirm what the state of the repo is before I break my brain. Thanks for your work on this!

[suremarc]

Hey @adriangb, so far the materialized module is pretty much complete, which is the piece related to incrementally maintaining materialized views. The rewrite module is for "query rewriting" which is more about exploiting materialized views to speed up queries. Everything in the materialized module is nearly ready, I'm cooking up an integration test which should serve dually as an example of how to use it, but I'm still ironing out some small bugs.

After finishing up the integration test I plan to port the query rewriting code to that empty file you mentioned.

[alamb]

BTW saw a great talk from @JanKaul about materialized views in DataFusion at

• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 apache/datafusion#12988

He even had some nice relational algebra (thanks to @jakthom for the pictures)

It was not 100% clear to me where that implementation was but I wanted to point it out in this repo as a source of potential collaboration

https://github.com/JanKaul/frostbow

[JanKaul]

The implementation is here:

https://github.com/JanKaul/iceberg-rust/tree/main/datafusion_iceberg%2Fsrc%2Fmaterialized_view

I have to update the documentation and upload the slides somewhere.

[alamb]

The implementation is here:

https://github.com/JanKaul/iceberg-rust/tree/main/datafusion_iceberg%2Fsrc%2Fmaterialized_view

I have to update the documentation and upload the slides somewhere.

Perhaps you can post the slides to

• DISCUSSION: January 2025 DataFusion Meetup in Amsterdam / CIDR 2025 apache/datafusion#12988