Best Practices for Memory Usage

[FannyFirst]

Hii Materialize team
I'm trying Materialize, but I have some confusion: through experiments I found that Materialize uses a lot of memory. But I don't want to store a lot of data in memory, I learned that Timly Dataflow has Capture and Replay mechanism, but I don't know how to use this function in Materialize. I want to reduce the memory usage through some mechanisms. In my scenario, there are probably more than 100 million to 2 billion rows of data, and the data size is roughly 10GB to 100GB. I cannot keep this data in memory. Is there any solution or best practice please.
thanks a lot

[bobbyiliev]

Hi there @FannyFirst,

One thing that you could do here is to use temporal filters, that way you can specify only a window of time that you want to materialize so not all of your data is kept in memory all of the time.

For more information on how to use temporal filters, I could suggest the documentation here:

https://materialize.com/docs/sql/patterns/temporal-filters/#details

And also the following blog post by Frank:

https://materialize.com/blog/temporal-filters/

Additionally, I would recommend the following documentation which includes some helpful steps that you can take in order to identify why Materialize is using so much memory:

https://materialize.com/docs/ops/troubleshooting/#why-is-materialize-using-so-much-memory

Let me know if you have any questions!