PIT (point in time) for pagination

[andresribeiro]

The most relevant results of a search can change frequently, especially when a lot of data is added. As a result, pagination may not have consistency in the data returned. In a listing, you could have duplicate or missing data. As a solution to this, Elasticsearch added PIT, which fixes this issue. I think it would be awesome if Meilisearch also implement some similar solution

[gmourier]

Hi @andresribeiro 👋

Are you currently experiencing difficulties with paging and thinking that stateful paging would solve your problem?

I see it as a use-case obviously to avoid drifting of results between pages if some new documents are repositioned or added as they come in under heavy load.

From what I understand, documents deleted in the meantime are still returned for the PIT pagination state, right?

Technically ElasticSearch has a ready-made solution to implement this type of paging through what they call segments which seems more complicated for Meilisearch in its current state to implement something like this. @Kerollmops will be more comfortable than me to give a thought about it.

[Kerollmops]

Hey @andresribeiro,

This is a good idea, and indeed ElasticSearch can start reading its index from a particular point in time (the segment ID) to be sure that results are consistent between calls to get the different pages.

Unfortunately, Meilisearch isn't a segment-based search engine, and doing so couldn't be done the same way. Meilisearch uses LMDB under the hood, we open transactions to be sure to get consistent results between calls in the database. One solution would be to keep a transaction open for as long as the user needs to fetch the pages but it is a very bad idea as it can make the database grow a lot, it is not recommended. For now, I have no real idea on how to implement this without impacting the database size.