[Feature Request] Introduce Sparse index for indexes with sort fields

[mgodwan]

Is your feature request related to a problem? Please describe

Lucene 10 and above supports sparse indexing on doc values via FieldType#setDocValuesSkipIndexType. The sparse index will record the minimum and maximum values per block of doc IDs. Used in conjunction with index sorting to cluster similar documents together, this allows for very space-efficient and CPU-efficient filtering.

Describe the solution you'd like

OpenSearch should leverage this for indices with sort fields as this allows to create sparse index, and for use cases such as datastreams/time-series indices with sorting on timestamp, we may benefit from not having to create the points data structure for the field, resulting in reduced storage use, and efficient filtering for aggregation use cases.

Related component

Indexing:Performance

Describe alternatives you've considered

No response

Additional context

No response

[mgodwan]

I'm working on analysing the integration points. I will come back with possible approaches on this, and if this can be solved with our upcoming OpenSearch 3.0 release.

[andrross]

Catch All Triage - 1, 2, 3, 4, 5