Learned transformer for String indexing

[Jiaweihu08]

Description

This PR explores the ml approach for String column indexing by conoceptualizing the task of ordering String values lexicographically as learning a Cumulative Density Function for their ranks. The goal is to place strings that are lexicographically close in neighbouring cubes in order to reduce file-level min/max.

The idea is explored more thoroughly in this paper.

Training and indexing

• Extract unique values from the given column, sort the values lexicographically
• Create labels as the rank of the sorted unique values
• Encode strings by converting each character to their ascii value, creating fix-sizes numeric Sequence, padding the shorter strings while truncating the longer ones
• Train a regressor to predict string ranks from their vector encodings
• Deploy model as Transformer / Transformation, index the inferred rank value

Reading and filtering

Relying on model prediction for filtering strings will lead to false negatives, but given the deterministic nature of the models, equalTo can be safely implemented.

Other posibilities can be explored, e.g. adding file string column min/max to cube block metadata.

TODOs:

• New feature / bug fix has been committed following the Contribution guide
• Add comments to the code (make it easier for the community!)
• Add tests
• Your branch is updated to the main branch (dependent changes have been merged)
• Benchmark the overhead
• Investigate spark and parquet file pruning
• Learning API(?) Model loading and saving(?)
• More robust default model
• xgboost4j for MAC Apple Silicon
• Update documentation

Test Configuration:

• Spark Version: 3.3.0
• Hadoop Version: 3.3.4

[cdelfosse]

To be closed

[osopardo1]

Let's close this one!

@Jiaweihu08 I give you the honor 🗝️

Replaced by solution with Histograms in #221