You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@Lordworms added a benchmark for extracting statistics from parquet files in #10610
As this code can be used to extract statistics from parquet files, we would like to make sure it is efficient (especially if we are going to extract statistics for many files at once)
The idea here is to improve the speed of the statistics extraction
Describe the solution you'd like
Make this go faster
cargo bench --bench parquet_statistic
Describe alternatives you've considered
I did some brief profiling:
I think they key would be to change these loops so they built the required Arrow Arrays directly from primitive values rather than from ScalarValue:
/// Returns an iterator over min values stored in `ValueStatistics<i32>`fnextract_i32_mins(stats:implIntoIterator<&Statistics>) -> implIterator<Item = Option<i32>>{
...}
And then with those iterators, we can make the arrays directly
Is your feature request related to a problem or challenge?
Part of #10453
@Lordworms added a benchmark for extracting statistics from parquet files in #10610
As this code can be used to extract statistics from parquet files, we would like to make sure it is efficient (especially if we are going to extract statistics for many files at once)
The idea here is to improve the speed of the statistics extraction
Describe the solution you'd like
Make this go faster
Describe alternatives you've considered
I did some brief profiling:
I think they key would be to change these loops so they built the required Arrow Arrays directly from primitive values rather than from
ScalarValue
:datafusion/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs
Lines 183 to 189 in 1bf7112
Additional context
No response
The text was updated successfully, but these errors were encountered: