Adopting lossy compression can be scary. Nobody wants to loose important real information when compressing their data. We want to make lossy compression fearless for weather and climate scientists so you specifically and we as a community can all benefit from higher compression ratios.
We are working on the following three steps:
-
Establish lossy compression requirements based on which compression errors we can tolerate in general and for specific usecases. This tolerance depends e.g. on the variable and any uncertainty it carries from its measurement or modelling.
-
Benchmark existing compression methods to identify which ones provide the best performance (compression ratio, computational cost, ...) while meeting the above requirements.
-
Provide recommendations for which compression methods to adopt.
We are reaching out to identify and take into account your requirements for lossy data compression. If you have a few minutes to spare, please contribute by
-
Select one or more field variables that you work with, or add a row for a new variable
-
Fill in or leave comments for one or more of
- the variable's distribution
- the inherent uncertainty/error from measurements and modelling
- your requirements for the
$L_1$ (MAE),$L_2$ (RMSE),$L_{\infty}$ (max), relative (%), and budget error bounds - any further requirements you have
here: https://docs.google.com/spreadsheets/d/1ADKTlDPzd0ZtW2hZqCSRssNz1UiDAuBj_FYj07K3Vpo
We appreciate your time in helping specify the requirements our community has for lossy data compression!
Our research on data compression in climate science and meteorology is part of ESiWACE3, the third phase of the Centre of Excellence in Simulation of Weather and Climate in Europe.
Funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) under grant agreement No 101093054.