Thoughts on regionally varying keep bits

[observingClouds]

In todays Pangeo talk presenting xbitinfo the idea came up to allow for regionally (spatial, temporal) varying keep bits. One example would be the height dimension. Depending on the variable, a surface layer might have more actual variability than is present ( or "real" ) at high altitudes. As a consequence the surface layer needs more keep-bits to preserve the mutual information than a stratospheric layer.

Theoretically this is already possible:

1. The bit information can be retrieved for regional subsets
2. Bit rounding can be applied to individual regions/chunks
3. Compression does not need to be specific to a region. Its efficiency just depends on the bit-pattern ( less keep-bits/ more zero bits --> better compression )
4. Metadata about the bit rounding e.g. _QuantizeBitRoundNumberOfSignificantBits would not be a global attribute and should rather be defined per chunk ( or not at all, as it can be retrieved by analysing the data )

How can this be made more user friendly?
In particular:

1. How should regions be defined? As a list of slices?
2. How should this be saved in a dataset?
3. Are there cases when the compression would benefit from chunk dependent compressors? Should zarr support different compressors per chunk?

Ultimately, one might want to experiment with retrieving the mutual information at every grid-point at every time and apply bit rounding accordingly. Ideally, the keep bits would dynamically adjust. If the grid-points are for example within an ocean-eddy the number of keep-bits increases compared to a rather laminar flow in other parts of the ocean. In this case the retrieval of the bitinformation needs to be directly linked with the bit-rounding.

I'm curious to see other thoughts.

[rsignell-usgs]

@observingClouds I'm super excited about calculating keepbits on a per-chunk basis! This would decrease the storage further as well as be more faithful to the information content because as you mention, the information content certainly varies with time and space! Snow-depth in the US is a good example!

We might not want to try this until the Dask workers can calculate the keepbits for each chunk in parallel using the keepbits algorithm in Python, right?

[observingClouds]

Glad you are excited! Yes, with Dask support it will make most sense. I haven't found time to fix the small nasty bug yet.

[augustus-thomas]

Hey!

I read that you are offering to mentor this project for GSoC and I am very interested in applying! I just had a few questions while thinking about a proposal:

1. You reference the attribute_QuantizeBitRoundNumberOfSignificantBits. Is there any algorithm already implemented to compute this attribute to optimize loss or compression? If not then writing one would be a central part of this project, since we could apply it to each chunk during testing.
2. Is it a good idea to enable the user to define a maximum tolerance for local loss of mutual information? I am unfamiliar with loss compression schemes and how they quantify loss, do you have any good reading suggestions? My frame of reference comes from rounding measurements in the lab and relative error considerations.
3. It seems to me that storing the keep bits for each individual address would not increase the size of the data structure, since we would always compress off more bits in the dataset than those in the keep bits table. I think I'm wrong though because you suggest we use non-unit sized chunks.
4. I plan to brush up on this codebase and related xarray projects before beginning. Do you have any suggestions for getting familiar, or specific aspects of this project that I should pay attention to?

I recognize that I am a complete beginner. I open to any and all of your feedback and suggestions :)

[milankl]

Just to chip in for

2. The maximum loss of bitwise real information is 1 minus the inflevel, often taken as 0.99 or 99%, meaning that you want to preserve at least 99% of information. This is how the keepbits are found. There are tons of ways of how to define loss... the standard ways are error metrics, which you'd want to be as small as possible to reduce loss. There are also several image quality metrics like the structural similarity index measure, penalising a lossy compression for a mean, variance and correlation error. This can be applied to an array too. You probably saw this paper there are also some references by Allison Baker and or Dorit Hammerling like this: https://onlinelibrary.wiley.com/doi/10.1111/cgf.13707 or https://gmd.copernicus.org/articles/9/4381/2016/

But other than error metrics, what we do here is information theory-based. So in the end you compare two arrays with respect to quantities like their mutual information, redundancy, cross entropy etc. In contrast to the error metrics you often want to maximise the information.

3. You never really have to store the number of keepbits explicitly, as they are just the number of trailing 0 bits all floats in a chunk have in common.

julia> bitstring.(A,:split)    # 1 sign, 8 exponent, 23 mantissa bits
5-element Vector{String}:
 "0 01111110 00110010000000000000000"
 "0 01111011 00100000000000000000000"
 "0 01111100 01011100000000000000000"
 "0 01111101 11110000000000000000000"
 "0 01111101 10111010000000000000000"

Here you'd know that keepbits was 7, which is actually quite neat because you encode the uncertainty within a dataset without explicitly storing this information. But maybe I'm misunderstanding your question?

[observingClouds]

Hi @augustus-thomas,
Thanks for your interest in this project and applying for the GSoC. Please note that we are currently working on updating the challenges for this year's GSoC.
Regarding your questions:

1. _QuantizeBitRoundNumberOfSignificantBits is indeed an attribute that is used by the netCDF library and its BitRound implementation. We adapt this attribute in xbitinfo as numcodecs BitRound algorithm (which we use in xbitinfo) does not set this attribute. We calculate the _QuantizeBitRoundNumberOfSignificantBits with get_keepbits. We might drop the attribute in the future as setting attributes per chunk is not supported by any current data format we use (netCDF, zarr). _QuantizeBitRoundNumberOfSignificantBits is only meaningful if the entire data array has the same number of keepbits. As Milan points out in his response to 3) you can infer the number of keepbits also at a later stage.

4. Please have a look at the issues that are marked with the label good_first_issue. These should in general be easier and do not require a specific knowledge on the xbitinfo algorithm but will help you to get more familiar with the package.