Using keepbits along the time dimension?

[rsignell-usgs]

I used the CDSAPI to extract snow-depth and soil-moisture from the ERA5-Land simulation and examined it with xbitinfo.

The fields look like this:

I looked at keepbits along the latitude, longitude and time dimensions for my two variables and they were:

Dimension	KeepBits [snow_depth, soil_moisture]
latitude	[1, 9]
longitude	[1,10]
time	[8,13]

the question: I'm guessing that data like snow_depth and soil-moisture, the keepbits along the spatial dimensions longitude and latitude will be misleading because even though they don't have missing values, they have large regions with constant values, so we should use KeepBits along the time dimension instead?

@milankl or @xbitinfo folks, what do you think?

If you want to try this workflow out, I copied a sample file to a public s3 location so here is a reproducible notebook. Here when I used the most conservative KeepBits along the time dimension, I still obtained over a factor of 3 space savings over the original netcdf file, so I was happy. (The original NetCDF file was netcdf3 which has no compression but is "packed" using short integers with scale_factor and add_offset attributes).

[milankl]

Snow depth: What I'd probably suggest doing in this case is to analyse the bitwise information content on a subarray of your field. E.g. the rectangle between 35˚N,-120˚E and 50˚N,-82˚E. You could use the time dimension instead, but in general this may not always be available, and also includes those constant grid points that are different to the regions that actually have a varying snow depth. The underlying problem is that while there is information in a sense that if there's no snow in some of the states of the US then you basically also know there's no snow further south. However, for many states there's also no entropy to begin with (is there ever snow in Florida?). This means the statistics of entropy and mutual information change throughout the field and are not actually coherent. In such a situation it's often possible and advisable to focus on the subarray which has the highest information content. While that choice is somewhat subjective, it's often intuitive, and I'm not sure that the exact choice matters much. Here, many values are 0 or missing, so bitrounding shouldn't affect them anyway. Hence given that you will only round the non-zero values, ideally we should also analyse the bitwise information over those.

Soil-moisture: I believe soil moisture still has a land-sea mask, but the 9,10,13 keepbits suggest that there's fewer regions with constant values? I reckon if the datasets encloses a desert, similar arguments apply. But from the high number of keepbits I assume that's not actually an issue here.

[rsignell-usgs]

@milankl, I'm glad I've learned something so far -- in cell [8] of the notebook I used I used a subset pretty much as you suggested to eliminate the masked areas:

I think the key thing I'm understanding, thanks to your answer above, is that it's not just masked values that can skew the keepbits towards lower values -- it's the zero entropy actual values in the fields themselves!

[milankl]

Yes, absolutely. In the end the bitwise information content is a statistic that can change throughout a field. Say you look at some humidity/cloud variable which is very patchy in the tropics but rather smooth in mid-latitudes following the weather systems. The bitwise information content is lower the patchier the field (and zero for noise) which makes sense as the uncertainty is higher; for very smooth fields however, the bitinformation content will be higher. Analysing the whole field as one will give you some average between the two regions. Hence in a situation where you really care to preserve information in one region, I'd suggest to analyse the information in the region with the highest information content, and then use it to compress the entire field.