Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zarr copy fails on write - variable uncompressable #3075

Open
mannreis opened this issue Jan 14, 2025 · 1 comment
Open

Zarr copy fails on write - variable uncompressable #3075

mannreis opened this issue Jan 14, 2025 · 1 comment

Comments

@mannreis
Copy link
Contributor

mannreis commented Jan 14, 2025

When experimenting with #3068 I bumped into this. I believe it was not introduced in such PR and trying with 4.8.1 it works fine.

nccopy file://ngc4008_P1D_0_crs.zarr.zip#mode=zarr,zip  file:///var/tmp/out.zarr#mode=zarr,file
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/k/k202186/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138

I pin pointed the variable that was causing the error and created a simplied dataset that could reproduce it: ngc4008_P1D_0_crs.zarr.zip

the respective ncdump works without errors:

$ ncdump file:///home/k/k202186/ngc4008_P1D_0_crs.zarr#mode=zarr,file
netcdf ngc4008_P1D_0_crs {
dimensions:
	crs = 1 ;
variables:
	float crs(crs) ;
		crs:grid_mapping_name = "healpix" ;
		crs:healpix_nside = 1 ;
		crs:healpix_order = "nest" ;
data:

 crs = NaNf ;
}

I'll try to follow it up.


Relevant logs

NCZ_close: ncid 0x20000
		ncz_closeorabort: file: 0x625d3453acf0
			ncz_sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x20000
		FILE - path: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr cmode: 0x1001 parallel: 0 redef: 0 fill_mode: 256 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 1
		 GROUP ATTRIBUTE - attnum: 0 name: _NCProperties type: 2 len: 47
		 DIMENSION - dimid: 0 name: crs len: 1 unlimited: 0
		 VARIABLE - varid: 0 name: crs ndims: 1 dimids: 0 storage: chunked
		 VAR ATTRIBUTE - attnum: 0 name: grid_mapping_name type: 2 len: 7
		 VAR ATTRIBUTE - attnum: 1 name: healpix_nside type: 4 len: 1
		 VAR ATTRIBUTE - attnum: 2 name: healpix_order type: 2 len: 4
					NCZ_write_ncproperties
			ncz_sync_file: file: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr
			ncz_sync_grp: dims: (null)
			ncz_sync_atts
		NCZ_inq_var_all: ncid 0x20000 varid 0
				ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
		NC4_inq_var_all: ncid 0x20000 varid 0
		NCZ_inq_var_all: ncid 0x20000 varid 0
				ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
		NC4_inq_var_all: ncid 0x20000 varid 0
				chunksizesp[0]=1
			ncz_sync_atts
			put_chunk: var: 0x625d3453fa30
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/reis/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138
					NC4_clear_provenance
					NCZ_clear_provenance
@DennisHeimbigner
Copy link
Collaborator

The failure is on the compression side. As you note, ncdump works because is doing
decompression only.
In looking at the cblosc doc. it appears that there is a simple fix, so I modified
plugins/H5Zblosc.c to include that fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants