Zarr Confronted with Buffer limit

[tjk9501]

Hello, I am using zarr to parallel processing large 2d arrays, as described in:

https://github.com/zarr-developers/zarr-python/discussions/1261#discussioncomment-4140324

I have use the following code to process Rt

filename = directionFolder+'R'+str(ii)+'d'+str(jj)+'_iter'+str(nit)+'.zarr'
filedim = directionFolder+'R'+str(ii)+'d'+str(jj)+'_iter'+str(nit)+'.zarrdim'
#Set compression
compressor = Blosc(cname='zstd', clevel=3, shuffle=Blosc.BITSHUFFLE)
Rt_Zarr = za.open(filename, mode='w', shape=Rt_gpu.shape, chunks=(Rt_gpu.shape[0],int(Rt_gpu.shape[1]/10)), compressor=compressor, dtype=Rt_gpu.dtype)
Rt_Zarr[:] = cp.asnumpy(Rt_gpu)
za.save(filedim,dim = Rt_gpu.shape)

but the following error prompted

File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "h:\jktong\resolutionexp_conv\train_cupy_v7_all_format.py", line 301, in <module>
    Rt_Zarr[:] = cp.asnumpy(Rt_gpu)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 1373, in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 1468, in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 1772, in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 1824, in _set_selection
    self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 2089, in _chunk_setitem
    self._chunk_setitem_nosync(chunk_coords, chunk_selection, value,

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 2100, in _chunk_setitem_nosync
    self.chunk_store[ckey] = self._encode_chunk(cdata)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\zarr\core.py", line 2227, in _encode_chunk
    cdata = self._compressor.encode(chunk)

  File "numcodecs/blosc.pyx", line 557, in numcodecs.blosc.Blosc.encode

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\numcodecs\compat.py", line 155, in ensure_contiguous_ndarray
    ensure_contiguous_ndarray_like(

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\numcodecs\compat.py", line 121, in ensure_contiguous_ndarray_like
    raise ValueError(msg)

ValueError: Codec does not support buffers of > 2147483647 bytes

if I change the code

Rt_Zarr = za.open(filename, mode='w', shape=Rt_gpu.shape, chunks=(Rt_gpu.shape[0],int(Rt_gpu.shape[1]/10)), compressor=compressor, dtype=Rt_gpu.dtype)

to

Rt_Zarr = za.open(filename, mode='w', shape=Rt_gpu.shape, chunks=(Rt_gpu.shape[0],int(Rt_gpu.shape[1]/20)), compressor=compressor, dtype=Rt_gpu.dtype)

no error will appear, but the speed will be 10 seconds slower than using pandas.to_pickle. So I want to know how to solve this problem. Thanks!!!