Issue with PersistentDataset

[virginiafdez]

Hi,

I am running a code that uses PersistentDataset in 2 different machines. On one of them it runs well, and on the other, which is running via a Docker container, I am getting this issue:

Traceback (most recent call last):
  File "~/models/brainSPADE_f/CODE/brainSPADE_f/train.py", line 78, in <module>
    for dind, data_i in enumerate(dataloader):
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1183, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/opt/conda/lib/python3.8/site-packages/monai/data/iterable_dataset.py", line 57, in __iter__
    for i, item in enumerate(self.source):
  File "~/models/brainSPADE_f/CODE/brainSPADE_f/data/spadenai_v2_sliced.py", line 129, in getIteratorFun
    for volumes in dataset:
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 97, in __getitem__
    return self._transform(index)
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 364, in _transform
    pre_random_item = self._cachecheck(self.data[index])
  File "/opt/conda/lib/python3.8/site-packages/monai/data/dataset.py", line 332, in _cachecheck
    return torch.load(hashfile)
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 600, in load
    with _open_zipfile_reader(opened_file) as opened_zipfile:
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 242, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

The cache files that are being created seem to have the correct permissions, as well as the directory.
Any idea why this issue might be happening?

Thank you.

Virginia

[wyli]

could these be corrupted files like discussed here pytorch/pytorch#31620 (comment), have you tried removing some of the cache files -- they should be regenerated when starting the persistent dataset.

[virginiafdez]

Yes, I get this error after removing the cache directory, which is then created again and errors out this way. So I made sure it's completely empty before running the code again.