Zstash produced corrupted tar file?

[TonyB9000]

The v2_LE_sp370 ensemble 8 archive tar file (v2.LR.SSP370_0131/000000.tar) appears corrupted at the source. We have pulled it over twice, so it appears to have been corrupted where it was produced. (No telling what zstash version was used).

Attempted extractions report corrupted entries:

(dsm_dev_LE2_nE2C_2) -bash-4.2$ zstash extract --hpss=none archive/atm/hist/v2.LR.SSP370_0131.eam.h0.2015*.nc
INFO: zstash/000000.tar exists. Checking expected size matches actual size.
INFO: Opening tar archive zstash/000000.tar
INFO: Extracting archive/atm/hist/v2.LR.SSP370_0131.eam.h0.2015-01.nc
Traceback (most recent call last):
  File "/home/bartoletti1/mambaforge/envs/dsm_dev_LE2_nE2C_2/lib/python3.9/site-packages/zstash/extract.py", line 532, in extractFiles
    tarinfo: tarfile.TarInfo = tar.tarinfo.fromtarfile(tar)
  File "/home/bartoletti1/mambaforge/envs/dsm_dev_LE2_nE2C_2/lib/python3.9/tarfile.py", line 1122, in fromtarfile
    obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
  File "/home/bartoletti1/mambaforge/envs/dsm_dev_LE2_nE2C_2/lib/python3.9/tarfile.py", line 1066, in frombuf
    raise InvalidHeaderError("bad checksum")
tarfile.InvalidHeaderError: bad checksum
ERROR: Retrieving archive/atm/hist/v2.LR.SSP370_0131.eam.h0.2015-01.nc

and a direct "tar tvf" reports:

(dsm_dev_LE2_nE2C_2) -bash-4.2$ tar tvf zstash/000000.tar > 000000_list
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

with output written to 000000_list:

(dsm_dev_LE2_nE2C_2) -bash-4.2$ more 000000_list
-rw-r--r-- nanr/mp9       7886 2022-06-30 08:08 log.o
-rw-r--r-- nanr/mp9       8266 2022-06-30 08:08 log.o60192129
-rw-r--r-- nanr/mp9       5263 2022-06-30 08:08 log.o60256424
-rw-r--r-- nanr/mp9       5728 2022-06-30 08:08 log.o60605844
-rw-r--r-- nanr/mp9        195 2022-07-25 09:32 zstash_create_0131_20220725.log

I'm not sure this is something that can be caught by zstash at creation time. It would be interesting to have "tar cvf | grep "failure status" post archive, just to see if it was clean at that time.

(Posted here at the behest of Jill).

[chengzhuzhang]

Thank you @TonyB9000 for post this discussion. The 000000.tar is the only tar that is problematic for this simulation. We'd like to get instruction on how to best update the zstash archive in this situation.

[TonyB9000]

Thanks Jill - I wasn't thinking about the replacement procedure (or even the recreation procedure.)

[TonyB9000]

@forsyth2 Ryan, if we obtain a replacement for an archive's 000000.tar, what procedure should be used to incorporate it into the body of remaining tar-files and index.db? Naively, I just thought a 1-1 replacement would suffice. (Now, it occurs to me that the archive creator cannot just specify "recreate the 000000.tar". The existing index.db lists all the files that should be there, so perhaps that could be a guide?

[forsyth2]

It would be interesting to have "tar cvf | grep "failure status" post archive

I can make an issue for that at least.

what procedure should be used to incorporate it into the body of remaining tar-files and index.db?

I will need to think about this. Your reasoning seems to be on the right track though.

[forsyth2]

It would be interesting to have "tar cvf | grep "failure status" post archive

I thought create already checks for failures? I.e., https://github.com/E3SM-Project/zstash/blob/main/zstash/create.py#L99

what procedure should be used to incorporate it into the body of remaining tar-files and index.db?

I unfortunately don't have expertise in this area. I've always run zstash to create new archives or update an existing archive with certain files. I don't think there's going to be an easy way to replace the contents of one particular tar. Your idea of listing all the files and then tarring all those into a new 000000.tar seems like the best solution.

Let me know if I should set up a meeting to discuss this further.

[TonyB9000]

@forsyth2 It is possible that the file became corrupted after creation, but before the Globus transfer ... somehow.

I'll use tar tvf to obtain the file list. My only concern is that the files may not "tar" identically to their original, and so I must trust that the "index.db" is employed only to identify which data files are located in which tar-files, etc. If any other checks are performed during "zstash extract", it may fail. (For instance, if the index.db says "file_xyz is located at index W in 000000.tar", and file_xyz is no longer located at index W, because the files are tarred in a different order, we could fail.) I have no idea what zstash does internally.

EDIT: I don't know what I'm saying/thinking. I can't use "tar tvf 000000.tar" because the tar file is corrupt. I must query the index.db file, to somehow list all of the files it "thinks" are in 000000.tar. Hopefully "zstash ls -l" indicated the tarfiles associated with filenames.

[golaz]

@TonyB9000 : I think I have seen this before. Here is my hypothesis of what might have gone wrong. The output of zstash create was piped into a log file that was itself included in the zstash archive leading to a corrupt tar file.

What happens if you try tar xvf? Does it bail out after trying to extract the bad file?

[golaz]

I found some old notes and this indeed appears to be the same issue that another user encountered a while back.

tar xvf 000000.tar will fail, but the following might be able to extract all good files from the tar file:

cpio -ivd -H ustar < 000000.tar

[TonyB9000]

Thanks Chris! I'll try that.

[TonyB9000]

@golaz @chengzhuzhang @forsyth2 That definitely works! Amazing. How did I not know of this?!

This will bring the v2_LE_ssp370 extraction to 100%

Thank you Chris!

[golaz]

Great! Don't remember exactly how I found the cpio command. Probably some Google search on how to extract data from corrupt tar files. The tar file format is very basic: header followed by data, then repeat as needed. So some bad data somewhere does not mean that everything is lost. Even if a header is bad, one can sometimes still salvage something by inspecting the tar using hexdump and manually identifying the beginning of the data.

If you want to check the md5 sums of the files you extracted, we can get those from the database directly. Something like:

sqlite3 index.db
select * from files where tar="000000.tar";

[TonyB9000]

@golaz Thanks Chris. I can get the original hashes from the index.db - also with "zstash ls -l" and some cutting. I wonder how long it will take to run md5 (dgst) on these... (all extracted perfectly, BTW).

Tail of cpio extractions: (Edited - initially extracted wrong tar file!)

. . .
-rw-r--r--. 1 bartoletti1 publishers 482385200 Aug 29 19:53 v2.LR.SSP370_0131.eam.h0.2038-05.nc
-rw-r--r--. 1 bartoletti1 publishers 482385200 Aug 29 19:53 v2.LR.SSP370_0131.eam.h0.2038-06.nc
-rw-r--r--. 1 bartoletti1 publishers 482385200 Aug 29 19:54 v2.LR.SSP370_0131.eam.h0.2038-07.nc
-rw-r--r--. 1 bartoletti1 publishers 482385200 Aug 29 19:54 v2.LR.SSP370_0131.eam.h0.2038-08.nc

head of existing ens 8 atmos mon warehouse files:

-rw-rw-r--. 1 bartoletti1 publishers 482385200 Jul  3  2022 v2.LR.SSP370_0131.eam.h0.2038-09.nc
-rw-rw-r--. 1 bartoletti1 publishers 482385200 Jul  3  2022 v2.LR.SSP370_0131.eam.h0.2038-10.nc
-rw-rw-r--. 1 bartoletti1 publishers 482385200 Jul  3  2022 v2.LR.SSP370_0131.eam.h0.2038-11.nc
-rw-rw-r--. 1 bartoletti1 publishers 482385200 Jul  3  2022 v2.LR.SSP370_0131.eam.h0.2038-12.nc
. . .

Awesomeness!

[forsyth2]

@TonyB9000 I want to confirm this issue is resolved. If so, I'll close the discussion. Thanks!

[TonyB9000]

@forsyth2 Yes, I would close this issue. As far as we can tell, the "corruption" was likely due to creating an archive in the very directory being archived (should test if that is the real issue - if so, a way to avoid that would be nice) but is not a real flaw with zstash. Otherwise, we don't know what caused the issue (although cpio -ivd -H ustar < 000000.tar enabled extraction of the tarred files.)