Create tar files for season 9 and and transfer to TACC

[julianpistorius]

Follow-up issue: #17

Task to do

Create tar files for season 9 and transfer to TACC

Reason

Ensure TERRA REF backed up to TACC and GDRIVE

Result

• Tar files on TACC containing raw_data/ plus Level_1/laser3Dply for season 9
• Also on TACC: Matching '.md5' files containing the contents of the tar files with the md5 checksums

Steps to take

See detailed instructions: https://hackmd.io/LrK3vS9eT1mOWeDZ5nuSBg

[julianpistorius]

The following sub-directories in /terraref/sites/ua-mac/raw_data/stereoTop/ are owned by root:

drwx------     2 root     root            4096 Nov 28  2017 2017-09-20
drwx------     2 root     root            4096 Nov 28  2017 2017-09-10
drwx------     2 root     root            4096 Nov 28  2017 2017-09-07
drwx------     2 root     root            4096 Nov 28  2017 2017-09-06
drwx------     2 root     root            4096 Nov 28  2017 2017-09-05
drwx------     2 root     root            4096 Nov 28  2017 2017-09-01

Asking Sean to change permissions/ownership for me.

[julianpistorius]

Progress

• Script to create tar files
  • See PR 88 on the TERRA REF admin repo
  • Tarring of sensor data on the VM at NCSA seems to take about 10 minutes per 50G. That's about 2 hours (to 2 hours and 15 minutes) per (512GB) tar file, or 4 hours per TB.
  • Outstanding bugs:
    • No .md5 files (known cause)
    • Only created 7 tar files instead of 14 (unknown cause)
• Tested Globus TACC transfer:
  • Success
  • It took 42 minutes to transfer and checksum the 500GB tar file, the effective speed is about 5 seconds per GB

Proof

$ cd /terraref/users/scratch/archive_work

$ ls -alth arch_ua-mac_raw_data_stereoTop_S9*
-rw-r--r-- 1 jpistorius prj_cg_arpae 449G Jun 30 09:39 arch_ua-mac_raw_data_stereoTop_S9.P7.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  13M Jun 30 09:39 arch_ua-mac_raw_data_stereoTop_S9.P7.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 30 07:41 arch_ua-mac_raw_data_stereoTop_S9.P7.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 30 07:41 arch_ua-mac_raw_data_stereoTop_S9.P6.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 30 07:41 arch_ua-mac_raw_data_stereoTop_S9.P6.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 30 05:46 arch_ua-mac_raw_data_stereoTop_S9.P6.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 30 05:46 arch_ua-mac_raw_data_stereoTop_S9.P5.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 30 05:46 arch_ua-mac_raw_data_stereoTop_S9.P5.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 30 03:49 arch_ua-mac_raw_data_stereoTop_S9.P5.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 30 03:49 arch_ua-mac_raw_data_stereoTop_S9.P4.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 30 03:49 arch_ua-mac_raw_data_stereoTop_S9.P4.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 30 01:53 arch_ua-mac_raw_data_stereoTop_S9.P4.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 30 01:53 arch_ua-mac_raw_data_stereoTop_S9.P3.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 30 01:53 arch_ua-mac_raw_data_stereoTop_S9.P3.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 29 23:42 arch_ua-mac_raw_data_stereoTop_S9.P3.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 29 23:42 arch_ua-mac_raw_data_stereoTop_S9.P2.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 29 23:42 arch_ua-mac_raw_data_stereoTop_S9.P2.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 29 21:35 arch_ua-mac_raw_data_stereoTop_S9.P2.err
-rw-r--r-- 1 jpistorius prj_cg_arpae 498G Jun 29 21:35 arch_ua-mac_raw_data_stereoTop_S9.P1.tar
-rw-r--r-- 1 jpistorius prj_cg_arpae  15M Jun 29 21:35 arch_ua-mac_raw_data_stereoTop_S9.P1.tar.toc
-rw-r--r-- 1 jpistorius prj_cg_arpae   44 Jun 29 19:28 arch_ua-mac_raw_data_stereoTop_S9.P1.err

$ globus task show 9645f86e-bb06-11ea-bef4-0e716405a293
Label:                   None
Task ID:                 9645f86e-bb06-11ea-bef4-0e716405a293
Is Paused:               False
Type:                    TRANSFER
Directories:             0
Files:                   2
Status:                  SUCCEEDED
Request Time:            2020-06-30T19:19:16+00:00
Faults:                  0
Total Subtasks:          4
Subtasks Succeeded:      4
Subtasks Pending:        0
Subtasks Retrying:       0
Subtasks Failed:         0
Subtasks Canceled:       0
Subtasks Expired:        0
Completion Time:         2020-06-30T20:02:52+00:00
Source Endpoint:         ncsa#terra
Source Endpoint ID:      da262cbf-6d04-11e5-ba46-22000b92c6ec
Destination Endpoint:    TACC Mig2 Ranch with XSede Authentication
Destination Endpoint ID: 1ef1b518-e6bf-11e8-8c9a-0a1d4c5c824a
Bytes Transferred:       534326952393
Bytes Per Second:        204203486

$ globus ls -a -l 1ef1b518-e6bf-11e8-8c9a-0a1d4c5c824a:/stornext/ranch_01/ranch/projects/TERRA-REF/season-9/Level_0/stereoTop/
Permissions | User     | Group    | Size         | Last Modified             | File Type | Filename                                    
----------- | -------- | -------- | ------------ | ------------------------- | --------- | --------------------------------------------
0644        | tg833798 | G-822207 | 534311976960 | 2020-06-30 19:40:42+00:00 | file      | arch_ua-mac_raw_data_stereoTop_S9.P1.tar    
0644        | tg833798 | G-822207 |     14975433 | 2020-06-30 19:21:43+00:00 | file      | arch_ua-mac_raw_data_stereoTop_S9.P1.tar.toc