Tar files + Globus -> TACC, Part 3

[julianpistorius]

Follow-on from #14

• Try activating endpoint according to Sean's instructions below Never mind. We're just going to use a different location for the tar & md5 files. Update: Got it working by using my NCSA identity.
• Test transfer from ncsa#terra to TACC Mig2 Ranch with XSede Authentication
• Start batching and transferring to TACC (Follow-up issue for next sprint: Create tar files for season 9 and and transfer to TACC #16 )
• Compile notes from Slack into Markdown and submit to Sean for review
• Get grep for season from Sean
• Sean: Talk to Rob about deletion policy/migration plan

[julianpistorius]

Regarding endpoint Sean is using, he says:

Sean Stevens - 2020-06-08 16:08 (UTC): ok, you probably need to go to "endpoints" on the left bar. Then, select ncsa#terra and "deactivate" it. Then you'll want to reactivate, but when you reactivate, use your NCSA logon. You might have to deactivate several times. Globus can be a little weird

[julianpistorius]

Julian 4:15 PM
Did you see the endpoint that David & Rob sent me in the @ncsa_internal group?
4:15
Is that different?
Sean Stevens 4:15 PM
once you have an endpoint somewhere authenticated using DUO, it tries to use those credentials everywhere
Julian 4:15 PM
Huh. OK.
Sean Stevens 4:15 PM
I'll take a look
4:18
the one Rob sent da262cbf-6d04-11e5-ba46-22000b92c6ec is the ncsa#terra endpoint. The one David sent is the shared endpoint, Terraref, shared from ncsa#terra. It's top level directory is /terraref/sites
Julian 4:22 PM
So, I should be able to use the one David sent me for using as the origin of the tar files?(I can probably answer this myself by logging in to the server and checking.)
4:25
Ah. No. Doesn't look like it.
4:26
/terraref/users/scratch/archive_work/
Sean Stevens 4:26 PM
tar files are under /terraref/users/scratch... so it won't be visible in the one David sent. If we changed where the tar files got generate, that one could be used.
4:26
yeah
Julian 4:26 PM
Snap.
4:26
That's what I was wondering. OK.
New
Sean Stevens 4:27 PM
if that'd be easier, we could probably add a tars directory somewhere under the /terraref/sites/ and then just use that as the output space for the .tar and .md5sum files or something like that
Sean Stevens 4:38 PM
I created /terraref/sites/tar_archives feel free to adjust the scripts to point the md5sum and tar files there and you should be able to start transfers. Let me know if it's be helpful to do a run through of some of while we're both looking at it sometime

[julianpistorius]

The endpoint has stopped working:

$ globus endpoint activate 403204c4-6004-11e6-8316-22000b97daec
Endpoint is already activated. Activation expires at 2020-06-10 16:39:11+00:00

$ globus ls 403204c4-6004-11e6-8316-22000b97daec:/
Globus CLI Error: A Transfer API Error Occurred.
HTTP status:      502
request_id:       PCGlTAHXH
code:             ExternalError.DirListingFailed.LoginFailed
message:
                  Command Failed: Error (login)
                  Endpoint: Terraref (403204c4-6004-11e6-8316-22000b97daec)
                  Server: cg-hm02.ncsa.illinois.edu:2811
                  Message: Login Failed
                  ---
                  Details: globus_ftp_control: gss_init_sec_context failed\nGSS Major Status: Unexpected Gatekeeper or Service Name\nglobus_gsi_gssapi: Authorization denied: The name of the remote entity (/DC=org/DC=incommon/C=US/ST=Illinois/L=Urbana/O=University of Illinois/OU=NCSA/CN=cg-hm02.ncsa.illinois.edu), and the expected name for the remote entity (/DC=org/DC=incommon/C=US/ST=IL/L=Urbana/O=University of Illinois/OU=NCSA/CN=cg-hm02.ncsa.illinois.edu) do not match\n

A: /DC=org/DC=incommon/C=US/ST=Illinois
B: /DC=org/DC=incommon/C=US/ST=IL

[julianpistorius]

The endpoint started working again. Sean said:

There was a new certificate that went into place, but it looks like the Subject DN was generated with Illinois instead of IL. On the Globus side it will capture that when the server is added to the endpoint. Had to reset that on the Globus side. Should be happy now.

[julianpistorius]

Achievement unlocked: Successfully transferred test files from NCSA to TACC.

[julianpistorius]

Update from Sean about deletion policy:

I did talk to Rob. Sounds like he's still in discussions regarding deletion.

[julianpistorius]

Instructions condensed from conversation with Sean Stevens:

https://hackmd.io/@julianp-diag/SkKtF09TI

[julianpistorius]

Did not get to actual batching and transferring. Follow-up issue: #16