How do I run the Globus unit test?

[forsyth2]

The Globus unit test is run with python -u -m unittest tests/test_globus.py.

First, you need to make sure the Globus endpoints are activated. Go to https://www.globus.org/. Log in with your NERSC credentials. Activate the following Globus endpoints using the corresponding credentials:

• Chrysalis
• Compy
• Perlmutter/NERSC

Now you can run the test. Unfortunately, unexpected errors often appear with this test. These were introduced by the new (~Fall 2023) requirement of Globus consents.

See #298 -- errors include this: 'ConsentRequired', 'Missing required data_access consent'. A workaround was provided in that issue, but it was cumbersome for users. Therefore, automated consent was implemented in #304.

Unfortunately, automated runs of zstash (e.g., when zstash is run via a unit test, when zstash is run from a bash script) tend to not send the consent prompt to users, causing hangs. At this point, it's simplest to try to provide the consent manually. E3SM-Project/e3sm_data_docs#34 (comment) and #326 (comment) provide some examples of this.

# Set up conda environment. If testing the latest Unified on Chrysalis, that would be:
$ source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
$ cd <a directory besides the `zstash` repo>
$ mkdir zstash_demo; echo 'file0 stuff' > zstash_demo/file0.txt
# Replace the endpoint below with the one you're trying to access.
# Chrysalis: 15288284-7006-4041-ba1a-6b52501e49f1
# Compy: 68fbd2fa-83d7-11e9-8e63-029d279f7e24
# Perlmutter: 6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
# Unit test's endpoint: 6c54cade-bde5-45c1-bdea-f4bd71dba2cc
$ zstash create --hpss=globus://6c54cade-bde5-45c1-bdea-f4bd71dba2cc/~/manual_run zstash_demo
$ cd <zstash directory>
$ python -u -m unittest tests/test_globus.py 
# The test should now run. Note that it's common for there to be a short hang, but then the test will finish.

Note that a hang or error may in fact be caused by a different issue. E.g., in #299, the hang was caused by a missing endpoint (the Cori endpoint had been deleted and the Perlmutter endpoint needed to be used instead). In #326, the TransferAPIError was also caused by needing to replace one endpoint with another.

Lastly, #322 (and related discussion #323) indicate one possible for solution of a 'ClientError.AuthenticationFailed', 'No credentials supplied' error is to run rm ~/.globus-native-apps.cfg.

[forsyth2]

Interestingly, sometimes the unit test will send the consent code prompt to the user but still hang. In this case, it is best to cancel the test (Control C) and try the manual solution above.

[forsyth2]

If I run rm ~/.globus-native-apps.cfg and then the manual solution right away, it in fact prompts for consents twice. Thus, it seems the automated test will send the first prompt to the user, but not the second.