OPERA / LPDAAC Interface

[maseca]

Currently OPERA PCM uses HTTPS to stream files from LPCLOUD to our S3 buckets. Ideally, we would like to perform direct S3 to S3 file transfer, but this is not currently possible as new credentials are required which would have read access to LPDAAC's buckets and write access to our buckets. The existing Earthdata/CMR interface allows for AWS token generation with read access from LPCLOUD but said token does not provide write access to our buckets. This could potentially be resolved with either a new IAM role or via generating a new specialized token that also includes write access on our buckets. Either solution would require some collaboration with LPDAAC in order to be implemented/configured.

[riverma]

Thanks for this write-up @maseca - so to summarize, you're requesting a conversation with LP.DAAC to enable better S3-to-S3 transfers via:

• Setup a new IAM role with read access to LP.DAAC S3 holdings and write access to OPERA SDS S3 holdings
• A new "special" token generated from Earthday / CMR with additional write access capability for OPERA SDS S3 buckets

The first option seems pretty straightforward to me. I'm not quite understanding the latter though - who would generate the token? Does this require a code change, and if so for which software?

[maseca]

I think I had a misunderstanding of how the temporary AWS credentials work. The IAM role is likely the only option moving forward.

Also noting here that storing S3 URIs from queries in an ES index to be transferred later by scalable workers was discussed during the WG meeting today. Need to determine if S3 locations are subject to change on the provider side.

[riverma]

@maseca - given some updates from conversations with the DAACs, it sounds like the IAM role option is going to require EOSDIS review.

Thus, we'll have to stick with our nominal approach at least for now. Is the following diagram a correct representation of our nominal approach? If not, please make some modifications to the

sequenceDiagram
autonumber

OPERA_PCM-->>CMR: Query for products for a given time range
CMR-->>OPERA_PCM: List of product metadata matches, including S3 URIs
OPERA_PCM-->>CMR: Request for AWS temporary creds
CMR-->>OPERA_PCM: AWS Temporary creds
OPERA_PCM-->>LPCLOUD: Request for downloading S3 URIs, via boto3 library in parallel
LPCLOUD-->>OPERA_PCM: Transfer S3 objects to PCM via boto3 in parallel

Loading

[riverma]

Thanks for the input @maseca 👍