Implications of using cloud-optimized formats on processing

[riverma]

How does the use of cloud-optimized formats for HLS input data products affect the PCM? Specifically, for a given input data product available in HDF5 format, that is also available in a cloud-optimized format (cloud-optimized geotiff) enabling pulling of only specific spatial subsets of a given tile, what are the implications to:

1. The input data volume of products to ingest
2. The number of HLS PGE jobs that need to be run on the PCM
3. The output data volume of products generated by the HLS PGE jobs

@hfattahi @collinss-jpl @hhlee445 - thoughts?

[hfattahi]

The input HLS data to our DSWX-HLS algorithm is already in Cloud Optimized Geotiff (COG) format. They are also multiple files (one file per band). So it will make it really easy to pull only the bands (files) that our algorithm needs. It turned out that the DSWX-HLS algorithm actually does not need all the bands of HLS. The HLS data on average are ~300 MB per tile. and We need to pull only ~ 160MB per tile, so right there we will have a significant reduction in the amount of data that we need to move to the instance that we want to run our SAS from.
Our outputs will be also COG format. And we are also going with multi-file (one file per band). The COG format will help to reduce the file size by turning the compression on.

As I mentioned the HLS data are relatively small. In my opinion for this product we don't need to pull spatial subsets of a given tile. We will pul the entire tile but we pull only the bands (i.e., files) that we need.

Let me know if that is clear or you need more info.

[hhlee445]

Can you tell us which bands you need? PCM wants to make sure that we download those bands.

[hfattahi]

@gshiroma could you please clarify the exact bands that we need for DSWX-HLS algorithm?

[gshiroma]

For L30 (Landsat) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B05, B06, B07, and Fmask

For S30 (Sentinel) HLS products, we need layers (6 reflectance bands + Q/A mask): B02, B03, B04, B8A, B11, B12, and Fmask

[riverma]

Thank you @gshiroma @hfattahi

[riverma]

Hi @hfattahi - thank you for that clarification.

Is the following summary accurate?

• DSWX-HLS algorithm will need a fraction (b/B_total) of the total number of band files available (B_total) from HLS
• DSWX-HLS algorithm will require an ingest of (160MB/300MB) fraction of the bytes per tile
• Each DSWX-HLS algorithm PCM job can process based on a single band and single tile combination (thus allowing for horizontal scaling)

If so, here's what I'm getting as implications:

• The input data volume of products to ingest: (b/B_total) * (160MB/300MB) * num_of_tiles_to_process
• The number of HLS PGE jobs that need to be run on the PCM: (b/B_total) * num_of_tiles_to_process
• The output data volume of products generated by the HLS PGE jobs: (b/B_total) * (output_file_size_per_band_file) * num_of_tiles_to_process

[hfattahi]

Hi @hfattahi - thank you for that clarification.

Is the following summary accurate?

• DSWX-HLS algorithm will need a fraction (b/B_total) of the total number of band files available (B_total) from HLS
• DSWX-HLS algorithm will require an ingest of (160MB/300MB) fraction of the bytes per tile
• Each DSWX-HLS algorithm PCM job can process based on a single band and single tile combination (thus allowing for horizontal scaling)

If so, here's what I'm getting as implications:

• The input data volume of products to ingest: (b/B_total) * (160MB/300MB) * num_of_tiles_to_process
• The number of HLS PGE jobs that need to be run on the PCM: (b/B_total) * num_of_tiles_to_process
• The output data volume of products generated by the HLS PGE jobs: (b/B_total) * (output_file_size_per_band_file) * num_of_tiles_to_process

The processing unit for the data system should be HLS tile. Each tile is max 300 MB. We don't need all bands of the tile. We only need a subset of bands which approximately sums to ~160 MB. So the math is simply the following:

The input data volume of HLS product to ingest to generate one DSWX-HLS product: ~160 MB (excluding ancillary data)
The input data volume of HLS products to ingest in total across many instances: ~160 MB * num_of_tiles_to_process
The number of HLS PGE jobs that need to be run on the PCM: num_of_tiles_to_process
The output data volume of products generated by the HLS PGE jobs: output_file_size_per_band_file * num_of_tiles_to_process

There will be additional ancillary products to ingest (e.g, DEM and landcover map) which are not accounted for above.

[riverma]

Hi @hfattahi -

Thanks for that thorough explanation. That clears things up!

Compared to the non-COG use case, looks like we're getting a 46% savings in input data volume storage, the same number of jobs to process in the PCM, and a reduced output file size of some percentage (TBD). If you happen to have the expected output file size handy (vs non-COG), that'd be great to know too.