Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delegate full granule query to earthaccess, keep smart subsetting logic in icepyx #575

Open
weiji14 opened this issue Aug 20, 2024 · 2 comments
Labels
Discussion - feedback solicited This is a generic issue post, and feedback is requested to move forward with implementation. enhancement New feature or request longer_contribution Issues that will take more than an afternoon to implement

Comments

@weiji14
Copy link
Member

weiji14 commented Aug 20, 2024

To reduce the confusion of ICESat-2 users in choosing whether to use icepyx or earthaccess to search for ATLAS products, this issue attempts to discuss a path forward to have proper separation of concerns in both libraries. Talking to @JessicaS11 and others at the 2024 UW Hackweek, we propose to change the 'search' workflow to:

  1. For 80% of the users who wants an easy entrypoint -> Delegate full granule search and download to earthaccess. Specifically use earthaccess to return the Granule IDs from the NSDIC DAAC, and connect/open that (ideally cloud-hosted) dataset via fsspec.
  2. For 20% of the users who want to do more advanced search -> Use earthaccess to look for the Granule IDs, and icepyx (as a plugin/extension) handles the smart open_subset logic.
    • a) Transition phase icepyx=v1.x - Have the variable subsetting logic remain in icepyx only, without integration into earthaccess. Users use this interface with a mix of earthaccess and icepyx
    • b) Breaking phase icepyx=v2.x - Allow users to install icepyx as an extension of earthaccess. Users will interface directly using earthaccess.smart_open, xref Develop a plugin interface to add new downloaders nsidc/earthaccess#328.

Taking a stab this week as part of ICESAT-2HackWeek#5

image

@weiji14 weiji14 added enhancement New feature or request Discussion - feedback solicited This is a generic issue post, and feedback is requested to move forward with implementation. longer_contribution Issues that will take more than an afternoon to implement labels Aug 20, 2024
@mfisher87
Copy link
Member

💯

@weiji14
Copy link
Member Author

weiji14 commented Aug 23, 2024

Implementation-wise, I'm gonna try to increase code coverage on query.py and granules.py in #581 first, and then refactor ipx.Query's internal logic to use earthaccess's functions/methods to get the granule list. Intent is to try to retain backwards compatibility as much as possible in icepyx=1.x, and then make any breaking changes in icepyx=2.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Discussion - feedback solicited This is a generic issue post, and feedback is requested to move forward with implementation. enhancement New feature or request longer_contribution Issues that will take more than an afternoon to implement
Projects
None yet
Development

No branches or pull requests

2 participants