Whether sub-packages like global_time_series belong in zppy or their own spaces

[xylar]

We were hijacking #640 to have this discussion but it seems worth giving its own space.

A quick summary:

• @forsyth2 was concerned that adding significant amounts of code to support the pcmdi_metrics package (PMP) might balloon into essentially a new sub-package of zppy.
• This concern came from his experience with the global_time_series task, which effectively acts as a separate package that lives within zppy
• @xylar raised concerns about this situation and the need for global_time_series and other "ghost" packages to somehow specify their dependencies and constraints.
• @chengzhuzhang was not convinced that having global_time_series be a standalone package was a good idea, mostly because the potential maintenance overhead.
• She also felt like only zppy would use this package, as it is closely tied to other zppy capabilities.
• She felt that an important capability of zppy is to closes any gaps that other diagnostics packages won't cover but E3SM requires. The zppy-PMP workflow is an example of this.
• @rljacob pointed out that maybe the issue was that global_time_series is currently poorly maintained because its buried inside zppy. The cost of making it a separate package is offset by the benefit of better testing.
• @xylar felt like zppy shouldn't become a kitchen-sink package, trying to fill in all the gaps that E3SM needs.
• @chengzhuzhang still felt that global_time_series doesn't have much use beyond zppy. So maybe it is possible to make it more modular and testable in zppy before considering packaging it as a stand-alone package.

[xylar]

I would enjoy discussing this further at Monday's EZ meeting or whenever the appropriate time is.

[chengzhuzhang]

@xylar thanks for creating this discussion item here! I was also going to suggest a meeting. We can cover this specific topic, as well as a discussion with over all zppy's testing system would be really great.

Either at Monday's EZ meeting or Thursday's IG meeting would work. Let's see if @rljacob is available to join for either of the time slots.

[xylar]

I feel that it is often appropriate to separate distinct tools even one tool is exclusively used by the other. In that sense, it feels to me like zppy should go back to being a tool that launches other diagnostic tools and maintains that workflow, but all the tools should be external to zppy.

I acknowledge that many of these tools are external to E3SM (ILAMB, PMP, etc.) and require adaptation to meet E3SM's needs. But it feels like a maintenance nightmare (particularly for @forsyth2) that all these adapters are ending up in zppy. It seems like each should rightly be its own, small package. Each likely has its own dependencies and needs to be updated and tested on its own schedule, largely determined by the external tools it interfaces with (but also the E3SM-Unified schedule).

I also acknowledge that I keep going back and forth on these things in my own tools. I am struggling with the trade-off between the convenience of a common infrastructure if we put both model testing and analysis in our Polaris tool and the fact that that leaves analysis tools mixed up with regression tests. Sometimes, it seems best to integrate everything and at other times I think the thing has become too big and distinct pieces need to be pulled out.

I don't think it's necessary that those distinct pieces are useful across more than one other tool for it to be useful to separate them out. Sometimes, it is worth having distinct repos just for clarity. But in my work on conda-forge, I have seen both things happen -- a bunch of distinct repos get combined into one because the maintenance became unwieldy and a mega-repo that finally gets broken into distinct tools.

[TonyB9000]

My Two Cents: I tend to agree with Xylar - but I would go with 2 repos:

One would include ALL subordinate or independent functionalities, separate modules to house or include ILAMB, PMP, etc, (the "sub-tools repo, irrespective of E3SM or external), each able to be wrapped in a simple "main" driver for stand-alone application (or unit testing), but each amenable to be imported elsewhere (e.g. zppy). Code updates to one should never incur merge conflicts with others.

The other repo would have zppy, stripped of all functionality except for its ability to coordinate a sequence or ensemble of operations, plus any purely common functionality.

This would make clear the boundary between the zppy configuration and housing operations, and the tool-functionalities it can manage. New tools could be added to the subordinate suite of tools, with clear rules for the zppy code interface to accommodate.

[chengzhuzhang]

I thought about the same idea over the weekend. If we opted to have a new repo, we can create one utility repo to house all external capabilities, i.e. global_time_series (I think the scope is to compute annual mean, generate plots and viewers), the PMP utility codes, etc.

[forsyth2]

Action items from today's meeting:

• Pull the de facto / pseudo / sub packages (e.g., Global Time Series, PMP) into the new zppy-interfaces package, leaving zppy to be a workflow manager.
  • Note this will affect the implementations of Create global time series Viewers #616 and Workflow for ZPPY- PCMDI for E3SM diagnostics (with fixes on issues found in #624)  #640.
  • @rljacob has created this new repo, here: https://github.com/E3SM-Project/zppy-interfaces
• Code in zppy/templates should be organized into distinct directories:
• .bash jinja2 templates should stay in zppy.
• Campaign .cfg files and default.ini should go in a defaults directory.
• The global_time_series .py files and the 3 subdirectories (e3sm_diags, e3sm_to_cmip, ilamb) should move to the new zppy-interfaces package.
• Build environment should be up-to-date with modern Python standards. E.g., using pyproject.toml rather than setup.py. @xylar and @altheaden can help with this effort.

Longer-term action items to consider in the future:

• Automate current weekly manual testing process (zppy weekly testing log #634)
• Avoid duplicate testing (e.g., testing code changes in both the the Diags testing suite and the zppy testing suite) by running zppy with Diags in the latest Unified release.
• May want to still test Diags on latest diags' main branch quarterly/monthly to check for unexpected changes.
• Create dummy objects or mocks for testing zppy, so as to make it less dependent on outside data.
• Move the bulk of bash code into python. It makes sense to leave job-launching code in bash though. This will ease debugging and testing.

These changes will be beneficial for a number of reasons:

• More modularized testing. zppy-interfaces can have more unit testing and development there can proceed largely without interacting with zppy. zppy testing by contrast can focus more squarely on running correctly set up jobs (rather than duplicate testing the packages it calls).
• Clearer environment usage. zppy-interfaces will have a distinct development environment than zppy, with different dependencies. These dependencies have been hidden in the current implementation of zppy.
• Clearer meaning of releases. A new patch/minor/major release will clearly be for new workflow improvements (zppy) or for new "last mile" code [to get desired output from the underlying packages] (zppy-interfaces)