From cf39b51715fbc6d10f483a5162bfc4cc9c567375 Mon Sep 17 00:00:00 2001 From: Brett Date: Mon, 22 Jul 2024 17:21:40 -0400 Subject: [PATCH 01/16] WIP: library docs --- docs/source/index.rst | 2 + docs/source/model_library.rst | 365 ++++++++++++++++++++++++++++++++++ 2 files changed, 367 insertions(+) create mode 100644 docs/source/model_library.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 91aca59c..c4a6d24a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -5,6 +5,8 @@ Welcome to stpipe's documentation! :maxdepth: 2 :caption: Contents: + model_library.rst + API === diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst new file mode 100644 index 00000000..46ed00b1 --- /dev/null +++ b/docs/source/model_library.rst @@ -0,0 +1,365 @@ +.. _model_library: + +Model Library +============= + +`AbstractModelLibrary` is a container designed to allow efficient processing of +collections of `AbstractDataModel` instances created from an association. + +`AbstractModelLibrary` is an ordered collection (like a `list`) but provides: + +- access to association metadata: `AbstractModelLibrary.asn` +- grouping API: `AbstractModelLibrary.group_indices` and `AbstractModelLibrary.group_names` +- compatibility with `Step` and `Pipeline` runs +- a consistent indexing API that is the same for "in memory" and "on disk" libraries + + +.. _library_association: + +Loading an association +---------------------- + +Most commonly an instance will be created from an association file: + +.. code-block:: pycon + + >>> library = ModelLibrary("my_asn.json") + +.. NOTE:: + For an association that contains a ``group_id`` for each member, + creating a library will not read any models. + +.. _library_borrowing_and_shelving: + +Borrowing and shelving models +----------------------------- + +Interacting with an `AbstractModelLibrary` involves "borrowing" and "shelving" +models, both of which must occur during a ``with`` statement (while the library +is "open"): + +.. code-block:: pycon + + >>> with library: + ... model = library.borrow(0) + ... # do stuff with the model... + ... library.shelve(model) + +Iteration is also supported (but don't forget to return your models!). + +.. code-block:: pycon + + >>> with library: + ... for model in library: + ... # do stuff with the model... + ... library.shelve(model) + + +.. _library_on_disk: + +On Disk Mode +------------ + +For large associations (like those larger than memory) it is important +that the library avoid reading all models at once. The borrow/shelve API +above maps closely to the loading/saving of input (or tempoary) files +containing the models. + +.. code-block:: pycon + + >>> library = ModelLibrary("my_big_asn.json", on_disk=True) + >>> with library: + ... model = library.borrow(0) # the input file for model 0 is loaded + ... library.shelve(model) # a temporary file for model 0 is written + +.. NOTE:: + In the above example, a temporary file was created for model 0. At no + point will the library overwrite the input file. + +If model is not modified during the time it's borrowed (for example if the +`model.dq` array was read, but not modified). It is helpful to tell the +library that the model was not modified. + +.. code-block:: pycon + + >>> with library: + ... model = library.borrow(0) # the input file for model 0 is loaded + ... # do some read-only stuff with the model + ... library.shelve(model, modify=False) # No temporary file will be written + +This can dramatically reduce the number of times a file is written saving +on both disk space and the time required to write. + + +.. _library_map_function: + +Map function +------------ + +Let's say you want to get the ``meta.filename`` attribute for all models +in a library. The above "open", "borrow", "shelve" pattern can be quite +verbose. Instead, the helper method `AbstractModelLibrary.map_function` +can be used to generate an iterator that returns the result of a function +applied to each model in the library: + +.. code-block:: pycon + + >>> def get_model_name(model, index): + ... return model.meta.filename + >>> + >>> filenames = list(library.map_function(get_model_name)) + +.. NOTE:: + `AbstractModelLibrary.map_function` does not require an open library + and will handle opening, borrowing, shelving and closing for you. + + +.. _library_grouping: + +Grouping +-------- + +Grouping also doesn't require an open library (as all grouping is +performed on the association metadata). + +.. code-block:: pycon + + >>> print(f"All group names: {library.group_names}") + >>> group_index_map = library.group_indices + >>> for group_name in group_index_map: + ... print(f"\tModel indices for {group_name}: {group_index_map[group_name]}") + +.. WARNING:: + Although `AbstractModelLibrary.group_names` and + `AbstractModelLibrary.group_indices` do not require an open library, + any "borrows" using the indices do. Be sure to open the library before + trying to borrow a model. + +`AbstractModelLibrary.asn` provides read-only access to the association data. + +.. code-block:: pycon + + >>> library.asn["products"][0]["name"] + >>> library.asn["table_name"] + + +.. _library_troubleshooting: + +Troubleshooting +=============== + +.. _library_closed_library_error: + +ClosedLibraryError +------------------ + +.. code-block:: pycon + + >>> model = library.borrow(0) + + ClosedLibraryError: ModelLibrary is not open + +The library must be "open" (used in a ``with`` statement) before +a model can be borrowed. This is important for keeping track of +which models were possibly modified. + +This error can be avoided by "opening" the library before calling +`AbstractModelLibrary.borrow` (and being sure to call +`AbstractModelLibrary.shelve`, more on that below): + +.. code-block:: pycon + + >>> with library: + ... model = library.borrow(0) + ... library.shelve(model) + +.. _library_borrow_error: + +BorrowError +----------- + +.. code-block:: pycon + + >>> with library: + ... model = library.borrow(0) + ... # do stuff with the model + ... # forget to shelve it + + BorrowError: ModelLibrary has 1 un-returned models + +Forgetting to `AbstractModelLibrary.shelve` a borrowed model will result in an +error. This is important for keeping track of model modifications and is +critical when the library uses temporary files to keep models out of memory. + +This error can be avoided by making sure to `AbstractModelLibrary.shelve` all +borrowed models: + +.. code-block:: pycon + + >>> with library: + ... model = library.borrow(0) + ... library.shelve(model) + +Attempting to "double borrow" a model will also result in a `BorrowError`. + +.. code-block:: pycon + + >>> with library: + ... model_a = library.borrow(0) + ... model_b = library.borrow(0) + + BorrowError: Attempt to double-borrow model + +This check is also important for the library to track model modifications. The +error can be avoided by only borrowing each model once (it's ok to borrow +more than one model if they are at different positions in the library). + +`BorrowError` exceptions can also be triggered when trying to replace +a model in the library. + +.. code-block:: pycon + + >>> with library: + ... library.shelve(some_other_model) + + BorrowError: Attempt to shelve an unknown model + +Here the library does not know where to shelve ``some_other_model`` (since +the ``some_other_model`` wasn't borrowed from the library). To replace +a model in the library you will need to first borrow the model at the index +you want to use and provide the index to the call to +`AbstractModelLibrary.shelve`. + +.. code-block:: pycon + + >>> with library: + ... library.borrow(0) + ... library.shelve(some_other_model, 0) + +Forgetting to first borrow the model at the index will also produce a +`BorrowError` (even if you provide the index). + +.. code-block:: pycon + + >>> with library: + ... library.shelve(some_other_model, 0) + + BorrowError: Attempt to shelve model at a non-borrowed index + +.. _library_implementing_a_subclass: + +Implementing a subclass +======================= + +Several methods are abstract and will need implementations: + +- Methods used by stpipe: + + - `AbstractModelLibrary.crds_observatory` + +- Methods used by `AbstractModelLibrary` + + - `AbstractModelLibrary._datamodels_open` + - `AbstractModelLibrary._load_asn` + - `AbstractModelLibrary._filename_to_group_id` + - `AbstractModelLibrary._model_to_group_id` + +It's likely that a few other methods might require overriding: + +- `AbstractModelLibrary._model_to_fileaname` +- `AbstractModelLibrary._assign_member_to_model` + +Consult the docstrings (and base implementations) for more details. + +It may also be required (depending on your usage) to update +`Step._datamodels_open` to allow stpipe to open and inspect an +`AbstractModelLibrary` when provided as a `Step` input. + +.. _library_developer_documentation: + +Developer Documentation +======================= + +What follows are note primarily aimed towards developers and +maintainers of `AbstractModelLibrary`. This section might be useful +to provide context to users but shouldn't be necessary for a user +to effectively use `AbstractModelLibrary`. + +.. _library_motivation: + +Motivation +---------- + +The development of `AbstractModelLibrary` was largely motivated by +the need for a container compatible with stpipe machinery +that would allow passing "on disk" models between steps. Existing +containers (when used in "memory saving" modes) were not compatible +with stpipe. These containers also sometimes allowed input files +to be overwritten. It was decided that a new container would be +developed to address these and other issues. This would allow +gradual migration for pipeline code where specific steps and pipelines +could update to `AbstractModelLibrary` while leaving the existing +container unchanged for other steps. + +A survey of container usage was performed with a few key findings: + +- Many uses could be replaced by simpler containers (lists) +- When loaded from an association, the container size never changed +- The order of models was never changed +- Needs various methods for stpipe +- Several steps implemented different memory optimizations + and had significant complexity added to deal with containers + that sometimes returned filenames and sometimes returned models + +Additionally, pipelines and steps may be expected to handle large +volumes of input data. For one example, consider a pipeline +responsible for generating a mosaic of a large number of input imaging +observations. As the size of the input data approaches (and exceeds) +the available memory it is critical that the pipeline, step, and +container code never read and hold all input data in memory. + +.. _library_design_priciples: + +Design principles +----------------- + +The high level goals of `AbstractModelLibrary` are: + +- Replace many uses of existing containers, focusing on areas + where large data is expected. +- Implement a minimal API that can be later expanded as needs + arise. +- Provide a consistent API for "on disk" and "in memory" modes + so step code does not need to be aware of the mode. +- Support all methods required by stpipe to allow a "on disk" + container to pass between steps. + +Most of the core functionality is public and described in the above +user documentation. What follows will be description of other parts +of the API (most private) and internal details. + +One core issue is how can the container know when to load and +save models (to temporary files) if needed? With a typical list +``__getitem__`` can map to load but what will map to save? +Initial prototypes used ``__setitem__`` which lead to some confusion +amongst reviewers. Treating the container like a list also +leads to expectations that the container also support +``append`` ``extend`` and other API that is unnecessary (as determined +in the above survey) and would be difficult to implement in a way that +would keep the container association information and model information +in sync. + +.. _library_integration_with_stpipe: + +Integration with stpipe +----------------------- + +An `AbstractModelLibrary` may interact with stpipe when used as an +input or output for a `Step`. + +- as a `Step` input where `AbstractModelLibrary.get_crds_parameters` and + `AbstractModelLibrary.crds_observatory` will be used (sometimes with + a limited model set, including only the first member of the input + association). +- as a `Step` output where `AbstractModelLibrary.finalize_result` will + be used. From 86946bcc3114c201b47d522c8d34701e030fd418 Mon Sep 17 00:00:00 2001 From: Brett Date: Mon, 29 Jul 2024 11:53:45 -0400 Subject: [PATCH 02/16] set default_role to obj --- docs/source/api.rst | 1 + docs/source/conf.py | 3 ++ docs/source/model_library.rst | 82 +++++++++++++++++------------------ src/stpipe/datamodel.py | 2 +- src/stpipe/pipeline.py | 10 ++--- src/stpipe/step.py | 48 ++++++++++---------- 6 files changed, 75 insertions(+), 71 deletions(-) diff --git a/docs/source/api.rst b/docs/source/api.rst index 7f200682..6fd74cae 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -4,4 +4,5 @@ stpipe API .. automodapi:: stpipe .. automodapi:: stpipe.library + .. automodapi:: stpipe.datamodel diff --git a/docs/source/conf.py b/docs/source/conf.py index 877b4cfa..027b5fd7 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -55,5 +55,8 @@ nitpicky = True nitpick_ignore = [] +# Set the default role for all single backtick annotations +default_role = "obj" + intersphinx_mapping = {} intersphinx_mapping["python"] = ("https://docs.python.org/3", None) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 46ed00b1..abe938da 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -3,14 +3,14 @@ Model Library ============= -`AbstractModelLibrary` is a container designed to allow efficient processing of -collections of `AbstractDataModel` instances created from an association. +`~stpipe.library.AbstractModelLibrary` is a container designed to allow efficient processing of +collections of `~stpipe.datamodel.AbstractDataModel` instances created from an association. -`AbstractModelLibrary` is an ordered collection (like a `list`) but provides: +`~stpipe.library.AbstractModelLibrary` is an ordered collection (like a `list`) but provides: -- access to association metadata: `AbstractModelLibrary.asn` -- grouping API: `AbstractModelLibrary.group_indices` and `AbstractModelLibrary.group_names` -- compatibility with `Step` and `Pipeline` runs +- access to association metadata: `~stpipe.library.AbstractModelLibrary.asn` +- grouping API: `~stpipe.library.AbstractModelLibrary.group_indices` and `~stpipe.library.AbstractModelLibrary.group_names` +- compatibility with `~stpipe.step.Step` and `~stpipe.pipeline.Pipeline` runs - a consistent indexing API that is the same for "in memory" and "on disk" libraries @@ -34,7 +34,7 @@ Most commonly an instance will be created from an association file: Borrowing and shelving models ----------------------------- -Interacting with an `AbstractModelLibrary` involves "borrowing" and "shelving" +Interacting with an `~stpipe.library.AbstractModelLibrary` involves "borrowing" and "shelving" models, both of which must occur during a ``with`` statement (while the library is "open"): @@ -77,7 +77,7 @@ containing the models. point will the library overwrite the input file. If model is not modified during the time it's borrowed (for example if the -`model.dq` array was read, but not modified). It is helpful to tell the +``model.dq`` array was read, but not modified). It is helpful to tell the library that the model was not modified. .. code-block:: pycon @@ -98,7 +98,7 @@ Map function Let's say you want to get the ``meta.filename`` attribute for all models in a library. The above "open", "borrow", "shelve" pattern can be quite -verbose. Instead, the helper method `AbstractModelLibrary.map_function` +verbose. Instead, the helper method `~stpipe.library.AbstractModelLibrary.map_function` can be used to generate an iterator that returns the result of a function applied to each model in the library: @@ -110,7 +110,7 @@ applied to each model in the library: >>> filenames = list(library.map_function(get_model_name)) .. NOTE:: - `AbstractModelLibrary.map_function` does not require an open library + `~stpipe.library.AbstractModelLibrary.map_function` does not require an open library and will handle opening, borrowing, shelving and closing for you. @@ -130,12 +130,12 @@ performed on the association metadata). ... print(f"\tModel indices for {group_name}: {group_index_map[group_name]}") .. WARNING:: - Although `AbstractModelLibrary.group_names` and - `AbstractModelLibrary.group_indices` do not require an open library, + Although `~stpipe.library.AbstractModelLibrary.group_names` and + `~stpipe.library.AbstractModelLibrary.group_indices` do not require an open library, any "borrows" using the indices do. Be sure to open the library before trying to borrow a model. -`AbstractModelLibrary.asn` provides read-only access to the association data. +`~stpipe.library.AbstractModelLibrary.asn` provides read-only access to the association data. .. code-block:: pycon @@ -164,8 +164,8 @@ a model can be borrowed. This is important for keeping track of which models were possibly modified. This error can be avoided by "opening" the library before calling -`AbstractModelLibrary.borrow` (and being sure to call -`AbstractModelLibrary.shelve`, more on that below): +`~stpipe.library.AbstractModelLibrary.borrow` (and being sure to call +`~stpipe.library.AbstractModelLibrary.shelve`, more on that below): .. code-block:: pycon @@ -187,11 +187,11 @@ BorrowError BorrowError: ModelLibrary has 1 un-returned models -Forgetting to `AbstractModelLibrary.shelve` a borrowed model will result in an +Forgetting to `~stpipe.library.AbstractModelLibrary.shelve` a borrowed model will result in an error. This is important for keeping track of model modifications and is critical when the library uses temporary files to keep models out of memory. -This error can be avoided by making sure to `AbstractModelLibrary.shelve` all +This error can be avoided by making sure to `~stpipe.library.AbstractModelLibrary.shelve` all borrowed models: .. code-block:: pycon @@ -200,7 +200,7 @@ borrowed models: ... model = library.borrow(0) ... library.shelve(model) -Attempting to "double borrow" a model will also result in a `BorrowError`. +Attempting to "double borrow" a model will also result in a `~stpipe.library.BorrowError`. .. code-block:: pycon @@ -214,7 +214,7 @@ This check is also important for the library to track model modifications. The error can be avoided by only borrowing each model once (it's ok to borrow more than one model if they are at different positions in the library). -`BorrowError` exceptions can also be triggered when trying to replace +`~stpipe.library.BorrowError` exceptions can also be triggered when trying to replace a model in the library. .. code-block:: pycon @@ -228,7 +228,7 @@ Here the library does not know where to shelve ``some_other_model`` (since the ``some_other_model`` wasn't borrowed from the library). To replace a model in the library you will need to first borrow the model at the index you want to use and provide the index to the call to -`AbstractModelLibrary.shelve`. +`~stpipe.library.AbstractModelLibrary.shelve`. .. code-block:: pycon @@ -237,7 +237,7 @@ you want to use and provide the index to the call to ... library.shelve(some_other_model, 0) Forgetting to first borrow the model at the index will also produce a -`BorrowError` (even if you provide the index). +`~stpipe.library.BorrowError` (even if you provide the index). .. code-block:: pycon @@ -255,25 +255,25 @@ Several methods are abstract and will need implementations: - Methods used by stpipe: - - `AbstractModelLibrary.crds_observatory` + - `~stpipe.library.AbstractModelLibrary.crds_observatory` -- Methods used by `AbstractModelLibrary` +- Methods used by `~stpipe.library.AbstractModelLibrary` - - `AbstractModelLibrary._datamodels_open` - - `AbstractModelLibrary._load_asn` - - `AbstractModelLibrary._filename_to_group_id` - - `AbstractModelLibrary._model_to_group_id` + - ``_datamodels_open`` + - ``_load_asn`` + - ``_filename_to_group_id`` + - ``_model_to_group_id`` It's likely that a few other methods might require overriding: -- `AbstractModelLibrary._model_to_fileaname` -- `AbstractModelLibrary._assign_member_to_model` +- ``_model_to_filename`` +- ``_assign_member_to_model`` Consult the docstrings (and base implementations) for more details. It may also be required (depending on your usage) to update -`Step._datamodels_open` to allow stpipe to open and inspect an -`AbstractModelLibrary` when provided as a `Step` input. +``stpipe.step.Step._datamodels_open`` to allow stpipe to open and inspect an +`~stpipe.library.AbstractModelLibrary` when provided as a `~stpipe.step.Step` input. .. _library_developer_documentation: @@ -281,16 +281,16 @@ Developer Documentation ======================= What follows are note primarily aimed towards developers and -maintainers of `AbstractModelLibrary`. This section might be useful +maintainers of `~stpipe.library.AbstractModelLibrary`. This section might be useful to provide context to users but shouldn't be necessary for a user -to effectively use `AbstractModelLibrary`. +to effectively use `~stpipe.library.AbstractModelLibrary`. .. _library_motivation: Motivation ---------- -The development of `AbstractModelLibrary` was largely motivated by +The development of `~stpipe.library.AbstractModelLibrary` was largely motivated by the need for a container compatible with stpipe machinery that would allow passing "on disk" models between steps. Existing containers (when used in "memory saving" modes) were not compatible @@ -298,7 +298,7 @@ with stpipe. These containers also sometimes allowed input files to be overwritten. It was decided that a new container would be developed to address these and other issues. This would allow gradual migration for pipeline code where specific steps and pipelines -could update to `AbstractModelLibrary` while leaving the existing +could update to `~stpipe.library.AbstractModelLibrary` while leaving the existing container unchanged for other steps. A survey of container usage was performed with a few key findings: @@ -323,7 +323,7 @@ container code never read and hold all input data in memory. Design principles ----------------- -The high level goals of `AbstractModelLibrary` are: +The high level goals of `~stpipe.library.AbstractModelLibrary` are: - Replace many uses of existing containers, focusing on areas where large data is expected. @@ -354,12 +354,12 @@ in sync. Integration with stpipe ----------------------- -An `AbstractModelLibrary` may interact with stpipe when used as an -input or output for a `Step`. +An `~stpipe.library.AbstractModelLibrary` may interact with stpipe when used as an +input or output for a `~stpipe.step.Step`. -- as a `Step` input where `AbstractModelLibrary.get_crds_parameters` and - `AbstractModelLibrary.crds_observatory` will be used (sometimes with +- as a `~stpipe.step.Step` input where `~stpipe.library.AbstractModelLibrary.get_crds_parameters` and + `~stpipe.library.AbstractModelLibrary.crds_observatory` will be used (sometimes with a limited model set, including only the first member of the input association). -- as a `Step` output where `AbstractModelLibrary.finalize_result` will +- as a `~stpipe.step.Step` output where `~stpipe.library.AbstractModelLibrary.finalize_result` will be used. diff --git a/src/stpipe/datamodel.py b/src/stpipe/datamodel.py index 7ec90b04..008e2aaa 100644 --- a/src/stpipe/datamodel.py +++ b/src/stpipe/datamodel.py @@ -57,7 +57,7 @@ def save(self, path, dir_path=None, *args, **kwargs): dir_path : str Directory to save to. If not None, this will override - any directory information in the `path` + any directory information in the ``path`` Returns ------- diff --git a/src/stpipe/pipeline.py b/src/stpipe/pipeline.py index 8be64a2d..70eaf00c 100644 --- a/src/stpipe/pipeline.py +++ b/src/stpipe/pipeline.py @@ -32,7 +32,7 @@ class Pipeline(Step): def __init__(self, *args, **kwargs): """ - See `Step.__init__` for the parameters. + See `~stpipe.step.Step` for the parameters. """ Step.__init__(self, *args, **kwargs) @@ -83,8 +83,8 @@ def _unskipped_steps(self): ] def get_ref_override(self, reference_file_type): - """Return any override for `reference_file_type` for any of the steps in - Pipeline `self`. OVERRIDES Step. + """Return any override for ``reference_file_type`` for any of the steps in + Pipeline ``self``. OVERRIDES Step. Returns ------- @@ -147,11 +147,11 @@ def get_config_from_reference(cls, dataset, disable=None, crds_observatory=None) Parameters ---------- - cls : `jwst.stpipe.step.Step` + cls : `stpipe.step.Step` Either a class or instance of a class derived from `Step`. - dataset : `jwst.datamodels.ModelBase` + dataset : `stpipe.datamodel.AbstractDataModel` A model of the input file. Metadata on this input file will be used by the CRDS "bestref" algorithm to obtain a reference file. diff --git a/src/stpipe/step.py b/src/stpipe/step.py index adf5cc3d..f956c454 100644 --- a/src/stpipe/step.py +++ b/src/stpipe/step.py @@ -131,18 +131,18 @@ def from_config_file(cls, config_file, parent=None, name=None): name : str, optional If provided, use that name for the returned instance. If not provided, the following are tried (in order): - - The `name` parameter in the config file + - The ``name`` parameter in the config file - The filename of the config file - The name of returned class Returns ------- step : Step instance - If the config file has a `class` parameter, the return - value will be as instance of that class. The `class` + If the config file has a ``class`` parameter, the return + value will be as instance of that class. The ``class`` parameter in the config file must specify a subclass of - `cls`. If the configuration file has no `class` - parameter, then an instance of `cls` is returned. + ``cls``. If the configuration file has no ``class`` + parameter, then an instance of ``cls`` is returned. Any parameters found in the config file will be set as member variables on the returned `Step` instance. @@ -180,7 +180,7 @@ def from_cmdline(args): Returns ------- step : Step instance - If the config file has a `class` parameter, the return + If the config file has a ``class`` parameter, the return value will be as instance of that class. Any parameters found in the config file will be set @@ -655,21 +655,21 @@ def call(cls, *args, **kwargs): Gets a config file from CRDS if one is available - To set configuration parameters, pass a `config_file` path or + To set configuration parameters, pass a ``config_file`` path or keyword arguments. Keyword arguments override those in the - specified `config_file`. + specified ``config_file``. - Any positional `*args` will be passed along to the step's - `process` method. + Any positional ``*args`` will be passed along to the step's + ``process`` method. Note: this method creates a new instance of `Step` with the given - `config_file` if supplied, plus any extra `*args` and `**kwargs`. + ``config_file`` if supplied, plus any extra ``*args`` and ``**kwargs``. If you create an instance of a Step, set parameters, and then use - this `call()` method, it will ignore previously-set parameters, as - it creates a new instance of the class with only the `config_file`, - `*args` and `**kwargs` passed to the `call()` method. + this ``call()`` method, it will ignore previously-set parameters, as + it creates a new instance of the class with only the ``config_file``, + ``*args`` and ``**kwargs`` passed to the ``call()`` method. - If not used with a `config_file` or specific `*args` and `**kwargs`, + If not used with a ``config_file`` or specific ``*args`` and ``**kwargs``, it would be better to use the `run` method, which does not create a new instance but simply runs the existing instance of the `Step` class. @@ -734,7 +734,7 @@ def search_attr(self, attribute, default=None, parent_first=False): Returns ------- value : obj - Attribute value or `default` if not found + Attribute value or default if not found """ if parent_first: try: @@ -765,7 +765,7 @@ def _precache_references(self, input_file): """ def get_ref_override(self, reference_file_type): - """Determine and return any override for `reference_file_type`. + """Determine and return any override for ``reference_file_type``. Returns ------- @@ -974,8 +974,8 @@ def save_model( default would be. force : bool - Regardless of whether `save_results` is `False` - and no `output_file` is specified, try saving. + Regardless of whether ``save_results`` is `False` + and no ``output_file`` is specified, try saving. components : dict Other components to add to the file name. @@ -1149,7 +1149,7 @@ def _datamodels_open(cls, init, **kwargs): def open_model(self, init, **kwargs): """Open a datamodel - Primarily a wrapper around `DataModel.open` to + Primarily a wrapper around ``DataModel.open`` to handle `Step` peculiarities Parameters @@ -1174,7 +1174,7 @@ def open_model(self, init, **kwargs): def make_input_path(self, file_path): """Create an input path for a given file path - If `file_path` has no directory path, use `self.input_dir` + If ``file_path`` has no directory path, use ``self.input_dir`` as the directory path. Parameters @@ -1187,7 +1187,7 @@ def make_input_path(self, file_path): Returns ------- full_path : str or obj - File path using `input_dir` if the input + File path using ``input_dir`` if the input had no directory path. """ full_path = file_path @@ -1280,7 +1280,7 @@ def update_pars(self, parameters): """Update step parameters Only existing parameters are updated. Otherwise, new keys - found in `parameters` are ignored. + found in ``parameters`` are ignored. Parameters ---------- @@ -1289,7 +1289,7 @@ def update_pars(self, parameters): Notes ----- - `parameters` is presumed to have been produced by the + ``parameters`` is presumed to have been produced by the `Step.get_pars` method. As such, the "steps" key is treated special in that it is a dict whose keys are the steps assigned directly as parameters to the current step. This is standard From d28487aff9708c252b3cbd5984d44619f4170a71 Mon Sep 17 00:00:00 2001 From: Brett Date: Mon, 29 Jul 2024 17:39:02 -0400 Subject: [PATCH 03/16] add future directions --- docs/source/model_library.rst | 89 +++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index abe938da..5317fd6a 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -363,3 +363,92 @@ input or output for a `~stpipe.step.Step`. association). - as a `~stpipe.step.Step` output where `~stpipe.library.AbstractModelLibrary.finalize_result` will be used. + + +.. _library_future_directions: + +Future directions +----------------- + +The initial implementation of `~stpipe.library.AbstractModelLibrary` was intentionally +simple. Several features were discussed but deemed unnecessary for the current code. +This section will describe some of the discussed features to in-part provide a +record of these discussions. + +.. _library_borrow_limits: + +Borrow limits +^^^^^^^^^^^^^ + +As `~stpipe.library.AbstractModelLibrary` handles the loading and saving of models +(when "on disk") it could be straightforward to impose a limit to the number +and/or combined size of all "borrowed" models. This would help to avoid crashes +due to out-of-memory issues (especially important for HPC environments where +the memory limit may be defined at the job level). Being able to gracefully +recover from this error could also allow pipeline code to load as many +models as possible for more efficient batch processing. + + +.. _library_hollowing_out_models: + +Hollowing out models +^^^^^^^^^^^^^^^^^^^^ + +Currently the `~stpipe.library.AbstractModelLibrary` does not close +models when they are "shelved" (it relies on the garbage collector). +This was done to allow easier integration with existing pipeline code +but does mean that the memory used for a model will not be freed until +the model is freed. By explicitly closing models and possibly +removing references between the model and the data arrays ("hollowing +out") memory could be freed sooner allowing for an overall decrease. + +.. _library_append: + +Append +^^^^^^ + +There is no way to append a model to a `~stpipe.library.AbstractModelLibrary` +(nor is there a way to pop, extend, delete, etc, any operation that changes the +number of models in a library). This was an intentional choice as any operation +that changes the number of models would obviously invalidate the +`~stpipe.library.AbstractModelLibrary.asn` data. It should be possible +(albeit complex) to support some if not all of these operations. However +serious consideration of their use and exhuasting of alternatives is +recommended as the added complexity would likely introduce bugs. + +.. _library_updating_asn_on_shelve: + +Updating asn on shelve +^^^^^^^^^^^^^^^^^^^^^^ + +Related to the note about :ref:`library_append` updating the +`~stpipe.library.AbstractModelLibrary.asn` data on +`~stpipe.library.AbstractModelLibrary.shelve` would allow step code +to modify asn-related attributes (like group_id) and have these changes +reflected in the `~stpipe.library.AbstractModelLibrary.asn` result. +A similar note of caution applies here where some consideration +of the complexity required vs the benefits is recommended. + +.. _library_get_sections: + +Get sections +^^^^^^^^^^^^ + +`~stpipe.library.AbstractModelLibrary` has no replacement for +the ``get_sections`` API provided with ``ModelContainer``. If it's use +is generally required it might make sense to model the API off of +the existing group_id methods (where the subclass provides 2 methods +for efficiently accessing either an in-memory section or an on-disk +section for the "in memory" and "on disk" modes). + +.. _library_parallel_map_function: + +Parallel map function +^^^^^^^^^^^^^^^^^^^^^ + +`~stpipe.library.AbstractModelLibrary.map_function` is applied to each model +in a library sequentially. If this method proves useful and is typically +used with an independent and stateless function, extending the method to +use parallel application seems straightforward (although a new API might +be called for since a parallel application would likely not behave +as a generator. From 977bb93062255dbc0a5f2bcbce7cf91b518ba58f Mon Sep 17 00:00:00 2001 From: Brett Date: Mon, 29 Jul 2024 19:15:01 -0400 Subject: [PATCH 04/16] add usage patterns --- docs/source/model_library.rst | 96 +++++++++++++++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 5317fd6a..cbefa964 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -142,6 +142,102 @@ performed on the association metadata). >>> library.asn["products"][0]["name"] >>> library.asn["table_name"] +.. _library_usage_patterns: + +Usage Patterns +============== + +What follows is a section about using `~stpipe.library.AbstractModelLibrary` +in `~stpipe.step.Step` and `~stpipe.pipeline.Pipeline` code. This section +is short at the moment and can be extended with additional patterns as +the `~stpipe.library.AbstractModelLibrary` is used in more pipeline code. + +.. _library_step_input_handling: + +Step input handling +------------------- + +It is recommended that any `~stpipe.step.Step` (or `~stpipe.pipeline.Pipeline`) +that accept an +`~stpipe.library.AbstractModelLibrary` consider the performance when +processing the input. It likely makese sense for any `~stpipe.step.Step` +that accepts a `~stpipe.library.AbstractModelLibrary` to also accept +an association filename as an input. The basic input handling could look +something like the following: + +.. code-block:: pycon + + >>> def process(self, init): + ... if isinstance(init, ModelLibrary): + ... library = init # do not copy the input ModelLibrary + ... else: + ... library = ModelLibrary(init, self.on_disk) + ... # process library without making a copy as + ... # that would lead to 2x required file space for + ... # an "on disk" model and 2x the memory for an "in memory" + ... # model + ... return library + +The above pattern supports as input (``init``): + +- an `~stpipe.library.AbstractModelLibrary` +- an association filename (via the `~stpipe.library.AbstractModelLibrary` constructor) +- all other inputs supported by the `~stpipe.library.AbstractModelLibrary` constructor + +It is generally recommended to expose ``on_disk`` in the ``Step.spec`` +allowing the `~stpipe.step.Step` to generate an :ref:`library_on_disk` +`~stpipe.library.AbstractModelLibrary`: + +.. code-block:: pycon + + >>> class MyStep(Step): + ... spec = """ + ... on_disk = boolean(default=False) # keep models "on disk" to reduce RAM usage + ... """ + +.. NOTE:: + As mentioned in :ref:`library_on_disk` at no point will the input files + referenced in the association be modified. However, the above pattern + does allow ``Step.process`` to "modify" ``init`` when + ``init`` is a `~stpipe.library.AbstractModelLibrary` (the models + in the library will not be copied). + +``Step.process`` can extend the above pattern to +support additional inputs (for example a single +`~stpipe.datamodel.AbstractDataModel` or filename containing +a `~stpipe.datamodel.AbstractDataModel`) to allow more +flexible data processings. Although some consideration +should be given to how to handle input that does not +contain association metadata. Does it make sense +to construct a `~stpipe.library.AbstractModelLibrary` when the +association metadata is made up? Alternatively, is +it safer (less prone to misattribution of metadata) +to have the step process the inputs separately +(more on this below)? + +.. _library_isolated_processing: + +Isolated Processing +------------------- + +Let's say we have a `~stpipe.step.Step`, ``flux_calibration`` +that performs an operation that is only concerned with the data +for a single `~stpipe.datamodel.AbstractDataModel` at a time. +This step applies a function ``calibrate_model_flux`` that +accepts a single `~stpipe.datamodel.AbstractDataModel` and index as an input. +It's ``Step.process`` function can make good use of +`~stpipe.library.AbstractModelLibrary.map_function` to apply +this method to each model in the library. + +.. code-block:: pycon + + >>> class FluxCalibration(Step): + ... spec = "..." # use spec defined above + ... def process(self, init): + ... # see input pattern described above + ... # list is used here to consume the generator produced by map_function + ... list(library.map_function(calibrate_model_flux)) + ... return library .. _library_troubleshooting: From 8cec78d37f1b3864c4626d5709ca4b3f4cf42ac4 Mon Sep 17 00:00:00 2001 From: Brett Date: Mon, 29 Jul 2024 19:16:36 -0400 Subject: [PATCH 05/16] Fix typos found during review Co-authored-by: Ned Molter --- CHANGES.rst | 1 + docs/source/model_library.rst | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index fecc301e..a4d63238 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -3,6 +3,7 @@ - remove Windows tests and add info box indicating lack of Windows support to README [#163] - add ``ModelLibrary`` container class [#156] +- add ``ModelLibrary`` docs [#168] 0.6.1 (unreleased) ================== diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index cbefa964..ac0c6597 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -26,8 +26,8 @@ Most commonly an instance will be created from an association file: >>> library = ModelLibrary("my_asn.json") .. NOTE:: - For an association that contains a ``group_id`` for each member, - creating a library will not read any models. + Creating a library does not read any models into memory, + as long as the association contains a ``group_id`` for each member .. _library_borrowing_and_shelving: @@ -62,7 +62,7 @@ On Disk Mode For large associations (like those larger than memory) it is important that the library avoid reading all models at once. The borrow/shelve API -above maps closely to the loading/saving of input (or tempoary) files +above maps closely to the loading/saving of input (or temporary) files containing the models. .. code-block:: pycon @@ -77,7 +77,7 @@ containing the models. point will the library overwrite the input file. If model is not modified during the time it's borrowed (for example if the -``model.dq`` array was read, but not modified). It is helpful to tell the +``model.dq`` array was read, but not modified), it is helpful to tell the library that the model was not modified. .. code-block:: pycon @@ -87,7 +87,7 @@ library that the model was not modified. ... # do some read-only stuff with the model ... library.shelve(model, modify=False) # No temporary file will be written -This can dramatically reduce the number of times a file is written saving +This tells the library not to overwrite the model's temporary file while shelving, saving on both disk space and the time required to write. @@ -160,7 +160,7 @@ Step input handling It is recommended that any `~stpipe.step.Step` (or `~stpipe.pipeline.Pipeline`) that accept an `~stpipe.library.AbstractModelLibrary` consider the performance when -processing the input. It likely makese sense for any `~stpipe.step.Step` +processing the input. It likely makes sense for any `~stpipe.step.Step` that accepts a `~stpipe.library.AbstractModelLibrary` to also accept an association filename as an input. The basic input handling could look something like the following: @@ -206,7 +206,7 @@ allowing the `~stpipe.step.Step` to generate an :ref:`library_on_disk` support additional inputs (for example a single `~stpipe.datamodel.AbstractDataModel` or filename containing a `~stpipe.datamodel.AbstractDataModel`) to allow more -flexible data processings. Although some consideration +flexible data processings, although some consideration should be given to how to handle input that does not contain association metadata. Does it make sense to construct a `~stpipe.library.AbstractModelLibrary` when the @@ -400,7 +400,8 @@ container unchanged for other steps. A survey of container usage was performed with a few key findings: - Many uses could be replaced by simpler containers (lists) -- When loaded from an association, the container size never changed +- When loaded from an association, the container size never changed; + that is, no use-cases required adding new models to associations within steps - The order of models was never changed - Needs various methods for stpipe - Several steps implemented different memory optimizations @@ -437,7 +438,7 @@ of the API (most private) and internal details. One core issue is how can the container know when to load and save models (to temporary files) if needed? With a typical list ``__getitem__`` can map to load but what will map to save? -Initial prototypes used ``__setitem__`` which lead to some confusion +Initial prototypes used ``__setitem__`` which led to some confusion amongst reviewers. Treating the container like a list also leads to expectations that the container also support ``append`` ``extend`` and other API that is unnecessary (as determined @@ -531,7 +532,7 @@ Get sections ^^^^^^^^^^^^ `~stpipe.library.AbstractModelLibrary` has no replacement for -the ``get_sections`` API provided with ``ModelContainer``. If it's use +the ``get_sections`` API provided with ``ModelContainer``. If its use is generally required it might make sense to model the API off of the existing group_id methods (where the subclass provides 2 methods for efficiently accessing either an in-memory section or an on-disk From a2a53bd1e6ea8db9da1dde62c85eed731fd03967 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 09:16:19 -0400 Subject: [PATCH 06/16] add not about implicit borrow --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index ac0c6597..148422aa 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -50,7 +50,7 @@ Iteration is also supported (but don't forget to return your models!). .. code-block:: pycon >>> with library: - ... for model in library: + ... for model in library: # implicitly calls borrow() ... # do stuff with the model... ... library.shelve(model) From 79517d3826b61ccdfe40532c2592d4ccbc8a93de Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 09:18:52 -0400 Subject: [PATCH 07/16] clarify bullet about stpipe needs --- docs/source/model_library.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 148422aa..3d48ade1 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -403,7 +403,8 @@ A survey of container usage was performed with a few key findings: - When loaded from an association, the container size never changed; that is, no use-cases required adding new models to associations within steps - The order of models was never changed -- Needs various methods for stpipe +- Must be compatible with stpipe infrastructure (eg implements + ``crds_observatory``, ``get_crds_parameters``, etc methods) - Several steps implemented different memory optimizations and had significant complexity added to deal with containers that sometimes returned filenames and sometimes returned models From f350f55e8256523f1a969e3a249c70fc3519b966 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 09:22:23 -0400 Subject: [PATCH 08/16] split bullet --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 3d48ade1..f7b60b78 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -406,7 +406,7 @@ A survey of container usage was performed with a few key findings: - Must be compatible with stpipe infrastructure (eg implements ``crds_observatory``, ``get_crds_parameters``, etc methods) - Several steps implemented different memory optimizations - and had significant complexity added to deal with containers +- Step code has additional complexity to deal with containers that sometimes returned filenames and sometimes returned models Additionally, pipelines and steps may be expected to handle large From f71b528fca3bad78b2e16d0c758b4e739473d0dc Mon Sep 17 00:00:00 2001 From: Brett Graham Date: Wed, 31 Jul 2024 09:27:22 -0400 Subject: [PATCH 09/16] More updates from review Co-authored-by: Ned Molter --- docs/source/model_library.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index f7b60b78..43eef1e7 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -225,7 +225,7 @@ that performs an operation that is only concerned with the data for a single `~stpipe.datamodel.AbstractDataModel` at a time. This step applies a function ``calibrate_model_flux`` that accepts a single `~stpipe.datamodel.AbstractDataModel` and index as an input. -It's ``Step.process`` function can make good use of +Its ``Step.process`` function can make good use of `~stpipe.library.AbstractModelLibrary.map_function` to apply this method to each model in the library. @@ -261,7 +261,7 @@ which models were possibly modified. This error can be avoided by "opening" the library before calling `~stpipe.library.AbstractModelLibrary.borrow` (and being sure to call -`~stpipe.library.AbstractModelLibrary.shelve`, more on that below): +`~stpipe.library.AbstractModelLibrary.shelve`): .. code-block:: pycon From b484e29311e5d523dd415cebaf4bb3ccbe33a891 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 09:34:31 -0400 Subject: [PATCH 10/16] add warning about model scope --- docs/source/model_library.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 43eef1e7..bb462ae9 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -90,6 +90,15 @@ library that the model was not modified. This tells the library not to overwrite the model's temporary file while shelving, saving on both disk space and the time required to write. +.. WARNING:: + In the above example ``model`` remains in scope after the call to + `~stpipe.library.AbstractModelLibrary.shelve` (and even after + the exit of the with statement). This means ``model`` will not + be garbage collected (and it's memory will not be freed) until + the end of the scope containing the ``with library`` exits. If + more work occurs within the scope please consider adding an + explicit ``del model`` when your code is finished with the model. + .. _library_map_function: From 8a01ef2b44f353a5c7d9a1f8391ac00486b13e27 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 09:43:04 -0400 Subject: [PATCH 11/16] remove eg --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index bb462ae9..1a10bb6a 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -412,7 +412,7 @@ A survey of container usage was performed with a few key findings: - When loaded from an association, the container size never changed; that is, no use-cases required adding new models to associations within steps - The order of models was never changed -- Must be compatible with stpipe infrastructure (eg implements +- Must be compatible with stpipe infrastructure (implements ``crds_observatory``, ``get_crds_parameters``, etc methods) - Several steps implemented different memory optimizations - Step code has additional complexity to deal with containers From 55ccf148a2fc9da06c231ebb678d299e0d14dfc4 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 31 Jul 2024 10:22:36 -0400 Subject: [PATCH 12/16] expand asn docs --- docs/source/model_library.rst | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 1a10bb6a..a3900609 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -144,6 +144,12 @@ performed on the association metadata). any "borrows" using the indices do. Be sure to open the library before trying to borrow a model. + +.. _library_association_information: + +Association Information +======================= + `~stpipe.library.AbstractModelLibrary.asn` provides read-only access to the association data. .. code-block:: pycon @@ -151,6 +157,21 @@ performed on the association metadata). >>> library.asn["products"][0]["name"] >>> library.asn["table_name"] +Although the specifics of what is returned by `~stpipe.library.AbstractModelLibrary.asn` +depends on how the subclass implements ``AbstractModelLibrary._load_asn``, it +is required that the association metadata dictionary contain a "members" list. This +can be inspected via ``library.asn["products"][0]["members"]`` and must contain a +dictionary for each "member" including key value pairs for: + +- "expname" for the exposure name with a string value corresponding to the + name of the file for this member +- "exptype" for the exposure type with a string value describing the type + of exposure (for example "science" or "background") + +Although not required "group_id" (with a string value corresponding to the +group name) should be added to each member dictionary (see +:ref:`library_association` for more details). + .. _library_usage_patterns: Usage Patterns From aa12f8df86e1e5e10164b5ca9ce1ba454ebd512a Mon Sep 17 00:00:00 2001 From: Brett Graham Date: Wed, 31 Jul 2024 12:51:14 -0400 Subject: [PATCH 13/16] Update docs/source/model_library.rst Co-authored-by: Ned Molter --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index a3900609..2e7a856a 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -168,7 +168,7 @@ dictionary for each "member" including key value pairs for: - "exptype" for the exposure type with a string value describing the type of exposure (for example "science" or "background") -Although not required "group_id" (with a string value corresponding to the +Although not required, "group_id" (with a string value corresponding to the group name) should be added to each member dictionary (see :ref:`library_association` for more details). From 5d41520fdc4d78eb116e86c5506ab65e5142fcff Mon Sep 17 00:00:00 2001 From: Brett Graham Date: Wed, 31 Jul 2024 12:51:25 -0400 Subject: [PATCH 14/16] Update docs/source/model_library.rst Co-authored-by: Ned Molter --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 2e7a856a..ef7260f9 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -163,7 +163,7 @@ is required that the association metadata dictionary contain a "members" list. T can be inspected via ``library.asn["products"][0]["members"]`` and must contain a dictionary for each "member" including key value pairs for: -- "expname" for the exposure name with a string value corresponding to the +- "expname" for the exposure name, with a string value corresponding to the name of the file for this member - "exptype" for the exposure type with a string value describing the type of exposure (for example "science" or "background") From bde7b1869b97687e06423d1306e9c6ba883227b5 Mon Sep 17 00:00:00 2001 From: Brett Graham Date: Wed, 31 Jul 2024 12:51:32 -0400 Subject: [PATCH 15/16] Update docs/source/model_library.rst Co-authored-by: Ned Molter --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index ef7260f9..31794932 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -165,7 +165,7 @@ dictionary for each "member" including key value pairs for: - "expname" for the exposure name, with a string value corresponding to the name of the file for this member -- "exptype" for the exposure type with a string value describing the type +- "exptype" for the exposure type, with a string value describing the type of exposure (for example "science" or "background") Although not required, "group_id" (with a string value corresponding to the From b66eb3fd58becb6716d05a175bd56714d29a2561 Mon Sep 17 00:00:00 2001 From: Brett Graham Date: Wed, 31 Jul 2024 12:52:21 -0400 Subject: [PATCH 16/16] Update docs/source/model_library.rst Co-authored-by: Ned Molter --- docs/source/model_library.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/model_library.rst b/docs/source/model_library.rst index 31794932..66d72e18 100644 --- a/docs/source/model_library.rst +++ b/docs/source/model_library.rst @@ -161,7 +161,7 @@ Although the specifics of what is returned by `~stpipe.library.AbstractModelLibr depends on how the subclass implements ``AbstractModelLibrary._load_asn``, it is required that the association metadata dictionary contain a "members" list. This can be inspected via ``library.asn["products"][0]["members"]`` and must contain a -dictionary for each "member" including key value pairs for: +dictionary for each "member" including key-value pairs for: - "expname" for the exposure name, with a string value corresponding to the name of the file for this member