Add front matter and test Github Actions (IDAES#12)

* Initial content

* added missed dir

* saving work eh

* Try to get pytest to work for us now

* Debug conda env var

* try again

* try again

* more debug info

* marker file

* Drop python 3.11

* Better logging and preprocessing

* Added artifact upload

* Log format

* Added pytest report log as artifact

* fooling more with the artifacts

* Try to get reporting to run on failed tests

* Update actions to those supporting node 16

* Don't bother with pytest report log

* moved ngcc-soec to archive and added some pytest.ini options

* modified copyright header see Issue IDAES#7

* removed soec notebook from ToC since it's now in archive

* fixed omlt version

* added some exclude patterns for docs

* taking out omlt version again (mistake)

* Move back ngcc_soec so it's tested

* Forgot to add back to toc

* Correct some docs

* New CSVs

* Temporarily moved HDA with distillation flowsheet to archive/

* NGCC SOEC restored

* improved docs

* bugfix spurious preprocessing

* some missing index files

* Tweaks to table of contents

* Added comment about the active index

* bugfix

* Adding RTD integration

* Added get-extensions to RTD(?)

* added petsc, changed fail-on-warning to false

* fixed link

* wording

* cleaned up scripts

* add a PATH export

* try again PATH

* autochanged

* don't execute

* add 'conf' command

* adding in generated _doc files

* play with RTD script

* cat some info

* Use cache not doc files

* no more catting info

* refactored deps for readthedocs

* Better conf updates, esp for RTD

* Better help and behavior

* Added cache_file option for conf

* Added cache file option

* remove generated file

* moving failing notebooks to archive

* exclude exercise cells from docs

* changes

* MD links

* outputs

* add JupyterBook requirements

* take out dev requirements (namely tensorflow!)

* add jupyterbook requirements (keep dev)

* reduced tolerance for water utilization value assert (to 1e-2)

* create held dir, update docs, move notebooks

* link target for copyright headers subsection

* don't need target!

* updated for moved notebook

* added missing noauto tag

* black formatting

.github/workflows/core.yml

@@ -55,7 +55,6 @@ jobs:
           - '3.8'
           - '3.9'
           - '3.10'
-          - '3.11'
         os:
           - linux
           - win64
@@ -65,16 +64,24 @@ jobs:
           - os: win64
             runner-image: windows-2022
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: Set up Conda environment
-        uses: conda-incubator/setup-miniconda@v2.1.1
+        uses: conda-incubator/setup-miniconda@v2.2.0
         with:
           activate-environment: ${{ env.IDAES_CONDA_ENV_NAME_DEV }}
           python-version: ${{ matrix.python-version }}
       - name: Install the code
         uses: ./.github/actions/install
         with:
-          install-target: .[dev]
+          install-target: .[dev,jb]
       - name: Run pytest
         run: |
-          pytest -v
+          pwd
+          ls idaes_examples
+          pytest -v idaes_examples
+      - name: Upload pytest-xdist worker logs
+        if: success() || failure()
+        uses: actions/upload-artifact@v3
+        with:
+          name: pytest_worker_logs
+          path: "tests_*.log"

.github/workflows/docs.yml

@@ -0,0 +1,21 @@
+name: readthedocs/actions
+on:
+  pull_request_target:
+    types:
+      - opened
+    # Execute this action only on PRs that touch
+    # documentation files.
+    # paths:
+    #   - "docs/**"
+
+permissions:
+  pull-requests: write
+
+jobs:
+  pull-request-links:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: readthedocs/actions/preview@v1
+        with:
+          project-slug: idaes-examples
+

.readthedocs.yml

@@ -1,23 +1,23 @@
 version: 2
 
 build:
-  os: ubuntu-22.04
+  os: ubuntu-20.04
   tools:
     python: "3.8"
   jobs:
     pre_build:
-      # Generate variations on the base notebook files, in particular one for docs.
-      - "idaesx pre"
-      # Generate the Sphinx configuration for this Jupyter Book, so it builds.
-      - "jupyter-book config sphinx idaes_examples/notebooks/"
-
+      # Create the _doc notebooks
+      - idaesx pre -v
+      # Use cached data instead of notebook execution (will use saved '*_doc.ipynb' files)
+      # This also updates the Sphinx configuration (conf.py)
+      - idaesx conf --execute cache --cache-file nbcache --timeout 600 --sphinx --show
 python:
   install:
     - method: pip
       path: .
       extra_requirements:
-        - dev
+        - jb
 
 sphinx:
   builder: html
-  fail_on_warning: true
+  fail_on_warning: false

README-developer.md

@@ -22,7 +22,7 @@ Clone the repository from Github, setup your Python environment as you usually d
 ```shell
 # to create a conda environment first:
 # conda create -n idaes-examples python=3; conda activate idaes-examples
-pip install -e .[dev]
+pip install -e .[dev,jb]
 ```
 
 The configuration of the installation is stored in `pyproject.toml`.
@@ -38,65 +38,33 @@ The examples are divided into a few top-level directories.
   * `docs`: Examples and tutorials that will be tested and also published in the online documentation.
   * `active`: Not published but tested against new PRs
   * `_dev`: Notebooks used as examples for development and testing only.
-* `archive`: Not published and *not* tested. Just for historical interest.
+  * `held`: Not published and *not* tested. For the next release (or removal).
 * `mod`: Supporting Python modules (as a Python package).  
   It is usually best to match the name of the Python subpackage with its notebook directory.
 
-### Where to put a new notebook
-
-Use the following to decide where to put any new Jupyter notebook:
-
-1. Will it be included in the online docs?  
-**Yes**: Put under `notebooks/docs`  
-**No**: Go to 2
-2. Will it be actively maintained to handle any changes in IDAES core?  
-**Yes**: Put under `notebooks/active`  
-**No**: Put under `archive`
-
-All these directories have their own subdirectories for grouping similar notebooks.
-Place the new notebook in an appropriate subdirectory.
-
-Data files and images should go in the same directory as the notebook.  
-Python modules should go under an appropriate subpackage under `mod`.
+For guidance on where to put a new notebook, see the [Examples Standards][standards] page in the IDAES examples repo wiki.
 
 ## Running tests
 
-There are two ways to run tests: running all the notebooks (integration tests), and 
-testing that notebooks work without running all their code (unit tests).
-
 ### Integration tests
 
 Run integration tests from the top-level (root) directory of the repository.
 In the root directory, tests are configured by `pyproject.toml`; see the *tool.pytest.ini_options* section.
 
 ```shell
 # from the root directory of the repository
-pytest
-```
-If you want to *exclude* certain notebooks from the integration tests, see the _Preprocessing -> Jupyter notebook metadata_ section.
-
-### Unit tests
-
-Run unit tests from the `idaes_examples` directory of the repository.
-In the idaes_examples directory, tests are configured by `idaes_examples/pytest.ini`. 
-To run the **unit tests** change to do the `idaes_examples` directory, then run the same command:
-
-```shell
-cd idaes_examples
-pytest
+pytest idaes_examples
 ```
 
 #### Testing a subset of the notebooks
 
-To test just one notebook, you need to (a) make sure the correct configuration file is picked up,
-e.g., even if you're not in the root directory and (b) use the name of the *test* notebook not the source.
-
+To test just one notebook, you need to use the name of the *test* notebook not the source.
 For example, to test the `compressor_src.ipynb` notebook (in the `unit_models/operations` subdirectory)
 you need to add `-c` and the path to the top-level *pyproject.toml*, which has the pytest configuration,
 then use the name of the test notebook:
 
 ```shell
-pytest -c ~\path/to/idaes-examples/pyproject.toml -v compressor_test.ipynb
+pytest -v idaes_examples/notebooks/docs/unit_models/operations/compressor_test.ipynb
 ```
 
 If you want to test several related notebooks, or perhaps just not type the whole notebook filename,
@@ -105,7 +73,7 @@ So, for example, from the top-level directory if you want to test all the *docs/
 you could take advantage of the fact that "operations" appears only in this particular path:
 
 ```shell
-pytest -k operations
+pytest -v idaes_examples -k operations
 # will match: 
 # docs/unit_models/operations/compressor_test.ipynb
 # docs/unit_models/operations/heat_exchanger_0d_test.ipynb
@@ -168,32 +136,32 @@ A diagram of how preprocessing relates to notebook usage is shown below:
 
 ## Notebook names
 
-Notebooks all have a file name that fits the pattern notebook-name`_ext`.ipynb*.
-When creating or modifying notebooks, you should always use `ext` = *src*.
+Notebooks all have a file name that fits the pattern notebook-name`_<ext>`.ipynb*.
+When creating or modifying notebooks, you should always use `<ext>` = *src*.
 Other extensions are automatically generated when running tests, building the documentation, and manually running the preprocessing step.
 See the <a href="#table-nbtypes">table of notebook types</a> for details.
 
 ## How to add a new notebook
 
-See `tutorial.md` in this directory for a more detailed step-by-step tutorial.
-
 There are two main steps to creating a new notebook example.
 
 1. Add Jupyter Notebook and supporting files
-   1. Look at the `README.md` file in *idaes_examples* to figure out where to put the notebook.
-   2. If you create a new directory for the notebook, the directory name *should* be in lowercase 
-      with underscores between words. For example: 'machine_learning'.
+   1. See the [standards][standards] to figure out the parent directory for the notebook -- usually, it's *docs*.
+   2. Put the notebook in the appropriate subdirectory.
+      If you create a new directory for the notebook, the directory name *should* be in lowercase 
+      with underscores between words. For example: '*docs/machine_learning*'.
    3. Notebook filename *should* be in lowercase with underscores and ***must*** end with '_src.ipynb'. For example: 
       'my_example_src.ipynb'.
    4. Add -- in the same directory as the notebook -- any data files or images it needs.
    5. Additional Python modules should be put in an appropriate place under *idaes_examples/mod*.
-Then your notebook can write: `from idaes_examples.mod.<subpackage> import <bla>`
-2. Add Jupyter notebook to the Jupyterbook table of contents in *idaes_examples/nb/_toc.yml*.
+      Then your notebook can write: `from idaes_examples.mod.<subpackage> import <bla>`
+2. Add Jupyter notebook to the Jupyterbook table of contents in *idaes_examples/notebooks/_toc.yml*.
    1. The notebook will be a *section*. If you added a new directory, you will create a new *chapter*, otherwise it will go under an existing one. See [Jupyterbook][jb] documentation for more details.
    2. Refer to the notebook as '*path/to/notebook-name*_doc' (replace '_src' with '_doc' and drop the '.ipynb' extension). For example: 'machine_learning/my_example_doc'.
    3. If you created a new directory for this notebook, make sure you add an *index.md* file to it. See other *index.md* files for the expected format.
 
 You *should*  test the new notebook and build it locally before pushing the new file, i.e., run `pytest` and `idaesx build`.
+Note that the cache for the documentation will be updated to contain the new output cells, which will modify files in *idaes_examples/notebooks/nbcache*; these files should also be committed and pushed.
 
 ### Jupyter Notebook file extensions
 
@@ -274,6 +242,7 @@ All existing notebooks and Python files will be automatically discovered and mod
 [jb]: https://jupyterbook.org/
 [hidecell]: https://jupyterbook.org/en/stable/interactive/hiding.html
 [celltags]: https://jupyterbook.org/en/stable/content/metadata.html
+[standards]: https://github.com/IDAES/examples/wiki/IDAES-Examples-Standards
 
 ----
 Author: Dan Gunter  

README.md

@@ -7,22 +7,44 @@ Below are basic instructions to install, view, and run the examples.
 
 If you are a developer who wishes to modify or add new notebooks, please refer to the file *README-developer.md*.
 
-## Installing
+### Categories of examples
 
-This repository can be installed with *pip*: **NOT WORKING YET**
+In the source code repository, you may note that there are a number of examples that are not in the documentation.
+There are two main categories of examples:
+
+  - "Docs" examples (under `idaes_examples/notebooks/docs`), which are tested and built into this documentation.
+  - "Active" examples (under `idaes_examples/notebooks/active`) that are tested but *not* in the documentation.
+
+There is also a third category of "Held" examples (under `idaes_examples/notebooks/held`),
+which could in the next release of IDAES in Docs or Active, or could be removed.
+These are *not* tested and *not* in the docs, and should generally be ignored by non-developers.
+
+## Installation
+
+This repository can be installed with *pip*:
 ```shell
-pip install idaes_examples
+pip install idaes-examples
 ```
 
-## Browsing notebooks
+We recommend you use a virtual environment tool such as
+[Miniconda](https://docs.conda.io/en/latest/miniconda.html)
+to install and run the notebooks in an isolated environment.
+
+## Run examples
+
+Use the command `idaesx gui` to get a simple graphical UI that lets you 
+browse and open notebooks for local execution and experimentation.
+
+## Build documentation
+
+Run the command `idaesx build` from the repository root to build the [JupyterBook](https://jupyterbook.org) 
+documentation.
 
-Use the `idaesx gui` command to get a simple graphical UI that lets you browse and open notebooks (in a Jupyter server) for local execution.
 
-## Building documentation locally
+*Note: This will take quite a while, as each example must be run first.
+You may want to step out and enjoy a beverage.*
 
-Run `idaesx build` from the repository root. This will take a while.
-You may want to step out and enjoy a warm beverage.
 
 ----
 Author: Dan Gunter  
-Last modified: 03 Feb 2023
+Last modified: 17 Feb 2023

file_header.txt

@@ -1,10 +1,10 @@
 The Institute for the Design of Advanced Energy Systems Integrated Platform
 Framework (IDAES IP) was produced under the DOE Institute for the
-Design of Advanced Energy Systems (IDAES), and is copyright (c) 2018-2022
-by the software owners: The Regents of the University of California, through 
-Lawrence Berkeley National Laboratory,  National Technology & Engineering 
-Solutions of Sandia, LLC, Carnegie Mellon University, West Virginia University
-Research Corporation, et al.  All rights reserved.
+Design of Advanced Energy Systems (IDAES).
 
-Please see the files COPYRIGHT.md and LICENSE.md for full copyright and
-license information.
+Copyright (c) 2018-2023 by the software owners: The Regents of the
+University of California, through Lawrence Berkeley National Laboratory,
+National Technology & Engineering Solutions of Sandia, LLC, Carnegie Mellon
+University, West Virginia University Research Corporation, et al.
+All rights reserved.  Please see the files COPYRIGHT.md and LICENSE.md
+for full copyright and license information.