Merge pull request #42 from CBroz1/main

Add NWB support for project schema, pre-commit and markdownlint

.gitignore

@@ -106,6 +106,7 @@ ENV/
 # datajoint
 dj_local_conf.json
 dj_local_conf_old.json
+temp*
 
 # emacs
 **/*~

.markdownlint.yaml

@@ -0,0 +1,16 @@
+# Markdown Linter configuration for docs
+# https://github.com/DavidAnson/markdownlint
+# https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md
+MD009: false # permit trailing spaces
+MD007: false # List indenting - permit 4 spaces
+MD013:
+    line_length: "88" # Line length limits
+    tables: false # disable for tables
+    headings: false # disable for headings
+MD030: false # Number of spaces after a list
+MD033: # HTML elements allowed
+    allowed_elements:
+        - "br"
+MD034: false # Permit bare URLs
+MD031: false # Spacing w/code blocks. Conflicts with `??? Note` and code tab styling
+MD046: false # Spacing w/code blocks. Conflicts with `??? Note` and code tab styling

.pre-commit-config.yaml

@@ -0,0 +1,58 @@
+default_stages: [commit, push]
+exclude: (^.github/|^docs/|^images/)
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files # prevent giant files from being committed
+      - id: requirements-txt-fixer
+      - id: mixed-line-ending
+        args: ["--fix=lf"]
+        description: Forces to replace line ending by the UNIX 'lf' character.
+
+  # black
+  - repo: https://github.com/psf/black
+    rev: 22.12.0
+    hooks:
+      - id: black
+      - id: black-jupyter
+        args:
+          - --line-length=88
+
+  # isort
+  - repo: https://github.com/pycqa/isort
+    rev: 5.11.2
+    hooks:
+      - id: isort
+        args: ["--profile", "black"]
+        description: Sorts imports in an alphabetical order
+
+  # flake8
+  - repo: https://github.com/pycqa/flake8
+    rev: 4.0.1
+    hooks:
+      - id: flake8
+        args: # arguments to configure flake8
+          # making isort line length compatible with black
+          - "--max-line-length=88"
+          - "--max-complexity=18"
+          - "--select=B,C,E,F,W,T4,B9"
+
+          # these are errors that will be ignored by flake8
+          # https://www.flake8rules.com/rules/{code}.html
+          - "--ignore=E203,E501,W503,W605,E402"
+          # E203 - Colons should not have any space before them.
+          #        Needed for list indexing
+          # E501 - Line lengths are recommended to be no greater than 79 characters.
+          #        Needed as we conform to 88
+          # W503 - Line breaks should occur after the binary operator.
+          #        Needed because not compatible with black
+          # W605 - a backslash-character pair that is not a valid escape sequence now
+          #        generates a DeprecationWarning. This will eventually become a SyntaxError.
+          #        Needed because we use \d as an escape sequence
+          # E402 - Place module level import at the top.
+          #        Needed to prevent circular import error

CHANGELOG.md

@@ -1,23 +1,31 @@
 # Changelog
 
-Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
+Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
+[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
+
+## [0.2.0] - 2022-01-20
 
-## [0.2.0] - 2022-12-12
 + Add - `project` schema for project/study/experiment related tables
++ Add - pre-commit and markdown lint checks
 
 ## [0.1.2] - 2022-10-17
+
 + Update - CICD for triggering PyPI release
 
 ## [0.1.1] - 2022-05-10
+
 + Add - Adopted black formatting into code base
 
 ## [0.1.0b1] - 2022-01-28
+
 + Add - Functions to generate dictionaries for NWB export.
 
 ## [0.1.0b0] - 2021-05-07
+
 + Add - First beta release
 
 ## [0.1.0a1] - 2021-04-30
+
 + Add - GitHub Action release process
 + Add - `lab` schema
 

CODE_OF_CONDUCT.md

@@ -1,4 +1,3 @@
-
 # Contributor Covenant Code of Conduct
 
 ## Our Pledge

CONTRIBUTING.md

@@ -1,2 +1,5 @@
 # Contribution Guidelines
-This project follows the [DataJoint Contribution Guidelines](https://docs.datajoint.io/python/community/02-Contribute.html). Please reference the link for more full details.
+
+This project follows the
+[DataJoint Contribution Guidelines](https://datajoint.com/docs/community/contribute/).
+Please reference the link for more full details.

README.md

@@ -1,47 +1,9 @@
 # DataJoint Element - Lab
 
-DataJoint Element for lab metadata. DataJoint Elements collectively standardize and automate data collection and analysis for neuroscience experiments. Each Element is a modular pipeline for data storage and processing with corresponding database tables that can be combined with other Elements to assemble a fully functional pipeline.
+DataJoint Element for lab metadata. DataJoint Elements collectively standardize and
+automate data collection and analysis for neuroscience experiments. Each Element is a
+modular pipeline for data storage and processing with corresponding database tables that
+can be combined with other Elements to assemble a fully functional pipeline.
 
-+ `element-lab` features a DataJoint pipeline design for lab related information, such as Lab, User, Project, Protocol, Source.
-
-+ `element-lab` is not a complete workflow by itself, but rather a modular design of tables and dependencies. 
-
-+ `element-lab` can be flexibly attached to any DataJoint workflow.
-
-+ See the [Element Lab documentation](https://elements.datajoint.org/description/lab/) for the background information and development timeline.
-
-+ For more information on the DataJoint Elements project, please visit https://elements.datajoint.org.  This work is supported by the National Institutes of Health.
-
-## Element architecture
-
-![element lab diagram](images/lab_diagram.svg)
-
-## Installation
-```
-pip install element-lab
-```
-
-If you already have an older version of ***element-lab*** installed using `pip`, upgrade with
-```
-pip install --upgrade element-lab
-```
-
-## Element usage
-
-+ `element-lab` is not a complete workflow by itself, but rather a modular design of tables and dependencies that can be flexibly attached to any DataJoint workflow.
-
-+ See the [workflow-calcium-imaging](https://github.com/datajoint/workflow-calcium-imaging) and [workflow-array-ephys](https://github.com/datajoint/workflow-array-ephys) repositories for example usages of `element-lab`.
-
-## Citation
-
-+ If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
-
-+ DataJoint for Python or MATLAB
-    + Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
-
-    + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
-
-+ DataJoint Elements
-    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
-
-    + DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Lab (version `<Enter version number>`)
+Installation and usage instructions can be found at the
+[Element documentation](https://datajoint.com/docs/elements/element-lab).

cspell.json

@@ -0,0 +1,44 @@
+// cSpell Settings
+//https://github.com/streetsidesoftware/vscode-spell-checker
+{
+    "version": "0.2", // Version of the setting file.  Always 0.2
+    "language": "en", // language - current active spelling language
+    "enabledLanguageIds": [
+        "markdown", "yaml", "python"
+    ],
+    // flagWords - list of words to be always considered incorrect
+    // This is useful for offensive words and common spelling errors.
+    // For example "hte" should be "the"
+    "flagWords": [],
+    "allowCompoundWords": true,
+    "ignorePaths": [
+        "./element_array_ephys.egg-info/*",
+        "./images/*"
+    ],
+    "words": [
+        "Berens",
+        "CICD",
+        "Ecker",
+        "elif",
+        "Ephys",
+        "Hoenselaar",
+        "IACUC",
+        "inlinehilite",
+        "linenums",
+        "mkdocs",
+        "mkdocstrings",
+        "pymdownx",
+        "Reimer",
+        "Roboto",
+        "RRID",
+        "Rxiv",
+        "Sasaki",
+        "Shen",
+        "Siapas",
+        "Sinz",
+        "Sitonic",
+        "Tolias",
+        "Yatsenko",
+        "Zuckerman"
+    ]
+}

docs/mkdocs.yaml

@@ -43,7 +43,7 @@ nav:
 #     HOST_UID=$(id -u) docker compose -f docs/docker-compose.yaml up --build
 #     ```
 # 02. Site analytics depend on a local environment variable GOOGLE_ANALYTICS_KEY
-#     You can find this in LastPass or declare with any string to suprress errors
+#     You can find this in LastPass or declare with any string to suppress errors
 # 03. The API section will pull docstrings.
 #     A. Follow google styleguide e.g.,
 #        https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
@@ -126,6 +126,7 @@ markdown_extensions:
       linenums: true
   - pymdownx.inlinehilite
   - pymdownx.snippets
+  - footnotes
 
 extra:
   PATCH_VERSION: !ENV PATCH_VERSION

docs/src/citation.md

@@ -1,14 +1,12 @@
 
 # Citation
 
-If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
+If your work uses this Element, please cite the following manuscript and Research
+Resource Identifier (RRID).
 
-+ DataJoint for Python or MATLAB
-    +  Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
++ Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D,
+  Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for
+  Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
 
-    + DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
-
-+ DataJoint Elements
-    + Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
-
-    + DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Lab (version {{ PATCH_VERSION }})
++ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) -
+  Element Lab (version {{ PATCH_VERSION }})

docs/src/concepts.md

@@ -2,16 +2,24 @@
 
 ## Storage of high-level information about a laboratory 
 
-Most pipelines track some information about the lab, including the facilities, experiment rigs, and users. This can help track which users lab members are associated with which projects or who is responsible for equipment.
+Most pipelines track some information about the lab, including the facilities,
+experiment rigs, and users. This can help track which lab members are associated
+with which projects or who is responsible for equipment.
 
-Through our interviews and direct collaboration on the precursor projects, we identified the common motifs in the lab schemas. This Element works for diverse downstream pipelines and is always used in combination with other Elements for specific experiments. As such, it is validated jointly with the acquisition Elements such as
-[Extracellular Array Electrophysiology](https://datajoint.com/docs/elements/element-array-ephys) 
-and 
-[Calcium Imaging](https://datajoint.com/docs/elements/element-calcium-imaging).
+Through our interviews and direct collaboration on the precursor projects, we identified
+the common motifs in the lab schemas. This Element works for diverse downstream
+pipelines and is always used in combination with other Elements for specific
+experiments. As such, it is validated jointly with the acquisition Elements such as
+[Extracellular Array
+Electrophysiology](https://datajoint.com/docs/elements/element-array-ephys) and [Calcium
+Imaging](https://datajoint.com/docs/elements/element-calcium-imaging).
 
 ## Key Partnerships
 
-Over the several years, many labs have developed DataJoint-based pipelines for lab management. The DataJoint team collaborated with several and interviewed these teams to understand their experiment workflow, associated tools, and interfaces. These teams include: 
+Over the several years, many labs have developed DataJoint-based pipelines for lab
+management. The DataJoint team collaborated with several and interviewed these teams to
+understand their experiment workflow, associated tools, and interfaces. These teams
+include: 
 
 - International Brain Laboratory
 - BrainCoGs (Princeton Neuroscience Institute), Python pipeline, MATLAB pipeline 
@@ -24,21 +32,37 @@ Over the several years, many labs have developed DataJoint-based pipelines for l
 
 ![element lab diagram](https://raw.githubusercontent.com/datajoint/element-lab/d222f673e590979a92ff815adb880f474eed338e/images/lab_diagram.svg)
 
+### `lab` schema ([API docs](../api/element_lab/lab/))
 
-### `lab` schema ([API docs](https://datajoint.com/docs/elements/element-lab/api/element_lab))
-
-| Table              | Description                                                      |
-| ------------------ | ---------------------------------------------------------------- |
-| Lab                | Table for storing general lab info.                              |
-| Location           | Location of research (e.g., animal housing or experimental rigs) |
-| UserRole           | Roles assigned to a user or a job title.                         |
-| User               | Table for storing user information.                              |
-| LabMembership      | Store lab membership information using three lookup tables.      |
-| ProtocolType       | Type of protocol or issuing agency.                              |
-| Protocol           | Protocol specifics (e.g., protocol number and title).            |
-| Project            | Projects within a lab.                                           |
-| ProjectKeywords    | Project keywords or meta-information.                            |
-| ProjectPublication | Project's resulting publications.                                |
-| ProjectSourceCode  | URL to source code for replication.                              |
-| ProjectUser        | Users participating in the project.                              |
-| Source             | Source or supplier of subject animals.                           |
+| Table                 | Description                                                  |
+| ------------------    | -------------------------------------------------------------|
+| Lab                   | Table for storing general lab info.                          |
+| Location              | Location of research (e.g., animal housing, experimental rigs)|
+| UserRole              | Roles assigned to a user or a job title.                     |
+| User                  | Table for storing user information.                          |
+| LabMembership         | Store lab membership information using three lookup tables.  |
+| ProtocolType          | Type of protocol or issuing agency.                          |
+| Protocol              | Protocol specifics (e.g., protocol number and title).        |
+| Project[^1]           | Projects within a lab.                                       |
+| ProjectKeywords[^1]   | Project keywords or meta-information.                        |
+| ProjectPublication[^1]| Project's resulting publications.                            |
+| ProjectSourceCode[^1] | URL to source code for replication.                          |
+| ProjectUser[^1]       | Users participating in the project.                          |
+| Source                | Source or supplier of subject animals.                       |
+
+[^1]: 
+    Depreciation warning. A subset of the tables above will be removed in a future
+    version of this Element in favor of the `project` schema.
+
+### `project` schema ([API docs](../api/element_lab/project/))
+
+| Table                 | Description                                                  |
+| ------------------    | -------------------------------------------------------------|
+| Project               | Top-level grouping of studies and experiments to investigate a scientific question. |
+| ProjectPersonnel      | List of individuals involved in a project.                   |
+| ProjectKeywords       | Project keywords. If the dataset is exported, this metadata is saved within the NWB file. |
+| ProjectPublication    | Project's resulting publications.                            |
+| ProjectSourceCode     | Web address of source code.                                  |
+| Study                 | A set of experiments designed to address a specific aim.     |
+| Protocol              | Info about institutional approval (e.g., IACUC, IRB, etc.)   |
+| Experiment            | Experimental tasks and their associated lab, study, and protocol. |

docs/src/index.md

@@ -1,12 +1,19 @@
 # Element Lab for storing lab information
 
+DataJoint Element for lab metadata. DataJoint Elements collectively standardize and
+automate data collection and analysis for neuroscience experiments. Each Element is a
+modular pipeline for data storage and processing with corresponding database tables that
+can be combined with other Elements to assemble a fully functional pipeline.
 
-DataJoint Element for lab metadata. DataJoint Elements collectively standardize and automate data collection and analysis for neuroscience experiments. Each Element is a modular pipeline for data storage and processing with corresponding database tables that can be combined with other Elements to assemble a fully functional pipeline.
+This Element ...
 
-+ `element-lab` features a DataJoint pipeline design for lab related information, such as Lab, User, Project, Protocol, Source.
+- features a DataJoint pipeline design for lab related information, such as Lab, User,
+  Project, Protocol, Source.
 
-+ `element-lab` is not a complete workflow by itself, but rather a modular design of tables and dependencies. 
-
-+ `element-lab` can be flexibly attached to any DataJoint workflow.
+- is not a complete workflow by itself, but rather a modular design of tables and
+  dependencies. 
 
+- can be flexibly attached to any DataJoint workflow.
 
+Visit the [Concepts page](./concepts.md) for more information on Element Lab.
+To get started with building your data pipeline navigate to the [Tutorials](./tutorials.md) page.