Merge branch 'main' into access_url

README.md

@@ -5,20 +5,14 @@ imaged exoplanets: a simple archive of complex objects. The tables and fields cu
 database are described in the [Documentation](documentation/README.md) 
 and currently include names, coordinates, photometry and reference and data provenance information and is visualized 
 in the [schema](#simple-database-schema) below. 
-We are currently working on including kinematics, spectra, images, and modeled and retrieved parameters. 
+The archive includes kinematics, spectra, modeled and retrieved parameters. 
 
-We are developing several different methods to interact with the database, including python, a website and API, and 
-database browsers.
+There are several different methods to interact with the database, including: 
+- Python
+- a website and API at https://simple-bd-archive.org/
+- database browsers such as [DB Browser for SQLite](https://sqlitebrowser.org/)
 
-While we are using brown dwarfs to build out the SIMPLE database, our intention is to build a database schema and 
-software which could be used by other subfields to roll their own collaborative databases and web interfaces.
-
-This database uses the [SQLAlchemy ORM](https://docs.sqlalchemy.org/en/14/orm/index.html) and is designed to be
-interacted with via the `astrodbkit2` package.
-
-If you'd like to participate or just stay in the loop as this project progresses, please request to join this discussion
- list:
-https://groups.google.com/forum/#!forum/simple-archive
+To report missing data, data bugs or make feature requests, please use the [SIMPLE-db issue tracker](https://github.com/SIMPLE-AstroDB/SIMPLE-db/issues).
 
 For day-to-day discussions, please join us in the #simple-db-dev channel in the [Astropy Slack workspace](https://astropy.slack.com/).
 If you are not already in the Astropy Slack, [request an account](http://joinslack.astropy.org/).
@@ -31,37 +25,28 @@ If you'd like to set up your own copy of the SIMPLE database, here's what we rec
 
 1. Clone or download a copy of this repo locally onto your computer.
 
-2. Set up an environment for the python code. 
+2. Set up an environment for the Python code and install dependencies. 
 A conda environment file `environment.yml` exists for convenience. The following commands will use that file to create and activate an 
    environment called `simple-db`:
 
     ```bash
     conda env create -f environment.yml
     conda activate simple-db
     ```
-
-3. Install the latest version of the AstrodbKit2 package:
-
-    ```bash
-     pip install git+https://github.com/dr-rodriguez/AstrodbKit2
-     ```
 
-4. Create an empty database and import the SIMPLE schema, and connect the database file `SIMPLE.db` as a Database object called `db
+3. In Python, connect a database file `SIMPLE.sqlite` as a Database object called `db` and recreate the database using the JSON files in the `data/` directory.
 
    ```python
-   from astrodbkit2.astrodb import create_database,Database
+   from astrodb_utils import load_astrodb
    from simple.schema import *
+   from simple.schema import REFERENCE_TABLES
 
-   connection_string = 'sqlite:///SIMPLE.db' # connection string for a SQLite database named SIMPLE.db
-   create_database(connection_string)
-   db = Database(connection_string)
-   ```
-5. Load in the database by reading in the directory where the JSON files are located
+   db = load_astrodb("SIMPLE.sqlite", recreatedb=True,  reference_tables=REFERENCE_TABLES)
+    ```
 
-   ```python
-   db.load_database('data/')
-   ```
-6. Use `astrodbkit2` to [explore](https://astrodbkit2.readthedocs.io/en/latest/#exploring-the-schema), [query](https://astrodbkit2.readthedocs.io/en/latest/#querying-the-database), and/or [modify](https://astrodbkit2.readthedocs.io/en/latest/#modifying-data) the database.
+    This step generates a "SIMPLE.sqlite" file which can be opened, explored, and modified using a variety of tools.
+
+4. Use `astrodbkit2` to [explore](https://astrodbkit2.readthedocs.io/en/latest/#exploring-the-schema), [query](https://astrodbkit2.readthedocs.io/en/latest/#querying-the-database), and/or [modify](https://astrodbkit2.readthedocs.io/en/latest/#modifying-data) the database.
 For example:
     - Find all objects in the database with "0141" in the name
         ```
@@ -73,12 +58,28 @@ For example:
         ```
         db.inventory('2MASS J01415823-4633574', pretty_print=True)
         ```
+5. The database can also be modified using helper scripts found in [`simple/utils`](simple/utils) and in the [`astrodb_utils`](https://github.com/astrodbtoolkit/astrodb_utils) package. Previously used scripts to modify and/or update the database are stored in the [`scripts/`](scripts) directory and can be used for inspiration.
+
+## Contributor Instructions
+If you've made changes to the SIMPLE Archive that you would like to contribute to the public version, here's how to make a contribution.
+
+1. If you've made changes to the database contents, write out the new/modified JSON files:
+    ```python
+    db.save_database(directory="data/")
+    ```
+
+2. (Optional, but highly recommended) Run and modify tests as necessary
+    ```bash
+    pytest
+    ```
+
+3. Open a pull request with the modified JSON files and optionally, your ingest script.
+
 
-
 ## SIMPLE Database Schema
 
 The schema for the SIMPLE database is described
-in the [Documentation](documentation) and can be found in `simple/schema.py`.
+in the [Documentation](documentation) and can be found in [`simple/schema.py`](simple/schema.py).
 
 A graphical representation of the SIMPLE schema:
-<img src="https://github.com/kelle/SIMPLE-db/blob/new-schema/documentation/figures/schema2023.png?raw=true" width=75%>
+<img src="https://github.com/SIMPLE-AstroDB/SIMPLE-db/blob/main/documentation/figures/schema2023.png?raw=true" width=75%>

simple/schema.py

@@ -3,21 +3,22 @@
 """
 
 import enum
+
 import sqlalchemy as sa
-from sqlalchemy.orm import validates
+from astrodbkit2.astrodb import Base
+from astrodbkit2.views import view
+from astropy.io.votable.ucd import check_ucd
 from sqlalchemy import (
     Boolean,
     Column,
+    DateTime,
+    Enum,
     Float,
     ForeignKey,
-    String,
-    Enum,
-    DateTime,
     ForeignKeyConstraint,
+    String,
 )
-from astrodbkit2.astrodb import Base
-from astrodbkit2.views import view
-from astropy.io.votable.ucd import check_ucd
+from sqlalchemy.orm import validates
 
 # -------------------------------------------------------------------------------------------------------------------
 # Reference tables
@@ -44,6 +45,12 @@ class Publications(Base):
     doi = Column(String(100))
     description = Column(String(1000))
 
+    @validates("reference")
+    def validate_reference(self, key, value):
+        if value is None or len(value) > 30:
+            raise ValueError(f"Provided reference is invalid; too long or None: {value}")
+        return value
+
 
 class Telescopes(Base):
     __tablename__ = 'Telescopes'
@@ -102,7 +109,7 @@ def validate_ucd(self, key, value):
             raise ValueError(f"UCD {value} not in controlled vocabulary")
         return value
 
-    @validates("effective_wavelength_angstroms")
+    @validates("effective_wavelength")
     def validate_wavelength(self, key, value):
         if value is None or value < 0:
             raise ValueError(f"Invalid effective wavelength received: {value}")
@@ -169,6 +176,18 @@ class Sources(Base):
     other_references = Column(String(100))
     comments = Column(String(1000))
 
+    @validates("ra")
+    def validate_ra(self, key, value):
+        if value > 360 or value < 0:
+            raise ValueError("RA not in allowed range (0..360)")
+        return value
+
+    @validates("dec")
+    def validate_dec(self, key, value):
+        if value > 90 or value < -90:
+            raise ValueError("Dec not in allowed range (-90..90)")
+        return value
+
 
 class Names(Base):
     __tablename__ = 'Names'

tests/test_schema.py

@@ -0,0 +1,49 @@
+"""Tests for the schema itself and any validating functions"""
+
+import pytest
+
+from simple.schema import PhotometryFilters, Publications, Sources
+
+
+def schema_tester(table, values, error_state):
+    """Helper function to handle the basic testing of the schema classes"""
+    if error_state is None:
+        _ = table(**values)
+    else:
+        with pytest.raises(error_state):
+            _ = table(**values)
+
+@pytest.mark.parametrize("values, error_state",
+                         [
+                             ({"band": "2MASS.J", "effective_wavelength": 1.2, "ucd": "phot;em.IR.J"}, None),
+                             ({"band": "2MASS.J", "effective_wavelength": 1.2, "ucd": "bad"}, ValueError),
+                             ({"band": "bad", "effective_wavelength": 1.2, "ucd": "phot;em.IR.J"}, ValueError),
+                             ({"band": "2MASS.J", "effective_wavelength": -99, "ucd": "phot;em.IR.J"}, ValueError),
+                          ])
+def test_photometryfilters(values, error_state):
+    """Validating PhotometryFilters"""
+    schema_tester(PhotometryFilters, values, error_state)
+
+
+@pytest.mark.parametrize("values, error_state",
+                         [
+                             ({"source": "FAKE", "ra": 1.2, "dec": 3.4, "reference": "Ref1"}, None),
+                             ({"source": "FAKE", "ra": 999, "dec": 3.4, "reference": "Ref1"}, ValueError),
+                             ({"source": "FAKE", "ra": -999, "dec": 3.4, "reference": "Ref1"}, ValueError),
+                             ({"source": "FAKE", "ra": 1.2, "dec": 999, "reference": "Ref1"}, ValueError),
+                             ({"source": "FAKE", "ra": 1.2, "dec": -999, "reference": "Ref1"}, ValueError),
+                          ])
+def test_sources(values, error_state):
+    """Validating Sources"""
+    schema_tester(Sources, values, error_state)
+
+
+@pytest.mark.parametrize("values, error_state",
+                         [
+                             ({"reference": "Ref1"}, None),
+                             ({"reference": None}, ValueError),
+                             ({"reference": "THIS-REFERENCE-IS-REALLY-REALLY-LONG"}, ValueError),
+                          ])
+def test_publications(values, error_state):
+    """Validating Publications"""
+    schema_tester(Publications, values, error_state)