Skip to content

Commit

Permalink
Update schema versioning such
Browse files Browse the repository at this point in the history
  • Loading branch information
Vebop committed Nov 8, 2024
1 parent 13fe9af commit 9a1dc89
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 15 deletions.
19 changes: 19 additions & 0 deletions doc/operator-guide/monitoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,24 @@
Monitoring
###########

* Reporting channels

- Users of ConsDB, ConsDBClient (`pqserver`) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues.
- ConsDB operators should monitor this channel and #ops-usdf, #ops-usdf-alerts for issues and outages reported, as well as escalate verified database issues.

* Database

- The ConsDB team is responsible for verifying whether or not the database is up when issues are reported
- They can check the method reported by the users, check using `psql`/ `pgcli`, and check in the #ops-usdf slack channel for currently reported issues.

- Once the ConsDB team has confirmed there is an issue with the database, they should notify #ops-usdf slack channel and USDF DBAs should be responsible for fixing/restarting.

* REST API Server

- If we suspect the API server died, the ConsDB team should be responsible for checking and restarting
- Use the appropriate argo-cd deployment graph to check deployment logs, and potentially restart the service.


- Other issues
- K8s infrastructure died The ConsDB team can verify that that is the problem, but there are likely to be wider issues seen
- USDF or Summit K8s/IT support should be responsible for fixing.
36 changes: 21 additions & 15 deletions doc/user-guide/schemas.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,29 @@ Schemas
* Schema browser

* Versioning
Is the ConsDB versioning strategy outlined someplace in terms of when the major/minor/patch numbers in the schema versions should be incremented based on which changes are being made? (edited)
We have defined a versioning strategy for the services of monthly YY.0M.DD, but I am not clear if that lines up with the schemas or if they are separate
I believe it is completely separate because the schemas are using semantic versioning.
I thought you defined semantic versioning for sdm_schemas somewhere, but it should be consistent across all schemas, not just ConsDB.
https://rubin-obs.slack.com/archives/C07QJMQ7L4A/p1730482605167509

We've never defined this for sdm_schemas generically but it is definitely worth discussing.
- schemas are using semantic versioning
- should be consistent across all schemas, not just ConsDB

major: backward incompatible changes to the database objects (adding a table, deleting a column)
- except adding a table is not backwards incompatible
minor: backward compatible changes to the database objects (adding a column)
patch: updates or additions to semantics/metdata (units, UCDs, etc.)
- changing units can create incompatibilities

My own take is incrementing like:
major: backward incompatible changes to the database objects (adding a table, deleting a column)
minor: backward compatible changes to the database objects (adding a column)
patch: updates or additions to semantics/metdata (units, UCDs, etc.)
And we should say what should happen in the case of changes to primary/foreign keys.
Semantic neutrality: becoming non-primary is unique and anything becoming primary was already unique
- or there can be ones that are not neutral.

The DP schemas don't use versioning, though some other schemas like apdb do have meaningful versions. (edited)
The Felis data format basically leaves this open ended so that a version can be any string and their meaning is up to the schema author. (edited)
The only philosophical issue I have is that adding a table is not usually backward-incompatible, but it is a major change.
I guess there's also a philosophical issue in terms of changing units/UCDs/descriptions in ways that are not actually compatible (modifications rather than clarifications). If all of a sudden something is in mm rather than µm, it's not compatible.
Think about the utility of these versions in terms of interaction with the ConsDB APIs, migrations, etc.

And we should say what should happen in the case of changes to primary/foreign keys. Again, there can be changes (like recent ones) that are semantically neutral/backward-compatible (because anything becoming non-primary is still unique and anything becoming primary was already unique), or there can be ones that are not.
https://rubin-obs.slack.com/archives/C07QJMQ7L4A/p1730482605167509
Do sdm_schemas versions appear in the db?

Currently the schemas are tagged and versioned as a set, at least w.r.t. the Science Platform.
So once ConsDB is available in TAP, it should be part of that set.

What do users see, how does TAP play into this, do the schema need this type of micro versioning?


Services/cosndb repo versioning strategy - services of monthly YY.0M.DD

0 comments on commit 9a1dc89

Please sign in to comment.