WIP: developer documentation

README.md

@@ -8,74 +8,7 @@ This repo houses (hopefully) everything related to doing graphics overlays for Y
 
 ## Developing
 
-You will need [Node.js](https://nodejs.org/en/) and [Yarn](https://yarnpkg.com/getting-started/install). If you want to work on the API backend, you will also need a [Couchbase Server](https://docs.couchbase.com/server/current/install/install-intro.html) (7.1 or later) - you can also use [Docker](https://docs.couchbase.com/server/current/install/getting-started-docker.html) for this. Finally, you will need [Redis](https://redis.io/docs/getting-started/).
-
-There is a docker-compose.yml file that should set all this up if you run `docker compose up` - though it hasn't been updated in a while so might not fully work.
-
-First, install NodeCG - we're using our own fork that has some small fixups:
-
-```shell
-$ git clone --branch ystv https://github.com/ystv/nodecg.git
-```
-
-Then, clone this repo inside the `bundles` folder:
-
-```shell
-$ git clone https://github.com/ystv/ystv-sports-graphics.git bundles/ystv-sports-graphics
-```
-
-Open it and run `yarn`.
-
-### Backend/UI
-
-`cd scores-src`.
-
-Create a .env file that looks something like this:
-
-```
-DB_CONNECTION_STRING=couchbase://your.couchbase.server
-DB_USER=sports-scores
-DB_PASSWORD=password
-DB_BUCKET=sports-scores
-REDIS_CONNECTION_STRING=redis://localhost
-PUBLIC_API_BASE=http://localhost:8000/api
-```
-
-In Couchbase, create a bucket and user called `sports-scores` (or whatever you used above).
-
-Then, run `yarn dev` and go to `http://localhost:3000`. If something didn't work check the console for clues.
-
-### Graphics
-
-Create a file in the NodeCG `cfg` directory called `ystv-sports-graphics.json` that looks like this:
-
-```json
-{
-  "scoresService": {
-    "apiURL": "http://localhost:8000/api"
-  }
-}
-```
-
-Optionally, create a `nodecg.json` in the same place that looks like this:
-
-```json
-{
-  "logging": {
-    "console": {
-      "enabled": true,
-      "timestamps": true,
-      "level": "debug"
-    }
-  }
-}
-```
-
-This will ensure you get logging.
-
-`cd bundle-src` and run `yarn bundle:dev` to start a live-reloading dev server, or `yarn bundle:build` to build the bundle once.
-
-In a second terminal, run `yarn nodecg` (in the `ystv-sports-graphics` folder) and go to `http://localhost:9090`.
+Take a look at the [docs folder](./docs) for instructions on getting started, as well as an overview of the codebase.
 
 ## Deploying
 

bundle-src/package.json

@@ -13,7 +13,8 @@
     "build": "cross-env NODE_ENV=production webpack",
     "bundle:storybook": "start-storybook -p 6006 --no-open",
     "build-storybook": "build-storybook",
-    "bundle:schema": "mkdir -p schemas && json2ts -i schemas/ -o src/common/types/ && json2ts -i configschema.json -o src/common/types/config.d.ts"
+    "bundle:schema": "mkdir -p schemas && json2ts -i schemas/ -o src/common/types/ && json2ts -i configschema.json -o src/common/types/config.d.ts",
+    "nodecg": "node ../../../index.js"
   },
   "repository": "https://github.com/ystv/ystv-sports-graphics.git",
   "private": true,

docs/00-getting-started.md

@@ -0,0 +1,140 @@
+# Getting Started
+
+There's a few things you'll need on your laptop before you can start hacking on sports-graphics:
+
+- Node.js
+- Yarn
+- Couchbase Server
+- Redis
+- NodeCG
+- Git
+- The code itself
+
+In addition, while it's not necessary, we recommend using [Docker](https://www.docker.com/get-started/) to install Couchbase Server, especially if you're on a M1 MacBook.
+
+It sounds like a long list, but you only need to go through it once and you're set! Let's get started.
+
+## Node.js
+
+Go to the [Node](https://nodejs.org/en/) website and download the latest LTS (long term support) version. Alternatively if you're on a Mac and have [Homebrew](http://homebrew.sh/) you can run `brew install node@16`.
+
+## Yarn
+
+Yarn is a package manager for Node, specifically an alternative to the usual npm. There's a few reasons this project uses Yarn (mostly "the original developers prefer it to npm").
+
+Follow the [Yarn installation instructions](https://yarnpkg.com/getting-started/install) to get it.
+
+## Couchbase Server
+
+Couchbase Server is the database that we use for storing event data. For this example we'll install it using Docker, though you can install it [directly](https://www.couchbase.com/downloads) (except on a M1 MacBook where Docker is the only way currently).
+
+First, create a volume to keep the data around between restarts:
+
+```sh
+$ docker volume create cbdata
+```
+
+Then launch the container:
+
+```sh
+$ docker run -d -v cbdata:/opt/couchbase/var/lib/couchbase --restart=always -p 8091-8096:8091-8096 -p 11207-11211:11207-11211 -p 18091-18096:18091-18096 --name cb couchbase/server:community-7.1.1
+```
+
+Then go to http://localhost:8091 (refresh a few times if you get nothing). Select "Setup New Cluster" and walk through the setup steps until you get to the Configure screen. Change the Data memory quota to 512MB, and leave everything else at the defaults.
+
+Then go to the Buckets tab on the left and click Add Bucket on the top-right. Name it `sports-scores` and leave everything else as the defaults.
+
+Finally, go into the Security tab, and create a user with the username `sports-scores`, password `password`, and Full Admin permissions. It should look like this:
+
+![](./images/00-cb-user.png)
+
+## Redis
+
+Redis is a "micro-database" that some things (namely real-time updates) rely on.
+
+We'll also install Redis using Docker. Run the following command, and you're all set.
+
+```
+$ docker run -d --restart=always -p 6379:6379 redis:6-alpine
+```
+
+## Git
+
+Git is the system we use to manage our code and combine the work of multiple developers. Get it from the [Git website](https://git-scm.com/downloads).
+
+## NodeCG
+
+NodeCG is the framework that powers the graphics themselves, and our code needs to be placed inside of it. YSTV uses a fork of NodeCG to work around some issues in the upstream version.
+
+Get the NodeCG code:
+
+```shell
+$ git clone --branch ystv https://github.com/ystv/nodecg.git
+```
+
+Inside the `nodecg` folder it just created, run `npm ci --production` to install NodeCG's dependencies.
+
+While we're here, let's set up some bits we'll need later: in the `cfg` folder create a file called `ystv-sports-graphics.json` with the following contents:
+
+```json
+{
+  "scoresService": {
+    "apiURL": "http://localhost:8000/api"
+  }
+}
+```
+
+Create another file in the `cfg` folder called `nodecg.json` with these contents:
+
+```json
+{
+  "logging": {
+    "console": {
+      "enabled": true,
+      "timestamps": true,
+      "level": "debug"
+    }
+  }
+}
+```
+
+## sports-scores code
+
+Go into the `nodecg/bundles` folder and run this command:
+
+```
+git clone https://github.com/ystv/ystv-sports-graphics.git
+```
+
+Open `ystv-sports-graphics` in your code editor of choice.
+
+# Check it all worked
+
+First, to get all the dependencies our code is built upon, invoke Yarn:
+
+```sh
+$ yarn
+```
+
+Then, configure it to talk to your Couchbase and Redis. Inside the `scores-src` folder, create a file called `.env.local` and put in the following values:
+
+```
+DB_CONNECTION_STRING=couchbase://localhost
+DB_USER=sports-scores
+DB_PASSWORD=password
+DB_BUCKET=sports-scores
+REDIS_CONNECTION_STRING=redis://localhost
+```
+
+Next, run the code and see if it works!
+
+```sh
+# inside the scores-src folder
+$ yarn dev
+```
+
+Watch the output and look for a line starting with `No existing application data found.`. Copy the long token at the end, and go to http://localhost:3000/bootstrap. Enter that token there, then set up a user (the username and password don't matter). Then you should be able to get to the home screen!
+
+Finally, let's check that we have NodeCG set up. Open a new terminal, switch to the `bundle-src` folder, and run `yarn bundle:build`, then `yarn nodecg`, then go to http://localhost:9090.
+
+You're all set! In the [next part](./01-code-structure.md), we'll go on a quick tour of how the code is laid out.

docs/01-code-structure.md

@@ -0,0 +1,62 @@
+# Code Structure
+
+The two most important folders in the codebase are `bundle-src` and `scores-src`, so let's talk about all the others first:
+
+- `.devcontainer`: used to have a [dev container](https://code.visualstudio.com/docs/devcontainers/containers) definition, except it's grown outdated. We should fix it up one day.
+- `.github/workflows`: definitions of our GitHub Actions - discussed later in [Testing](./03-testing.md)
+- `.husky`: can be ignored
+- `dashboard`, `graphics`: can be ignored (NodeCG requires that they exist at the top level, though they're actually built from `bundle-src`)
+- `patches`: local modifications to libraries to work around upstream issues
+- `schemas`: a symbolic link to `bundle-src/schemas`, discussed later
+- `scripts`: miscellaneous scripts
+
+Also there's quite a few files at the top level:
+
+- `.dockerignore`, `.gitignore`, `.prettierignore`: telling various tools which files to not care about
+- `.editorconfig`: configures code editors to all follow the same coding style (indent size, line endings, etc.)
+- `.eslintrc.js`: configures ESLint, a tool that checks the JavaScript/TypeScript for quality
+- `.yarnrc.yml`: configures the Yarn package manager
+- `Dockerfile.*`, `client-nginx.conf`: used to build the Docker images - discussed in [Deployment](./05-deployment.md)
+- `docker-compose.yml`: used to be used for quickly setting up a development environment, but is outdated now
+- `cypress.json`: configures the Cypress test runner - discussed later in [Testing](./03-testing.md)
+- `Jenkinsfile`, `Jenkinsfile.prod-release`: used by the Jenkins build automation - discussed in [Deployment](./05-deployment.md)
+- `package.json`, `yarn.lock`: specifies all the dependencies of the code
+  - Note that `bundle-src` and `scores-src` have their own `package.json` files, that are combined using [Yarn Workspaces](https://yarnpkg.com/features/workspaces)
+
+## scores-src
+
+scores-src houses the scores management and data entry application. It contains both the client-side and server-side code, as well as some that is common to both. Each of these lives in the folders `client`, `server`, and `common` respectively.
+
+### common
+
+This has a few top-level files that are useful across both `client` and `server`, for example calculating times. However the most interesting part of `common` is the `sports` subfolder, which has definitions of each of the sport types tha tthe system supports. The exact contents are discussed further in the [Data Model](./02-data-model.md) section.
+
+### server
+
+On the server-side the main entry point is `index.server.ts`, which sets up the application's Web server. In the process it imports (among many others):
+
+- `loggingSetup.ts`, which sets up logging
+- `config.ts`, where the structure of the server's configuration is defined
+  - The actual configuration is in `scores-src/config/{NODE_ENV}.json`, where `{NODE_ENV}` is the value of the `NODE_ENV` environment variable - if none is set it uses the values in `defaults.json`.
+- `db.ts`, which houses the logic for connecting to Couchbase Server
+- `redis.ts`, ditto for Redis
+- `...Routes.ts`, which define the various "routes" (API endpoints), such as `/events`, `/teams` etc. Some of these are discussed in more detail below, but the general principle is that each one exports a function called `createXxxRouter()`, which returns an Express [`Router`](https://expressjs.com/en/guide/routing.html), which `index.ts` combines together.
+- `metrics.ts`, which defines our Prometheus metrics, used for debugging issues.
+- `bootstrap.ts`, which is responsible for handling setting up the application for the first time.
+
+There's a few other files in the `server` folder which merit calling out:
+
+- `auth.ts` has the user authentication logic, including checking passwords and setting session cookies.
+- `*.spec.ts` - [tests](./03-testing.md)
+- `__mocks__` - mock files for [testing](./03-testing.md)
+
+The app's functionality is exposed through various APIs (for example, `localhost:8000/api/events`) that all take in and return JSON. The most important ones are:
+
+- `/events` (`eventsRoutes.ts`) - handles listing events. Notably it doesn't handle things like creating or updating events, instead those are handled by...
+- `/events/<league>/<type>` (`eventTypeRoutes.ts`) - this one is a little interesting, because it creates a bunch of routers that all do more or less the same thing. Essentially, for each event type (such as `football`, `swimming`, etc.) we create a router at `/events/<league>/football`, `/events/<league>/swimming`, which handles creating, updating, and deleting events of that type. There's no real reason why this couldn't be a single generic router that determines the event based on the path, this was just simpler.
+- `/updates/stream/v2` (`liveRoutes.ts`) - handles WebSocket connections to get real-time notifications about event changes.
+  - (Historical note: the `v2` comes from [the first iteration of this system](https://github.com/ystv/roses-scores-api/tree/main/update_stream), which also had a `v1`. This one only has `v2`.)
+
+### client
+
+The client-side is the part that humans will interact with. It's a relatively typical React app, using React Router for client-side routing (NB: completely unrelated to routers in the `server` section). Everything kicks off in `index.html`, which (through [build-time magic](https://vitejs.dev/)) loads `index.client.tsx` and renders the `App` component from `App.tsx`.

docs/02-data-model.md

@@ -0,0 +1,90 @@
+# Data Model
+
+## NoSQL recap
+
+As you will have seen, Sports Graphics stores data using Couchbase Server, which is a NoSQL database. This may be a little different to other databases you may have used, especially relational (SQL) ones, so it's perhaps worth briefly recapping the idea.
+
+In a NoSQL database there's no defined structure to your data - Couchbase Server groups it into "buckets"[^1] but that's about it. The only thing to identify your data is a so-called "key", a unique reference to a bit of data (called a "document"). Given a document's key you can get and update its value - blindingly quickly (as in, sub-millisecond at times).
+
+The contents of a document can be anything, but most documents are JSON - though there are a few that aren't which are called out below. Beyond that though, there's no rules about what this JSON can store, which we take advantage of.
+
+Couchbase Server does also allow us to query the contents of the documents using a language called [SQL++](https://docs.couchbase.com/server/current/getting-started/try-a-query.html), which should be familiar if you've used SQL.
+
+[^1]: Couchbase does allow us to further segment the data into scopes and collections, though at the time of writing we only use the default collection.
+
+## Sports Scores data
+
+Given all that, how do we use Couchbase? This is perhaps best answered by listing all the "families" of documents we have, grouped by key:
+
+- `League/<slug>`: stores some basic information about "leagues", which are used to group together events (such as all BUCS or Roses events) to make browsing easier
+- `EventMeta/<league>/<type>/<id>`: stores some basic information about an event, such as the name, time, and teams that are playing
+- `EventHistory/<league>/<type>/<id>`: stores the full history of everything that happened at an event - discussed further below
+- `Team/<slug>`: stores information about a team
+- `Attachment/<id>`: used to store team crest images (NB: this is not JSON - we store the raw contents of the image file as the document body, and the file format as the [extended attribute](https://docs.couchbase.com/server/current/learn/data/extended-attributes-fundamentals.html) `mimeType`)
+- `User/<username>`: stores information about a user who can access the system, including their name, hash of their password, and what level of access they have.
+- `Session/<id>`: used to sign in a user (NB: technically JSON, but the value is only a string representing the user's username)
+- `BootstrapState`: used to remember whether this instance of Sports Graphics has been fully set up
+
+## Event data model
+
+Briefly mentioned above, but the information for an event is split up into the "metadata" and "history". The metadata stores some basic information like the name, start time, and participating teams, while the history is the full sequence of events.
+
+The history is the more interesting of the two: it's one big JSON array with objects containing everything that has happened in a match, for example goals or timer starts/stops. Here's an (abridged) example:
+
+```json
+[
+  {
+    "type": "@@init",
+    "payload": {
+      // ...
+      "clock": {
+        "startingTime": 900000,
+        "timeLastStartedOrStopped": 0,
+        "wallClockLastStarted": 0,
+        "state": "stopped",
+        "type": "downward"
+      },
+      "scoreAway": 0,
+      "scoreHome": 0,
+      "players": {
+        "away": [],
+        "home": []
+      }
+    },
+    "meta": {
+      "ts": 1668253910811
+    }
+  },
+  {
+    "type": "netball/startNextQuarter",
+    "payload": {},
+    "meta": {
+      "ts": 1668263425000
+    }
+  },
+  ,
+  {
+    "type": "netball/goal",
+    "payload": {
+      "side": "home",
+      "player": null
+    },
+    "meta": {
+      "ts": 1668263941336
+    }
+  }
+  // ...
+]
+```
+
+Each entry in the history is an object, sometimes referred to as an "action" (if you've written applications that use [Redux](https://redux.js.org/) this will be _very_ familiar - though don't worry if not!), which represents, more or less, "something happened". In the above example, the event was created, a quarter started, and the "home" side scored a goal.
+
+So how does the system get from that array to "the score is James 1 Derwent 0"? It runs the array of actions through a "reducer" function - the one for Netball lives in [common/sports/netball/index.tsx](../scores-src/src/common/sports/netball/index.tsx). It takes the current state and an action, and returns the new state - for example, given the state `{"scoreHome": 0, "scoreAway": 0}` and the action `{"type": "netball/goal", "payload": { "side": "home" }}`, it will return the state `{"scoreHome": 1, "scoreAway": 0}`.
+
+But what's the point of going through all this hassle? In a word (or two), time travel!
+
+If, for example, whoever's pitchside happens to make a mistake and enters a Derwent goal when James scored, they can simply undo the goal action, and the system will re-calculate the score while pretending that the mistaken goal never happened. Alternatively they can edit the action post-factum and the system will re-calculate the score with the corrected goal information. This wouldn't be possible without this data model.
+
+This is enabled by the `meta.ts` ("time stamp") field, which is also used as an identifier for a specific action - you'll see `ts` thrown around a lot in the codebase.
+
+So, after all that, why split the metadata and history? In fact, we didn't use to, and only stored the history array, however writing queries for it (such as "give me all events where James College are playing") is a bit of a pain on arrays like that, so the decision was made to split the two.