diff --git a/README.md b/README.md index 710a4055..11aea770 100644 --- a/README.md +++ b/README.md @@ -8,74 +8,7 @@ This repo houses (hopefully) everything related to doing graphics overlays for Y ## Developing -You will need [Node.js](https://nodejs.org/en/) and [Yarn](https://yarnpkg.com/getting-started/install). If you want to work on the API backend, you will also need a [Couchbase Server](https://docs.couchbase.com/server/current/install/install-intro.html) (7.1 or later) - you can also use [Docker](https://docs.couchbase.com/server/current/install/getting-started-docker.html) for this. Finally, you will need [Redis](https://redis.io/docs/getting-started/). - -There is a docker-compose.yml file that should set all this up if you run `docker compose up` - though it hasn't been updated in a while so might not fully work. - -First, install NodeCG - we're using our own fork that has some small fixups: - -```shell -$ git clone --branch ystv https://github.com/ystv/nodecg.git -``` - -Then, clone this repo inside the `bundles` folder: - -```shell -$ git clone https://github.com/ystv/ystv-sports-graphics.git bundles/ystv-sports-graphics -``` - -Open it and run `yarn`. - -### Backend/UI - -`cd scores-src`. - -Create a .env file that looks something like this: - -``` -DB_CONNECTION_STRING=couchbase://your.couchbase.server -DB_USER=sports-scores -DB_PASSWORD=password -DB_BUCKET=sports-scores -REDIS_CONNECTION_STRING=redis://localhost -PUBLIC_API_BASE=http://localhost:8000/api -``` - -In Couchbase, create a bucket and user called `sports-scores` (or whatever you used above). - -Then, run `yarn dev` and go to `http://localhost:3000`. If something didn't work check the console for clues. - -### Graphics - -Create a file in the NodeCG `cfg` directory called `ystv-sports-graphics.json` that looks like this: - -```json -{ - "scoresService": { - "apiURL": "http://localhost:8000/api" - } -} -``` - -Optionally, create a `nodecg.json` in the same place that looks like this: - -```json -{ - "logging": { - "console": { - "enabled": true, - "timestamps": true, - "level": "debug" - } - } -} -``` - -This will ensure you get logging. - -`cd bundle-src` and run `yarn bundle:dev` to start a live-reloading dev server, or `yarn bundle:build` to build the bundle once. - -In a second terminal, run `yarn nodecg` (in the `ystv-sports-graphics` folder) and go to `http://localhost:9090`. +Take a look at the [docs folder](./docs) for instructions on getting started, as well as an overview of the codebase. ## Deploying diff --git a/bundle-src/package.json b/bundle-src/package.json index 78d19ff5..ec3af46b 100644 --- a/bundle-src/package.json +++ b/bundle-src/package.json @@ -13,7 +13,8 @@ "build": "cross-env NODE_ENV=production webpack", "bundle:storybook": "start-storybook -p 6006 --no-open", "build-storybook": "build-storybook", - "bundle:schema": "mkdir -p schemas && json2ts -i schemas/ -o src/common/types/ && json2ts -i configschema.json -o src/common/types/config.d.ts" + "bundle:schema": "mkdir -p schemas && json2ts -i schemas/ -o src/common/types/ && json2ts -i configschema.json -o src/common/types/config.d.ts", + "nodecg": "node ../../../index.js" }, "repository": "https://github.com/ystv/ystv-sports-graphics.git", "private": true, diff --git a/docs/00-getting-started.md b/docs/00-getting-started.md new file mode 100644 index 00000000..60ca497c --- /dev/null +++ b/docs/00-getting-started.md @@ -0,0 +1,140 @@ +# Getting Started + +There's a few things you'll need on your laptop before you can start hacking on sports-graphics: + +- Node.js +- Yarn +- Couchbase Server +- Redis +- NodeCG +- Git +- The code itself + +In addition, while it's not necessary, we recommend using [Docker](https://www.docker.com/get-started/) to install Couchbase Server, especially if you're on a M1 MacBook. + +It sounds like a long list, but you only need to go through it once and you're set! Let's get started. + +## Node.js + +Go to the [Node](https://nodejs.org/en/) website and download the latest LTS (long term support) version. Alternatively if you're on a Mac and have [Homebrew](http://homebrew.sh/) you can run `brew install node@16`. + +## Yarn + +Yarn is a package manager for Node, specifically an alternative to the usual npm. There's a few reasons this project uses Yarn (mostly "the original developers prefer it to npm"). + +Follow the [Yarn installation instructions](https://yarnpkg.com/getting-started/install) to get it. + +## Couchbase Server + +Couchbase Server is the database that we use for storing event data. For this example we'll install it using Docker, though you can install it [directly](https://www.couchbase.com/downloads) (except on a M1 MacBook where Docker is the only way currently). + +First, create a volume to keep the data around between restarts: + +```sh +$ docker volume create cbdata +``` + +Then launch the container: + +```sh +$ docker run -d -v cbdata:/opt/couchbase/var/lib/couchbase --restart=always -p 8091-8096:8091-8096 -p 11207-11211:11207-11211 -p 18091-18096:18091-18096 --name cb couchbase/server:community-7.1.1 +``` + +Then go to http://localhost:8091 (refresh a few times if you get nothing). Select "Setup New Cluster" and walk through the setup steps until you get to the Configure screen. Change the Data memory quota to 512MB, and leave everything else at the defaults. + +Then go to the Buckets tab on the left and click Add Bucket on the top-right. Name it `sports-scores` and leave everything else as the defaults. + +Finally, go into the Security tab, and create a user with the username `sports-scores`, password `password`, and Full Admin permissions. It should look like this: + +![](./images/00-cb-user.png) + +## Redis + +Redis is a "micro-database" that some things (namely real-time updates) rely on. + +We'll also install Redis using Docker. Run the following command, and you're all set. + +``` +$ docker run -d --restart=always -p 6379:6379 redis:6-alpine +``` + +## Git + +Git is the system we use to manage our code and combine the work of multiple developers. Get it from the [Git website](https://git-scm.com/downloads). + +## NodeCG + +NodeCG is the framework that powers the graphics themselves, and our code needs to be placed inside of it. YSTV uses a fork of NodeCG to work around some issues in the upstream version. + +Get the NodeCG code: + +```shell +$ git clone --branch ystv https://github.com/ystv/nodecg.git +``` + +Inside the `nodecg` folder it just created, run `npm ci --production` to install NodeCG's dependencies. + +While we're here, let's set up some bits we'll need later: in the `cfg` folder create a file called `ystv-sports-graphics.json` with the following contents: + +```json +{ + "scoresService": { + "apiURL": "http://localhost:8000/api" + } +} +``` + +Create another file in the `cfg` folder called `nodecg.json` with these contents: + +```json +{ + "logging": { + "console": { + "enabled": true, + "timestamps": true, + "level": "debug" + } + } +} +``` + +## sports-scores code + +Go into the `nodecg/bundles` folder and run this command: + +``` +git clone https://github.com/ystv/ystv-sports-graphics.git +``` + +Open `ystv-sports-graphics` in your code editor of choice. + +# Check it all worked + +First, to get all the dependencies our code is built upon, invoke Yarn: + +```sh +$ yarn +``` + +Then, configure it to talk to your Couchbase and Redis. Inside the `scores-src` folder, create a file called `.env.local` and put in the following values: + +``` +DB_CONNECTION_STRING=couchbase://localhost +DB_USER=sports-scores +DB_PASSWORD=password +DB_BUCKET=sports-scores +REDIS_CONNECTION_STRING=redis://localhost +``` + +Next, run the code and see if it works! + +```sh +# inside the scores-src folder +$ yarn dev +``` + +Watch the output and look for a line starting with `No existing application data found.`. Copy the long token at the end, and go to http://localhost:3000/bootstrap. Enter that token there, then set up a user (the username and password don't matter). Then you should be able to get to the home screen! + +Finally, let's check that we have NodeCG set up. Open a new terminal, switch to the `bundle-src` folder, and run `yarn bundle:build`, then `yarn nodecg`, then go to http://localhost:9090. + +You're all set! In the [next part](./01-code-structure.md), we'll go on a quick tour of how the code is laid out. diff --git a/docs/01-code-structure.md b/docs/01-code-structure.md new file mode 100644 index 00000000..b10e2f4f --- /dev/null +++ b/docs/01-code-structure.md @@ -0,0 +1,62 @@ +# Code Structure + +The two most important folders in the codebase are `bundle-src` and `scores-src`, so let's talk about all the others first: + +- `.devcontainer`: used to have a [dev container](https://code.visualstudio.com/docs/devcontainers/containers) definition, except it's grown outdated. We should fix it up one day. +- `.github/workflows`: definitions of our GitHub Actions - discussed later in [Testing](./03-testing.md) +- `.husky`: can be ignored +- `dashboard`, `graphics`: can be ignored (NodeCG requires that they exist at the top level, though they're actually built from `bundle-src`) +- `patches`: local modifications to libraries to work around upstream issues +- `schemas`: a symbolic link to `bundle-src/schemas`, discussed later +- `scripts`: miscellaneous scripts + +Also there's quite a few files at the top level: + +- `.dockerignore`, `.gitignore`, `.prettierignore`: telling various tools which files to not care about +- `.editorconfig`: configures code editors to all follow the same coding style (indent size, line endings, etc.) +- `.eslintrc.js`: configures ESLint, a tool that checks the JavaScript/TypeScript for quality +- `.yarnrc.yml`: configures the Yarn package manager +- `Dockerfile.*`, `client-nginx.conf`: used to build the Docker images - discussed in [Deployment](./05-deployment.md) +- `docker-compose.yml`: used to be used for quickly setting up a development environment, but is outdated now +- `cypress.json`: configures the Cypress test runner - discussed later in [Testing](./03-testing.md) +- `Jenkinsfile`, `Jenkinsfile.prod-release`: used by the Jenkins build automation - discussed in [Deployment](./05-deployment.md) +- `package.json`, `yarn.lock`: specifies all the dependencies of the code + - Note that `bundle-src` and `scores-src` have their own `package.json` files, that are combined using [Yarn Workspaces](https://yarnpkg.com/features/workspaces) + +## scores-src + +scores-src houses the scores management and data entry application. It contains both the client-side and server-side code, as well as some that is common to both. Each of these lives in the folders `client`, `server`, and `common` respectively. + +### common + +This has a few top-level files that are useful across both `client` and `server`, for example calculating times. However the most interesting part of `common` is the `sports` subfolder, which has definitions of each of the sport types tha tthe system supports. The exact contents are discussed further in the [Data Model](./02-data-model.md) section. + +### server + +On the server-side the main entry point is `index.server.ts`, which sets up the application's Web server. In the process it imports (among many others): + +- `loggingSetup.ts`, which sets up logging +- `config.ts`, where the structure of the server's configuration is defined + - The actual configuration is in `scores-src/config/{NODE_ENV}.json`, where `{NODE_ENV}` is the value of the `NODE_ENV` environment variable - if none is set it uses the values in `defaults.json`. +- `db.ts`, which houses the logic for connecting to Couchbase Server +- `redis.ts`, ditto for Redis +- `...Routes.ts`, which define the various "routes" (API endpoints), such as `/events`, `/teams` etc. Some of these are discussed in more detail below, but the general principle is that each one exports a function called `createXxxRouter()`, which returns an Express [`Router`](https://expressjs.com/en/guide/routing.html), which `index.ts` combines together. +- `metrics.ts`, which defines our Prometheus metrics, used for debugging issues. +- `bootstrap.ts`, which is responsible for handling setting up the application for the first time. + +There's a few other files in the `server` folder which merit calling out: + +- `auth.ts` has the user authentication logic, including checking passwords and setting session cookies. +- `*.spec.ts` - [tests](./03-testing.md) +- `__mocks__` - mock files for [testing](./03-testing.md) + +The app's functionality is exposed through various APIs (for example, `localhost:8000/api/events`) that all take in and return JSON. The most important ones are: + +- `/events` (`eventsRoutes.ts`) - handles listing events. Notably it doesn't handle things like creating or updating events, instead those are handled by... +- `/events//` (`eventTypeRoutes.ts`) - this one is a little interesting, because it creates a bunch of routers that all do more or less the same thing. Essentially, for each event type (such as `football`, `swimming`, etc.) we create a router at `/events//football`, `/events//swimming`, which handles creating, updating, and deleting events of that type. There's no real reason why this couldn't be a single generic router that determines the event based on the path, this was just simpler. +- `/updates/stream/v2` (`liveRoutes.ts`) - handles WebSocket connections to get real-time notifications about event changes. + - (Historical note: the `v2` comes from [the first iteration of this system](https://github.com/ystv/roses-scores-api/tree/main/update_stream), which also had a `v1`. This one only has `v2`.) + +### client + +The client-side is the part that humans will interact with. It's a relatively typical React app, using React Router for client-side routing (NB: completely unrelated to routers in the `server` section). Everything kicks off in `index.html`, which (through [build-time magic](https://vitejs.dev/)) loads `index.client.tsx` and renders the `App` component from `App.tsx`. diff --git a/docs/02-data-model.md b/docs/02-data-model.md new file mode 100644 index 00000000..2020c6e4 --- /dev/null +++ b/docs/02-data-model.md @@ -0,0 +1,90 @@ +# Data Model + +## NoSQL recap + +As you will have seen, Sports Graphics stores data using Couchbase Server, which is a NoSQL database. This may be a little different to other databases you may have used, especially relational (SQL) ones, so it's perhaps worth briefly recapping the idea. + +In a NoSQL database there's no defined structure to your data - Couchbase Server groups it into "buckets"[^1] but that's about it. The only thing to identify your data is a so-called "key", a unique reference to a bit of data (called a "document"). Given a document's key you can get and update its value - blindingly quickly (as in, sub-millisecond at times). + +The contents of a document can be anything, but most documents are JSON - though there are a few that aren't which are called out below. Beyond that though, there's no rules about what this JSON can store, which we take advantage of. + +Couchbase Server does also allow us to query the contents of the documents using a language called [SQL++](https://docs.couchbase.com/server/current/getting-started/try-a-query.html), which should be familiar if you've used SQL. + +[^1]: Couchbase does allow us to further segment the data into scopes and collections, though at the time of writing we only use the default collection. + +## Sports Scores data + +Given all that, how do we use Couchbase? This is perhaps best answered by listing all the "families" of documents we have, grouped by key: + +- `League/`: stores some basic information about "leagues", which are used to group together events (such as all BUCS or Roses events) to make browsing easier +- `EventMeta///`: stores some basic information about an event, such as the name, time, and teams that are playing +- `EventHistory///`: stores the full history of everything that happened at an event - discussed further below +- `Team/`: stores information about a team +- `Attachment/`: used to store team crest images (NB: this is not JSON - we store the raw contents of the image file as the document body, and the file format as the [extended attribute](https://docs.couchbase.com/server/current/learn/data/extended-attributes-fundamentals.html) `mimeType`) +- `User/`: stores information about a user who can access the system, including their name, hash of their password, and what level of access they have. +- `Session/`: used to sign in a user (NB: technically JSON, but the value is only a string representing the user's username) +- `BootstrapState`: used to remember whether this instance of Sports Graphics has been fully set up + +## Event data model + +Briefly mentioned above, but the information for an event is split up into the "metadata" and "history". The metadata stores some basic information like the name, start time, and participating teams, while the history is the full sequence of events. + +The history is the more interesting of the two: it's one big JSON array with objects containing everything that has happened in a match, for example goals or timer starts/stops. Here's an (abridged) example: + +```json +[ + { + "type": "@@init", + "payload": { + // ... + "clock": { + "startingTime": 900000, + "timeLastStartedOrStopped": 0, + "wallClockLastStarted": 0, + "state": "stopped", + "type": "downward" + }, + "scoreAway": 0, + "scoreHome": 0, + "players": { + "away": [], + "home": [] + } + }, + "meta": { + "ts": 1668253910811 + } + }, + { + "type": "netball/startNextQuarter", + "payload": {}, + "meta": { + "ts": 1668263425000 + } + }, + , + { + "type": "netball/goal", + "payload": { + "side": "home", + "player": null + }, + "meta": { + "ts": 1668263941336 + } + } + // ... +] +``` + +Each entry in the history is an object, sometimes referred to as an "action" (if you've written applications that use [Redux](https://redux.js.org/) this will be _very_ familiar - though don't worry if not!), which represents, more or less, "something happened". In the above example, the event was created, a quarter started, and the "home" side scored a goal. + +So how does the system get from that array to "the score is James 1 Derwent 0"? It runs the array of actions through a "reducer" function - the one for Netball lives in [common/sports/netball/index.tsx](../scores-src/src/common/sports/netball/index.tsx). It takes the current state and an action, and returns the new state - for example, given the state `{"scoreHome": 0, "scoreAway": 0}` and the action `{"type": "netball/goal", "payload": { "side": "home" }}`, it will return the state `{"scoreHome": 1, "scoreAway": 0}`. + +But what's the point of going through all this hassle? In a word (or two), time travel! + +If, for example, whoever's pitchside happens to make a mistake and enters a Derwent goal when James scored, they can simply undo the goal action, and the system will re-calculate the score while pretending that the mistaken goal never happened. Alternatively they can edit the action post-factum and the system will re-calculate the score with the corrected goal information. This wouldn't be possible without this data model. + +This is enabled by the `meta.ts` ("time stamp") field, which is also used as an identifier for a specific action - you'll see `ts` thrown around a lot in the codebase. + +So, after all that, why split the metadata and history? In fact, we didn't use to, and only stored the history array, however writing queries for it (such as "give me all events where James College are playing") is a bit of a pain on arrays like that, so the decision was made to split the two. diff --git a/docs/03-testing.md b/docs/03-testing.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/04-workflow.md b/docs/04-workflow.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/05-deployment.md b/docs/05-deployment.md new file mode 100644 index 00000000..e69de29b diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..5c9ec925 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,18 @@ +# Sports Graphics Developer Docs + +If you want to hack on the sports graphics application, or just curious about how it's put together, you've come to the right place! + +These documents are meant to be read in a roughy chronological order, though you can skip back and forth if you like: + +- [Getting Started](./00-getting-started.md) - setting up a local copy of the system to work on +- [Code Structure](./01-code-structure.md) - how things are laid out and where to look for what +- [Data Model](./02-data-model.md) - where we store data and how it's structured +- [Testing](./03-testing.md) - the various bits of automation that check your code for mistakes +- [Workflow](./04-workflow.md) - how to get the code from your laptop into the main codebase +- [Deployment](./05-deployment.md) - the process of that code getting into users' hands + +There's also a rundown of the [data sync protocol](./a-live-protocol.md) - this isn't required reading but may be useful. + +## Assumed Knowledge + +We will assume that you know at least the basics of JavaScript and React.js. If you don't, there's plenty of resources online that teach it far better than we ever could - for React specifically, the [beta documentation](https://beta.reactjs.org/) provides a great introduction to the key concepts. diff --git a/docs/a-live-protocol.md b/docs/a-live-protocol.md new file mode 100644 index 00000000..3fa6eb36 --- /dev/null +++ b/docs/a-live-protocol.md @@ -0,0 +1,139 @@ +# Appendix A: Live Protocol + +The Sports Graphics live data protocol uses JSON over WebSockets. Every object sent by either the client or server must have a `kind` property. + +The API can operate in two modes: + +- in "state mode", whenever an event changes, the client is sent the complete state of the event right now +- in "actions mode", whenever an event changes, the client is sent an object describing the change (per the [data model](./02-data-model.md)) + +## Connecting + +The client should open a WebSocket connection to `https:///api/updates/stream/v2?token=&mode=`, where `` is a valid session ID and `mode` is either `state` or `actions` (discussed below). If no mode is specified, `state` will be used. Optionally it can supply the `sid` and `last_mid` query parameters, discussed later. + +If the user is valid, the server will send a `HELLO` message: + +```json +{ + "kind": "HELLO", + "sid": "", + "subs": [], + "mode": "" +} +``` + +`sid` will be a random string that the client should remember for later use. `subs` will be all the event IDs that the client is subscribed to, which will be an empty array the first time it connects. + +The server will now start periodically sending `PING` messages: + +```json +{ "kind": "PING" } +``` + +The client must reply to each `PING` with a `PONG`, or the server will close the connection: + +```json +{ "kind": "PONG" } +``` + +The client can also send its own `PING`s, to which the server will reply with `PONG`s. + +## Getting Event Updates + +To subscribe to an event the client sends a `SUBSCRIBE` message: + +```json +{ + "kind": "SUBSCRIBE", + "to": "Event///" +} +``` + +If successful the server will reply with a `SUBSCRIBE_OK` message: + +```json +{ + "kind": "SUBSCRIBE_OK", + "to": "Event///", + "current": {} +} +``` + +The value of `current` depends on the mode: + +- in state mode it is an object representing the current state of the event +- in actions mode it is an array with the full actions history so far + +The server will now start sending messages whenever the event changes. In state mode it will send `CHANGE` messages: + +```json +{ + "kind": "CHANGE", + "changed": "Event///", + "mid": "", + "data": {} +} +``` + +`data` is the current state of the event. + +In actions mode the server will send `ACTION` messages: + +```json +{ + "kind": "ACTION", + "event": "Event///", + "mid": "", + "type": "", + "payload": {}, + "meta": {} +} +``` + +`payload` and `meta` are the action payload and metadata. + +In both modes, `mid` is the Message ID - the client should remember the last `mid` it receives. + +To unsubscribe, the client can send a `UNSUBSCRIBE`: + +```json +{ + "kind": "UNSUBSCRIBE", + "to": "Event///" +} +``` + +The server will reply with a `UNSUBSCRIBE_OK`. + +### Resyncing + +A resync is when the server sends the client the complete event state (in state mode) or complete actions history (in actions mode). The client can request a resync by sending a `RESYNC` message: + +```json +{ + "kind": "RESYNC", + "what": "Event///" +} +``` + +The server can also initiate a resync at any time. In either case, the server will either send the client a `CHANGE` as above (in state mode), or a `BULK_ACTIONS` (in actions mode): + +```json +{ + "kind": "BULK_ACTIONS", + "event": "Event///", + "actions": [] +} +``` + +The client is expected to discard its knowledge of the action history and use the `actions` it has just received. + +## Reconnecting + +The internet is a scary place and connections can be lost. The server supports replaying the messages that a client missed while it was disconnected. + +To do so, the client should establish a new connection as in [Connecting](#connecting), but also include the `sid` and `last_mid` query parameters, which should be the `sid` the server first sent on connecting and the `mid` of the last message it received before disconnecting. + +If successful, the server will send a `HELLO` with a matching `sid`, followed by all the `CHANGE`s or `ACTION`s the client missed. The client will be re-subscribed with no need to send a `SUBSCRIBE` message. + +If unsuccessful, the server will send a `HELLO` with a different `sid`. In this case the client is expected to discard its knowledge of the event state and send a `SUBSCRIBE` as if connecting for the first time. diff --git a/docs/images/00-cb-user.png b/docs/images/00-cb-user.png new file mode 100644 index 00000000..11e84149 Binary files /dev/null and b/docs/images/00-cb-user.png differ