Setup workflow to publish ooni data docs (#73)

This implements: #54

.github/workflows/build_docs.yml

@@ -0,0 +1,50 @@
+name: build docs
+on: push
+
+jobs:
+  build_docs:
+    runs-on: "ubuntu-20.04"
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Build docs
+        run: make docs
+
+      - name: Get current git ref
+        id: rev_parse
+        run: echo "COMMIT_HASH=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
+
+      - name: Checkout ooni/docs
+        uses: actions/checkout@v2
+        with:
+          repository: "ooni/docs"
+          ssh-key: ${{ secrets.OONI_DOCS_DEPLOYKEY }}
+          path: "ooni-docs"
+
+      - name: Update docs
+        run: |
+          mkdir -p ooni-docs/src/content/docs/data/
+          cp -R dist/docs/* ooni-docs/src/content/docs/data/
+
+      - name: Check for conflicting slugs
+        run: |
+          cat ooni-docs/src/content/docs/data/*.md \
+            | grep "^slug:" | awk -F':' '{gsub(/^ +/, "", $2); print $2}' | sort | uniq -c \
+            | awk '{if ($1 > 1) { print "duplicate slug for: " $2; exit 1}}'
+
+      - name: Print the lines of the generated docs
+        run: wc -l ooni-docs/src/content/docs/data/*
+
+      - name: Commit changes
+        # Only push the docs update when we are in master
+        # if: github.ref == 'refs/heads/master'
+        run: |
+          cd ooni-docs
+          git config --global user.email "[email protected]"
+          git config --global user.name "OONI Github Actions Bot"
+          git add .
+          git commit -m "auto: update ooni/data docs to ${{ steps.rev_parse.outputs.COMMIT_HASH }}" || echo "No changes to commit"
+          git push origin

Makefile

@@ -0,0 +1,7 @@
+docs:
+	./scripts/build_docs.sh
+
+clean:
+	rm -rf dist/
+
+.PHONY: docs clean

Readme.md

@@ -1,26 +1,117 @@
-## OONI Data
+There are different ways to access OONI data, wether that is via: [OONI
+Explorer](https://explorer.ooni.org/), the [OONI API](https://api.ooni.io/) or
+clickhouse table dumps.
 
-OONI Data is a collection of tooling for downloading, analyzing and interpreting
-OONI network measurement data.
+The [OONI API](https://api.ooni.io/) is meant for developers and researches and allows [searching for
+measurement metadata](https://api.ooni.io/apidocs/#/default/get_api_v1_measurements), [fetching single measurements](https://api.ooni.io/apidocs/#/default/get_api_v1_measurement_meta), and [generating statistics](https://api.ooni.io/apidocs/#/default/get_api_v1_aggregation).
 
-Most users will likely be interested in using this as a CLI tool for downloading
-measurements.
+**Hovever the OONI API, is not designed for large data transfers (i.e. extracting tens of thousands of measurements or many GB of data) and implements rate limiting API.**
+If you are interested in a dump of the clickhouse tables, please [reach out to us](https://ooni.org/about/) instead of scraping our API.
 
-If that is your goal, getting started is easy, run:
+Researchers can access the raw measurement data from an S3 bucket. The
+specifications of the OONI data formats can be found in
+[ooni/spec](https://github.com/ooni/spec).
 
+## Accessing raw measurement data
+
+"Raw measurement data" refers to data structures uploaded by OONI Probes (run by volunteers worldwide) to the
+processing pipeline.
+
+Thanks to the [Amazon Open Data program](https://aws.amazon.com/government-education/open-data/), the whole OONI dataset
+can be fetched from the [`ooni-data-eu-fra` Amazon S3 bucket](https://ooni-data-eu-fra.s3.eu-central-1.amazonaws.com/).
+
+A single chunk of data is called "a measurement" and its uncompressed size can vary between 1KB to 1MB, roughly.
+
+Probes usually upload multiple measurements on each execution. Measurements are stored temporarily and then batched together, compressed and uploaded to the S3 bucket once every hour. To ensure transparency, incoming measurements go through basic content validation and the API returns success or error;
+once a measurement is accepted it will be published on S3.
+
+OONI measurements are also processed by the fastpath and made immediately available on OONI Explorer. See the "receive_measurement" function in the probe_services.py file in the API codebase for details.
+
+The commands which follow will be using the [aws s3 cli
+tool](https://aws.amazon.com/cli/). See [their documentation on how to install
+it](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
+
+Since [OONI data is part of the AWS Open Data
+program](https://registry.opendata.aws/ooni/), you don't have to pay for access
+and you can use the `--no-sign-request` flag to access it for free.
+
+## File paths in the S3 bucket in JSONL format
+
+Contains a JSON document for each measurement, separated by newline and compressed, for easy processing.
+The path structure allows to easily select, identify and download data based on the researcher's needs.
+
+In the path template:
+- `cc` is an uppercase [2 letter country code](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2)
+- `testname` is a test name where underscores are removed
+- `timestamp` is a `YYYYMMDD` timestamp
+- `name` is a unique filename
+
+### Compressed JSONL from measurements starting from 2020-10-20
+
+The path structure is: `s3://ooni-data-eu-fra/raw/<timestamp>/<hour>/<cc>/<testname>/<ts2>_<cc>_<testname>.<host_id>.<counter>.jsonl.gz`
+
+Example: `s3://ooni-data-eu-fra/raw/20210817/15/US/webconnectivity/2021081715_US_webconnectivity.n0.0.jsonl.gz`
+
+Note: The path will be updated in the future to live under `/jsonl/`
+
+Listing JSONL files:
+```
+aws s3 --no-sign-request ls \
+    s3://ooni-data-eu-fra/raw/20210817/15/US/webconnectivity/
 ```
-pip install oonidata
+
+#### Downloading entire dates
+
+If you would like to download the raw measurements for a particular country,
+you can use the `aws s3 sync` command.
+
+For example to download all measurements from Italy on the 1st of February 2024, you can run:
+```
+aws s3 --no-sign-request sync \
+    s3://ooni-data-eu-fra/raw/20240201/ ./ \
+    --exclude "*" --include "*/IT/*"
+```
+
+**Note**: the difference in paths compared to older data
+
+### Compressed JSONL from measurements before 2020-10-21
+
+The path structure is: `s3://ooni-data-eu-fra/jsonl/<testname>/<cc>/<timestamp>/00/<name>.jsonl.gz`
+
+Example: `s3://ooni-data-eu-fra/jsonl/webconnectivity/IT/20200921/00/20200921_IT_webconnectivity.l.0.jsonl.gz`
+
+Listing JSONL files:
 ```
+aws s3 --no-sign-request ls s3://ooni-data-eu-fra/jsonl/
+aws s3 --no-sign-request ls \
+    s3://ooni-data-eu-fra/jsonl/webconnectivity/US/20201021/00/
+```
+
+#### Downloading entire dates
 
-You will then be able to download measurements via:
+If you would like to download the raw measurements for a particular country,
+you can use the `aws s3 sync` command.
 
+For example to download webconnectivity measurements from Italy on the 1st of February 2024, you can run:
 ```
-oonidata sync --probe-cc IT --start-day 2022-10-01 --end-day 2022-10-02 --output-dir measurements/
+aws s3 --no-sign-request sync \
+    s3://ooni-data-eu-fra/jsonl/webconnectivity/IT/20200201/ ./ \
+    --exclude "*" \
+    --include "*"
 ```
 
-This will download all OONI measurements for Italy into the directory
-`./measurements` that were uploaded between 2022-10-01 and 2022-10-02.
+**Note**: the difference in paths compared to newer data
 
-### OONI Pipeline
+## Raw "postcans" from measurements starting from 2020-10-20
 
-For documentation on OONI Pipeline v5, see the subdirectory `oonipipeline`.
+A "postcan" is tarball containing measurements as they are uploaded by the probes, optionally compressed.
+Each HTTP POST is stored in the tarball as `<timestamp>_<cc>_<testname>/<timestamp>_<cc>_<testname>_<hash>.post`
+
+Example: `s3://ooni-data-eu-fra/raw/20210817/11/GB/webconnectivity/2021081711_GB_webconnectivity.n0.0.tar.gz`
+
+Listing postcan files:
+```
+aws s3 --no-sign-request ls s3://ooni-data-eu-fra/raw/20210817/
+aws s3 --no-sign-request ls \
+    s3://ooni-data-eu-fra/raw/20210817/11/GB/webconnectivity/
+```

oonidata/Readme.md

@@ -0,0 +1,22 @@
+# OONI Data
+
+OONI Data is a collection of tooling for downloading, analyzing and interpreting
+OONI network measurement data.
+
+Most users will likely be interested in using this as a CLI tool for downloading
+measurements.
+
+If that is your goal, getting started is easy, run:
+
+```
+pip install oonidata
+```
+
+You will then be able to download measurements via:
+
+```
+oonidata sync --probe-cc IT --start-day 2022-10-01 --end-day 2022-10-02 --output-dir measurements/
+```
+
+This will download all OONI measurements for Italy into the directory
+`./measurements` that were uploaded between 2022-10-01 and 2022-10-02.

scripts/build_docs.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+DOCS_ROOT=dist/docs/
+REPO_NAME="ooni/data"
+COMMIT_HASH=$(git rev-parse --short HEAD)
+
+mkdir -p $DOCS_ROOT
+
+strip_title() {
+    # Since the title is already present in the frontmatter, we need to remove
+    # it to avoid duplicate titles
+    local infile="$1"
+    cat $infile | awk 'BEGIN{p=1} /^#/{if(p){p=0; next}} {print}'
+}
+
+cat <<EOF>$DOCS_ROOT/00-index.md
+---
+# Do not edit! This file is automatically generated
+# version: $REPO_NAME:$COMMIT_HASH
+title: Accessing OONI data
+description: How to access OONI data
+slug: data
+---
+EOF
+strip_title Readme.md >> $DOCS_ROOT/00-index.md
+
+cat <<EOF>$DOCS_ROOT/01-oonidata.md
+---
+# Do not edit! This file is automatically generated
+# version: $REPO_NAME:$COMMIT_HASH
+title: OONI Data CLI
+description: Using the oonidata command line interface
+slug: data/oonidata
+---
+EOF
+strip_title oonidata/Readme.md >> $DOCS_ROOT/01-oonidata.md
+
+cat <<EOF>$DOCS_ROOT/02-pipeline.md
+---
+# Do not edit! This file is automatically generated
+# version: $REPO_NAME:$COMMIT_HASH
+title: OONI Data Pipepline
+description: OONI Data Pipeline documentation
+slug: data/pipeline
+---
+EOF
+strip_title oonipipeline/Readme.md >> $DOCS_ROOT/02-pipeline.md