Skip to content

Commit

Permalink
Refactor app setup and remove build app
Browse files Browse the repository at this point in the history
  • Loading branch information
jochenklar committed Dec 5, 2024
1 parent a797c2b commit 781961b
Show file tree
Hide file tree
Showing 18 changed files with 164 additions and 2,709 deletions.
5 changes: 4 additions & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,11 @@ jobs:
pip install --upgrade pip
pip install -r requirements.txt
- name: Build app 🔧
run: make app

- name: Run make 🔧
run: make
run: make prod

- name: Deploy 🚀
uses: JamesIves/github-pages-deploy-action@v4
Expand Down
4 changes: 1 addition & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ __pycache__/
.pytest_cache

/env
/venv
/*.sh

/output
.Rproj.user

deploy*.sh
24 changes: 20 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
export NVM_DIR="${PWD}/app/nvm"

.PHONY: prod dev serve assets links csvtables definitions glossary pattern protocol schema tree \
typos env app clean distclean

prod: assets csvtables definitions glossary pattern protocol schema tree

dev: links csvtables definitions glossary pattern protocol schema tree

serve:
python3 -m http.server -b 127.0.0.1 8080 -d output

clean:
rm -fr output

assets:
python3 build/assets.py

Expand Down Expand Up @@ -38,4 +40,18 @@ tree:
typos:
typos --write-changes --force-exclude

.PHONY: prod dev serve clean assets links csvtables definitions glossary pattern protocol schema tree typos
env:
python3 -m venv env
env/bin/pip install --upgrade pip
env/bin/pip install -r requirements.txt

app:
make -C app

clean:
rm -fr output

distclean:
rm -fr output
rm -fr env
make -C app clean
140 changes: 37 additions & 103 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,56 +14,49 @@ your updates will be visible at `https://protocol.isimip.org`.
As a rule, the sector-specific text should be kept to a minimum and cover
as much structure as possible by machine-readable code under [definitions](definitions/).

The React JavaScript app is build in a separate step which are described under [app/README.md]([app/README.md]).

Setup
-----

Building the protocol needs git and a recent Python version (> 3.6). The installation of Python (and its developing packages), however differs from operating system to operating system. Instructions can be found [here](https://github.com/ISI-MIP/isimip-qc/blob/main/README.md#prerequisites).

If you work with different Python applications, we recommend to create a virtual environment for the protocol.
A `Makefile` is provided to help with the installation process.

```bash
# setup venv on Linux/macOS/Windows WSL
python3 -m venv env
source env/bin/activate
If you work with different Python applications, we recommend to create a virtual environment for the protocol:

# setup venv on Windows cmd
python -m venv env
call env\Scripts\activate.bat
```bash
make env
```

The Python dependencies are installed using:
Otherwise, the Python dependencies are installed using:

```bash
pip install -r requirements.txt
```

The JavaScript part of the protocol needs to be build using NodeJS and Webpack. For convenience this can be done by using only:

```
make app
```


Build
-----

```bash
make # should work on Linux/macOS
make dev # like make, but for development of the JavaScript app
make dev # like make, but lining the front-end assets for development

make serve # starts a http server on port :8000 so that you can access the protocol in your browser

sh tools/build.sh # Linux/macOS/WSL
sh tools/serve.sh # Linux/macOS/WSL, start http server
call tools/build.cmd # Windows cmd
call tools/serve.cmd # Windows cmd, start http server
```

On Windows, a double click on `tools/build.cmd` should also build the protocol (unless you use a virtual environment).

The output files are located in `output`. The files, e.g. `index.html` can opened with a web browser.


Development server
------------------

The command `make serve` will open a local webserver on port `:8000`. The protocol can than be accessed at http://localhost:8000 from a browser.
The command `make serve` will open a local webserver on port `:8000`. The protocol can than be accessed at http://localhost:8080 from a browser.


Editing
Expand All @@ -79,96 +72,37 @@ The interactive tables have the following syntax:

where `number` is simply the table number to be displayed in the caption and `identifier` will not only connect the table to its definition file (see below), but will also define which JavaScript component to use. Changes of the layout of a table or the creation of new tables require work on the [app](app).

The definition JSON files however can be changed without touching the JavaScript source code. Each definition is a list of JSON objects. Every object must have an attribute `specifier` which is used to refer to it in other objects/tables but also in file names. An example for a relatively simple definition file is [definitions/bias_correction.json](definitions/bias_correction.json):
The definition YAML files however can be changed without touching the JavaScript source code. Each definition is a list of objects. Every object must have an attribute `specifier` which is used to refer to it in other objects/tables but also in file names. An example for a relatively simple definition file is [definitions/bias_adjustment.yaml](definitions/bias_adjustment.yaml):

```
[
{
"specifier": "nobc",
"description": "Indicates that no bias correction was performed on the climate data (e.g. ocean data)."
},
{
"specifier": "localbc",
"description": "Refers to local data from weather stations used for the bias-correction in e.g. the forest sector.",
"sectors": [
"forestry"
]
},
{
"specifier": "ewembi",
"description": "Refers to EWEMBI data used for the bias-correction globally on a 0.5° grid."
},
{
"specifier": "ewembi-isimip3basd",
"description": "Refers to EWEMBI data used for the bias-correction globally on a 0.5°, using improved bias-correction methods (Lange 2018, doi: 10.5194/esd-9-627-2018), and with statistical downscaling (instead of interpolation) of GCM data to the 0.5° grid prior to bias-correction."
}
]
- specifier: w5e5
description: >-
Refers to W5E5 data used for the bias-correction globally on a 0.5° grid.
- specifier: nobasd
description: >-
Indicates that no bias correction was performed on the climate data (e.g. ocean
data).
- specifier: localbc
description: >-
Refers to local data from weather stations used for the bias-correction in e.g.
the forest sector.
products:
- OutputData
- SecondaryOutputData
sectors:
- forestry
```

Here `localbc` only applies to the `forestry` sector, while the other objects are used in every sector. The longest and most complicated definition is [definitions/variable.json](definitions/variable.json):

```
[
{
"specifier": "qtot",
"title": "Runoff",
"unit": "kg m-2 s-1",
"resolution": "grid cell",
"frequency": {
"biomes": "monthly",
"permafrost": "monthly",
"water_global": "daily",
"water_regional": "daily"
},
"comment": "Total runoff leaving the land portion of the grid cell",
"sectors": [
"water_global",
"water_regional",
"biomes",
"permafrost"
]
},
...
]
```

Some attributes (e.g. `frequency`) can have objects as value, which the are evaluated for the particular sector. For reference, the full list for `simulation_rounds` and `sectors` are:

```
"simulation_rounds": [
"ISIMIP3a",
"ISIMIP3b"
],
```

```
"sectors": [
"agriculture",
"biodiversity",
"biomes",
"coastal",
"diarrhea",
"energy",
"forestry",
"health",
"labour",
"lakes_global",
"lakes_local",
"marine-fishery_global",
"marine-fishery_regional",
"permafrost",
"water_global",
"water_regional"
]
```
Here `localbc` only applies to the `forestry` sector, while the other objects are used in every sector. Some attributes (e.g. `frequency` in `definitions/variable`) can have objects as value, which the are evaluated for the particular sector.

In order to add a new sector, the following steps need to be taken:

* Add the sector with `specifier` and `title` to `definitions/sector.json`.
* Add `pattern/ISIMIP3a/OutputData/<sector>.json` and `pattern/ISIMIP3b/OutputData/<sector>.json` with the file patterns for the new sector.
* Add new `variable` group(s) to `definitions/group.json`.
* Add sector variables to `definitions/variable.json` and/or update existing variables with the new sector.
* Add the sector with `specifier` and `title` to `definitions/sector.yaml`.
* Add `pattern/ISIMIP3a/OutputData/<sector>.yaml` and `pattern/ISIMIP3b/OutputData/<sector>.yaml` with the file patterns for the new sector.
* Add new `variable` group(s) to `definitions/group.yaml`.
* Add sector variables to `definitions/variable.yaml` and/or update existing variables with the new sector.


Printing
Expand Down
3 changes: 2 additions & 1 deletion app/.gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
nvm
node_modules
dist
output
27 changes: 23 additions & 4 deletions app/Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,24 @@
prod:
source ${NVM_DIR}/nvm.sh; nvm use; npm run build:prod
export NVM_DIR=$(dir $(realpath $(lastword $(MAKEFILE_LIST))))/nvm

watch:
source ${NVM_DIR}/nvm.sh; nvm use; npm run watch
.PHONY: prod build watch

prod: nvm node_modules
. "${NVM_DIR}/nvm.sh"; nvm use; npm run build:prod

build: nvm node_modules
. "${NVM_DIR}/nvm.sh"; nvm use; npm run build

watch: nvm node_modules
. "${NVM_DIR}/nvm.sh"; nvm use; npm run watch

nvm:
mkdir -p ${NVM_DIR}
NVM_DIR=${NVM_DIR} PROFILE=/dev/null \
bash -c 'curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash > /dev/null'
. "${NVM_DIR}/nvm.sh"; nvm install

node_modules:
. "${NVM_DIR}/nvm.sh"; nvm use; npm ci

clean:
rm -rf nvm node_modules
7 changes: 4 additions & 3 deletions app/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
"name": "isimip-protocol-3",
"version": "v1",
"scripts": {
"build:prod": "webpack --config webpack/prod.config.js --mode production",
"build": "webpack --config webpack/dev.config.js --mode development",
"watch": "webpack --config webpack/dev.config.js --mode development --watch"
"build:prod": "webpack --config webpack.config.js --mode production",
"build": "webpack --config webpack.config.js --mode development",
"watch": "webpack --config webpack.config.js --mode development --watch",
"lint": "eslint --ext .js src/"
},
"author": "Potsdam Institute for Climate Impact Research",
"license": "MIT",
Expand Down
62 changes: 62 additions & 0 deletions app/webpack.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
const webpack = require('webpack')
const { merge } = require('webpack-merge')
const path = require('path')
const MiniCssExtractPlugin = require('mini-css-extract-plugin')
const CopyWebpackPlugin = require('copy-webpack-plugin')
const TerserPlugin = require('terser-webpack-plugin')

// base config for all environments
const baseConfig = {
entry: './src/app.js',
module: {
rules: [
{
test: /\.(js|jsx)$/,
exclude: /(node_modules|bower_components)/,
loader: 'babel-loader',
options: { presets: ['@babel/env'] }
}
]
},
resolve: { extensions: ['*', '.js', '.jsx'] },
output: {
path: path.resolve(__dirname, './output/'),
filename: 'app.js'
}
}

// special config for development
const developmentConfig = {
devtool: 'eval',
plugins: [
new webpack.DefinePlugin({
'process.env.NODE_ENV': JSON.stringify('development')
})
]
}

// special config for production
const productionConfig = {
bail: true,
plugins: [
new webpack.DefinePlugin({
'process.env.NODE_ENV': JSON.stringify('production')
})
],
optimization: {
minimize: true,
minimizer: [new TerserPlugin()]
}
}

// combine config depending on the provided --mode command line option
module.exports = (env, argv) => {
switch (argv.mode) {
case 'development':
return merge(baseConfig, developmentConfig)
case 'production':
return merge(baseConfig, productionConfig)
default:
throw new Error('Invalid mode')
}
}
24 changes: 0 additions & 24 deletions app/webpack/common.config.js

This file was deleted.

Loading

0 comments on commit 781961b

Please sign in to comment.