feat: add script to compute all impacts of databases

[vjousse]

🔧 Problem

We need a way to not rely on direct calls to Brightway for our future backend. This an experimental script to check if computing detailed impacts for all the processes of all our databases would be something doable (spoiler alert: it is).

🍰 Solution

Parse all the activities of type process in Brightway and compute the detailed impacts for each activity.

On my computer (12th Gen Intel i9-12900HK (20) @ 4.900GHz), using the 20 cores, it takes 1h46m to compute all the impacts of the following databases (~= 70K):

➜ cat bw_processes.json | jq "keys"                          
[
  "Agribalyse 3.1.1",
  "CTCPA",
  "Ecobalyse",
  "Ecoinvent 3.10",
  "Ecoinvent 3.9.1",
  "Ginko",
  "PastoEco",
  "WFLDB",
  "Woolmark",
  "biosphere3"
]

🚨 Points to watch/comments

By default the script takes all your cores minus one to make the computations and all the databases. You can specify the number of cores to use and the databases as options.

Run uv run bin/export_icv.py --help to check for all the possible options.

Full example with all options:

uv run bin/export_icv.py bw_processes.json --max 2 --db Ecobalyse --db "Ecoinvent 3.9.1" --cpu-count 10

🏝️ How to test

To compute everything:

export BRIGHTWAY2_DIR=/path/to/your/bw/db
uv run bin/export_icv.py bw_processes.json

To compute a max of 4 processes per DB (for testing purposes):

uv run bin/export_icv.py bw_processes.json --max 4

If you’re brave enough to compute everything, you can then compare that raw processes in public/data/…/processes_impacts.json have the same impacts than the one computed using this script.

For example, you can check for the process with a source id of MTE00149000081182202414 and a name of electricity, medium voltage//[RAS] market group for electricity, medium voltage.

[n1k0]

I successfully managed to export the sample processes as indicated in the test procedure. I also timed the operation if that's useful:

$ time uv run bin/export_icv.py bw_processes.json --max 4
(…)
uv run bin/export_icv.py bw_processes.json --max 4  226.15s user 95.80s system 813% cpu 39.597 total

It's really promising being able to export lots of impacts like that, nice work!

I have no idea how to review the Python code other than nitpicking on unimportant stuff so I'm gonna leave that part to others more knowledgeable than me.

[n1k0]

Also please note I had to export PYTHONPATH=. in order for the script to work so I think it should be mentioned somewhere in the docs and/or added to the default sample .env.

[paulboosz]

Promising work ! It work fine for me (I only did uv run bin/export_icv.py bw_processes.json --max 4)

I also had to export PYTHON_PATH=. +1 on Nico's remark.

I checked the coherence between this export and ecobalyse's. I only had 2 process in common. It's mostly the same impacts except for pear lduwhere we have 1.57% diff.

Comparing Ground beef, fresh, case ready, for direct consumption, at plant {FR} U [feedlot], constructed by Ecobalyse

Impact differences (file2 - file1)

Impact	Value 1	Value 2	Diff	% Diff
acd	0.785403	0.785403	2.919e-07	0.00%
cch	47.3618	47.3618	2.998e-05	0.00%
ecs	5461.59	5461.59	-0.00020446	-0.00%
etf	236.199	236.199	0.0003553	0.00%
etf_c	342.857	342.857	-0.0004809	-0.00%
fru	210.546	210.546	0.0003271	0.00%
fwe	0.00515778	0.00515778	-4.083e-09	-0.00%
htc	3.47681e-08	3.47681e-08	-9.8e-16	-0.00%
htc_c	1.23116e-08	1.23116e-08	-3.7518e-14	-0.00%
htn	9.58014e-07	9.58014e-07	2.53e-13	0.00%
htn_c	1.07319e-07	1.07319e-07	-1.1518e-13	-0.00%
ior	1.8396	1.8396	-8.67e-07	-0.00%
ldu	2978.07	2978.07	-0.003424	-0.00%
mru	0.000202173	0.000202173	-4.92e-11	-0.00%
ozd	1.60061e-06	1.60061e-06	1.167e-12	0.00%
pco	0.220615	0.220615	1.142e-07	0.00%
pef	5876.92	5876.92	-0.000605131	-0.00%
pma	5.03214e-06	5.03214e-06	5.48e-13	0.00%
swe	0.438692	0.438692	-4.753e-07	-0.00%
tre	3.52426	3.52426	-1.517e-06	-0.00%
wtu	12.0226	12.0226	-2.04e-05	-0.00%

Comparing Pear, consumption mix, organic 2023 {FR} U

Impact differences (file2 - file1)

Impact	Value 1	Value 2	Diff	% Diff
acd	0.00186208	0.00186208	-2.124e-09	-0.00%
cch	0.0954277	0.0954277	4.306e-08	0.00%
ecs	43.4986	43.47	-0.0285601	-0.07%
etf	3.53345	3.53331	-0.000144462	-0.00%
etf_c	6.40167	6.40153	-0.000142542	-0.00%
fru	2.33111	2.33111	3.851e-06	0.00%
fwe	2.71228e-05	2.71228e-05	-1.026e-11	-0.00%
htc	4.30102e-10	4.30102e-10	-1.9e-16	-0.00%
htc_c	4.85976e-10	4.85976e-10	4.172e-16	0.00%
htn	1.56728e-08	1.56711e-08	-1.702e-12	-0.01%
htn_c	2.08817e-08	2.08817e-08	-3.816e-14	-0.00%
ior	0.0424665	0.0424665	-3.601e-08	-0.00%
ldu	23.4449	23.0764	-0.368538	-1.57%
mru	3.35147e-06	3.35147e-06	-2.571e-12	-0.00%
ozd	2.09119e-08	2.09119e-08	-6.8e-15	-0.00%
pco	0.000987048	0.000987048	-4.653e-10	-0.00%
pef	40.8348	40.7988	-0.0359792	-0.09%
pma	1.43182e-08	1.43182e-08	4.449e-14	0.00%
swe	0.000545582	0.000545582	-5.18e-11	-0.00%
tre	0.00850604	0.00850604	-9.45e-10	-0.00%
wtu	2.15282	2.15282	-1.86e-06	-0.00%

[vjousse]

As I said on mattermost, we are currently mainly using Simapro to compute impacts in production so, having differences with Brightway is not really surprising. But if only have differences with ldu on the pear it’s good news!