recreate perfmetrics portrait plot in python #3551

lukruh · 2024-03-20T14:22:40Z

Description

Closes Recreate portrait plot from NCL perfmetrics in python #3550
Link to documentation:

Before you get started

☝ Create an issue to discuss what you are going to do Recreate portrait plot from NCL perfmetrics in python #3550

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

🛠 This pull request has a descriptive title
🛠 Code is written according to the code quality guidelines
🛠 Documentation is available
🛠 Tests run successfully
🛠 The list of authors is up to date
🛠 Any changed dependencies have been added or removed correctly
🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

🧪 Recipe runs successfully
🧪 Recipe is well documented
🧪 Figure(s) and data look as expected from literature
🛠 Provenance information has been added

New or updated data reformatting script

🛠 Documentation is available
🛠 The dataset has been added to the CMOR check recipe
🛠 The dataset has been added to the shared data pools of DKRZ and Jasmin by the @ESMValGroup/OBS-maintainers team
🧪 Numbers and units of the data look physically meaningful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

… into perfmetric_python

valeriupredoi · 2024-03-22T13:37:29Z

what a noble venture, Lukas! Godspeed 🏁 🍺

… into perfmetric_python

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

schlunma

Just a couple of more comments of things I found while running an example recipe. Apart from these small issues, the diagnostic worked really nicely and the plot looks awesome 🚀

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

schlunma · 2024-05-02T09:49:11Z

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

+        dataset[cfg["x_by"]].str.lower(), dataset[cfg["y_by"]].str.lower(),
+        dataset[cfg["group_by"]].str.lower(),
+        dataset[cfg["split_by"]].str.lower()


This can lead to errors with any of the facets is None. For example, the default value if no split is given for a dataset is None.

Yes, that actually happened to me as well. I worked around it by setting everything that is None to 'None', but that is probably not the best solution ;).

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

schlunma · 2024-05-02T10:00:24Z

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

+    log.warning("Metadata found for %s", selection)
+    das = xr.open_dataset(metas[0]["filename"])
+    varname = list(das.data_vars.keys())[0]
+    return das[varname].values.item()


Would be great to raise a nicer error here if the data is non-scalar, at the moment this gives ValueError: can only convert an array of size 1 to a Python scalar.

Something like "Expected scalar in input file {filename}"?

Sounds good!

lukruh · 2024-05-02T10:08:59Z

So far we used the distance_metrics prerpocessor from within the recipe, which works but has some limitations in practise and can lead to messy recipes. I added an optional function to apply it from within the plot diagnostic. I.e. use reference_metric: rmse in the script section of the recipe instead of the distance_metric preprocessor. The reference_for_metric facet for the reference dataset i.e. ERA-interim is still required. @diegokam if you are available, you could have a look and try to adjust the recipe accordingly.

schlunma · 2024-05-02T11:22:27Z

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

+    if None in data.coords[cfg["split_by"]].values:
+        cfg.update({"default_split": None})
+    else:
+        cfg.update({"default_split": data.coords[cfg["split_by"]].values[0]})


Won't this overwrite a user-defined default_split? Do we want this?

mo-gill · 2024-05-02T14:41:11Z

Hi, we are currently working on the ESMValTool release for v2.11.0. We're wondering if you'd be able to complete this PR by the end of next week (Friday 10th May).

Otherwise, please let us know, and we'll move it into the next milestone for you 🙂

lukruh · 2024-05-03T13:38:45Z

I don't see this PR be ready until next week. The next milestone is more realistic I'd say.

lukruh · 2024-05-03T14:04:41Z

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

+        except IndexError as exc:
+            raise IndexError("No reference found for metric.") from exc
+        ref_cube = iris.load_cube(reference["filename"])
+        for meta in y_metas:


We should support multiple references (with different split facets). The split (maybe the full set of split/group/x/y) needs also to be part of the filename to avoid overwriting files.

rbeucher · 2024-06-11T07:30:31Z

I just came across this. It looks great. Any progress?

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py

schlunma · 2024-06-11T11:53:19Z

I just came across this. It looks great. Any progress?

We are working on it 😄 It's planned to include this in v2.12

lukruh · 2024-11-06T23:16:15Z

Its also possible know to apply MultiModel Mean and Median over each group (i.e. project). I added a simple recipe recipe_portrait_CMIP.yml with a few models and variables as example for the diagnostic. It runs faster and is less complex than the CMIP5 one. As discussed with @bettina-gier, the long recipe to reproduce the output of the NCL diagnostic will be removed before merge. It's attached here in case someone needs it after the branch is deleted.
recipe_perfmetrics_CMIP5_py.yml.txt

axel-lauer · 2024-11-07T07:40:31Z

I am not in favor of removing the old recipe as long as we do not reproduce the full output (e.g. including Taylor diagrams, etc.). and therefore have a new and complete "perfmetrics". So long, this PR just includes the portrait diagram. For this reason and to avoid confusion between "perfmetrics" and "portrait diagram", the new recipe (and documentation) should be simply called e.g. "recipe_portrait_plot.yml" and not contain anything like "perfmetrics" in the filenames.

As a side note, I am not sure helper scripts such as "compare_netCDFs.py" need to go into the "diagnostics" folder. Maybe they do not need to go into the repository at all?

axel-lauer · 2024-11-07T07:56:48Z

I just talked to @bettina-gier and I think I misunderstood the removal of the recipe. So this is fine. My points about the naming remain.

lukruh · 2024-11-07T15:23:43Z

Sorry for the confusion. I didn't plan to remove the NCL based recipe just the one that reproduces its results. The helper scripts and different recipe versions will be removed before merge. Should we keep the script and the results as comments in this PR, in case some one is interested?

lukruh added 4 commits March 14, 2024 16:29

example variables and models

bc2a6b2

added some datasets for fixed calendar

b74414f

first approach group gaps WIP

f78d2e5

imagegrid for grouping, labels, colorbar, docstring

93c5544

lukruh added the diagnostic label Mar 20, 2024

lukruh assigned diegokam and lukruh Mar 20, 2024

lukruh mentioned this pull request Mar 20, 2024

Recreate portrait plot from NCL perfmetrics in python #3550

Open

20 tasks

lukruh and others added 5 commits March 20, 2024 16:11

WIP triangle overlays

db30131

added CMIP6 experiments

215ee95

WIP split by reference

b4f8238

added icon example

5c30d4a

Merge branch 'perfmetric_python' of github.com:ESMValGroup/ESMValTool…

07e676a

… into perfmetric_python

lukruh and others added 12 commits March 27, 2024 13:08

split square feature, xarray instead of nested dicts

42fdd81

3 recipe examples

4ab76cb

fix config overwrite

32ad034

legend for multiple references

d66e5e2

Merge branch 'main' into perfmetric_python

e1bb9c4

pre-commit

30541be

Merge branch 'perfmetric_python' of github.com:ESMValGroup/ESMValTool…

5eb01f2

… into perfmetric_python

add normalization for metrics

d6a541f

codacy fixes

7fb8a10

WIP Documentation

eba93db

adding myself to list of authors

ffee698

number of local vars, alphabetical sorting by x labels

54ad3d8

lukruh added this to the v2.11.0 milestone Apr 16, 2024

lukruh added 2 commits April 16, 2024 12:18

add dpi option and plot legend only for multiple splits

7cbcf86

WIP recreate example perfmetric recipe

f7b6c87

schlunma reviewed Apr 18, 2024

View reviewed changes

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py Outdated Show resolved Hide resolved

add optional calculation of distance metrics

39df74e

schlunma reviewed May 2, 2024

View reviewed changes

fix bug for non default "split" facet

a896ce6

schlunma reviewed May 2, 2024

View reviewed changes

lukruh commented May 3, 2024

View reviewed changes

mo-gill modified the milestones: v2.11.0, v2.12.0 May 24, 2024

schlunma reviewed Jun 11, 2024

View reviewed changes

esmvaltool/diag_scripts/perfmetrics/portrait_plot.py Show resolved Hide resolved

diegokam and others added 5 commits November 4, 2024 16:22

recipe CMIP5 python, new python script creating netcdf file

62086aa

comparison script NetCDF files

8c9aa90

fixed pp heights, name changed, add provenance

9b20ac3

format recipe (pre commit)

db01441

simplify recipe p1

efc026a

lukruh changed the title ~~recreate perfmetric plots in python~~ recreate perfmetrics portrait plot in python Nov 6, 2024

lukruh added 2 commits November 6, 2024 23:56

MM stats as first columns (wip)

af917bc

simpler cmip5 vs cmip6 plot

d25290e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

recreate perfmetrics portrait plot in python #3551

recreate perfmetrics portrait plot in python #3551

lukruh commented Mar 20, 2024 •

edited

Loading

valeriupredoi commented Mar 22, 2024

schlunma left a comment

schlunma May 2, 2024

lukruh May 2, 2024

schlunma May 2, 2024

lukruh Nov 8, 2024 •

edited

Loading

schlunma Nov 8, 2024

lukruh commented May 2, 2024

schlunma May 2, 2024

mo-gill commented May 2, 2024 •

edited

Loading

lukruh commented May 3, 2024 •

edited

Loading

lukruh May 3, 2024

rbeucher commented Jun 11, 2024

schlunma commented Jun 11, 2024

lukruh commented Nov 6, 2024 •

edited

Loading

axel-lauer commented Nov 7, 2024

axel-lauer commented Nov 7, 2024

lukruh commented Nov 7, 2024

recreate perfmetrics portrait plot in python #3551

Are you sure you want to change the base?

recreate perfmetrics portrait plot in python #3551

Conversation

lukruh commented Mar 20, 2024 • edited Loading

Description

Before you get started

Checklist

valeriupredoi commented Mar 22, 2024

schlunma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukruh Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukruh commented May 2, 2024

Choose a reason for hiding this comment

mo-gill commented May 2, 2024 • edited Loading

lukruh commented May 3, 2024 • edited Loading

Choose a reason for hiding this comment

rbeucher commented Jun 11, 2024

schlunma commented Jun 11, 2024

lukruh commented Nov 6, 2024 • edited Loading

axel-lauer commented Nov 7, 2024

axel-lauer commented Nov 7, 2024

lukruh commented Nov 7, 2024

lukruh commented Mar 20, 2024 •

edited

Loading

lukruh Nov 8, 2024 •

edited

Loading

mo-gill commented May 2, 2024 •

edited

Loading

lukruh commented May 3, 2024 •

edited

Loading

lukruh commented Nov 6, 2024 •

edited

Loading