Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping of internal model variables and parameters to standard names #518

Merged
merged 49 commits into from
Feb 3, 2025

Conversation

vers-w
Copy link
Collaborator

@vers-w vers-w commented Dec 10, 2024

Issue addressed

Fixes #481

  • model type sbm: change model parameters, states, and output variables (sub-selection) to standard names.
  • model typesbm_gwf: change model parameters, states, and output variables (sub-selection) to standard names.
  • model type sediment: change model parameters, states, and output variables (sub-selection) to standard names.
  • update/refactor check of state variables.
  • possibly refactor hydrological forcing in model type sediment, see also Mapping of internal model variables and parameters to standard names #518 (comment), for now we decided to go with improvement in commit 8b38fb2.
  • check list model variables allowed to exchange (standard_name_map).
  • check units sediment model, not always clear from code/metadata/docs as the unit is also part of the standard name (quantity).
  • reduce allocations (increased compared to base branch). the increase in allocations is introduced in commit fbc2383, most probably caused by a new version of NCDatasets (need to look further into this, not part of this PR).

@SouthEndMusic
Copy link
Contributor

@vers-w I see you got rid of the nested names with periods in your TOML headers, but I make use of this structure in #519. I know I just started working on this without discussing with you, so let me know what your thoughts are (:

@JoostBuitink
Copy link
Contributor

@vers-w I see you got rid of the nested names with periods in your TOML headers, but I make use of this structure in #519. I know I just started working on this without discussing with you, so let me know what your thoughts are (:

Just to jump in this discussion here: we indeed decided to reorganize the toml file, so perhaps better to wait with #519 to prevent double work ;) ).

Rename these parameters to standard names.
 Additionally, made a start with standard name mapping (external standard name => internal model name). It makes use of `Wflow.param` functionality as much as possible, allowing to write any model output variable with the former TOML keys.
to standard names using `model_boundary_condition` (e.g. CSDMS uses `model_grid_ edge`) as objects.
Use lenses from `Accessors` for mapping standard names and for input parameters (`ncread`). Change parameter names of demand and groundwater to standard names. Writing any model output variable with the former TOML keys is possible (using `param`).
As the `sediment` model type makes use of hydrological forcing timeseries that are stored differently (internally) than the same timeseries used by model types `sbm` and `sbm_gwf`, the standard name mapping is done separately for the `sediment` model type (different dict). The same hydrological forcing data is used by the `SoilLoss` and `OverlandFlowSediment` components. This data is now only stored in the `SoilLoss` component and shared (memory) with the `OverlandFlowSediment` component.
@vers-w
Copy link
Collaborator Author

vers-w commented Dec 18, 2024

Hydrological forcing data in model type sediment is stored as part of the SoilLoss, OverlandFlowSediment and RiverSediment components, independently. In commit 8b38fb2 this data is now shared (in memory) between the SoilLoss and OverlandFlowSediment components. The same hydrological forcing time series are also present in the sbm and sbm_gwf model types stored at different internal model locations, compared to the sediment model. Because of this two different mappings (between the standard name and internal model name) are required.

Possible improvements:

  1. Store hydrological forcing data at one location in the sediment model (two mappings are required).
  2. Share the hydrological forcing data for SoilLoss and OverlandFlowSediment components (commit 8b38fb2, two mappings are required).
  3. Store the hydrological forcing data at the same internal model locations as the sbm and sbm_gwf models. One mapping is required and is likely most future-proof (e.g. run sbm and sediment in a combined wflow simulation).

@vers-w vers-w marked this pull request as ready for review January 15, 2025 13:06
@vers-w
Copy link
Collaborator Author

vers-w commented Jan 15, 2025

@JoostBuitink and @hboisgon, this PR is ready for review, couple of comments from my side:

  • For the standard names I did follow as much as possible the CSDMS standard names, also the examples are quite useful.
  • It is not possible to output model parameters and some variables through the standard names. It is possible to output these through the previous implementation (with dots) if needed. I suggest we consider only the standard names as the (official) API and not the previous implementation, otherwise we will have a breaking change (or make it backward compatible) each time when we change the internal structure. Please also check if variables are missing in the standard_name.jl file.
  • As I am not that familiar with the sediment code it would be good if @hboisgon could focus on checking the standard names of this component (besides checking generally if the standard names of the other components make sense). Also, the units of the sediment variables are not always that clear to me (sometimes there are differences between code, metadata and docs).
  • For a variable with dimension :layer, the standard name starts with soil_layer to indicate that this variable can represent multiple soil layers (allowed as input and output).
  • The standard name that starts with land_surface_water can refer to overland flow (for example land_surface_water__volume_flow_rate) or total surface water on the land surface including rivers, reservoirs and lakes (land_surface_water_abstraction__volume_flux and land_surface_water__potential_evaporation_volume_flux).

Use `count` instead of `number` as quantity as it preferable according CSDMS. Move reservoir and lake location and area IDs to this list (from `input.parameters`), for the `sediment` model type reservoir and lake area  IDS were already part of `input`.
Copy link
Contributor

@hboisgon hboisgon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the sediment part and most of it looks good already! Thanks @vers-w :)

Sorry that the metadata or unit in the code is not clear enough sometimes, I think also for me it's not clear that variables like precipitation or interception in mm are volumes or fluxes so the same happens for the sediment. If you know places in the code where it can be made clearer let me know and I can assist.
Note that the docs were not updated yet for sediment so that will still be on the todo list.

src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Outdated Show resolved Hide resolved
test/sediment_config.toml Outdated Show resolved Hide resolved
Copy link
Contributor

@JoostBuitink JoostBuitink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, and great steps for wflow as well! I have added a couple of comments and suggestions for additional parameters. I agree with the names are they are currently there, I think they nicely describe what they represent.

src/sediment/sediment_transport/river_transport.jl Outdated Show resolved Hide resolved
src/standard_name.jl Outdated Show resolved Hide resolved
src/standard_name.jl Outdated Show resolved Hide resolved
src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Show resolved Hide resolved
src/standard_name.jl Outdated Show resolved Hide resolved
test/io.jl Outdated Show resolved Hide resolved
test/sediment_config.toml Outdated Show resolved Hide resolved
vers-w and others added 15 commits January 27, 2025 09:24
Co-authored-by: JoostBuitink <[email protected]>
For soil erosion and overland flow.

Co-authored-by: hboisgon <[email protected]>
And rename standard name for river `inflow` (`inflow` not part of quantity).
Use `instantaneous` in quantity part for instantaneous (state) variables. For average values (per model time step), that are most useful for wflow users, `time_average_of` has been removed from the quantity part.
Remove `water` from objects in standard name as it is about river or land surface sediment properties (and not sediment transported by water).
These are average variables for the model time step. Rename reservoir `volume` to `storage` (consistent with lake).
Also rename `storage` in quantity in standard names to `depth` (it is not listed as quantity by CSDMS).
Also used by CSDMS and more consistent with other standard names.
`volume` is a quantity (and used as such in CSDMS standard names),  use of `storage` is preferred.
Added `storage_av` to routing components with an internal time step. Moved computation of average flow routing variables from update function of instanteneous values to the loop over model time step. Added helper functions for the computation of average flow routing variables.
For computation of average flow routing variables.
Copy link
Contributor

@JoostBuitink JoostBuitink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work; looks good to me!

@vers-w vers-w merged commit 6db3b2b into master Feb 3, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Mapping of internal model variables and parameters to standard names
4 participants