Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to pydantic v2 #297

Merged
merged 24 commits into from
Dec 21, 2023

Conversation

phackstock
Copy link
Contributor

@phackstock phackstock commented Dec 19, 2023

Closes #291, closes #214.

Changes

Pydantic v2 introduced a lot of API changes and deprecated a lot of classes and functions so quite a few changes were necessary.

CustomErrors

Previously we made use of a lot of custom errors. Pydantic v2 no longer allows this. There is a class PydanticCustomError but it cannot be subclassed.
After consideration I came to the conclusion that it's still useful to make use of these custom errors since this gives us finer control over the formatting of the error messages.
As a fix I ripped out the error folder and replaced it with a single error.py module. In this module there's now a dictionary called pydantic_custom_error_config that defines the values for our various custom errors. For each error this is a short name and the error message.
Custom errors can now be raised like so:

from pydantic_core import PydanticCustomError
from nomenclature.error import custom_pydantic_errors
...
raise PydanticCustomError(*custom_pydantic_errors.MyCustomError, {"value1": value1, "value2": value2})

where value1 and value2 are to be interpolated as part of the error message.

Error collection

Previously I had (ab)-used pydantic achieve the behavior of collecting a number of errors and then raising them all at once. This is no longer possible since pydantic 2 removed the class I used to do that. It was also never really the cleanest way to implement this mechanic since pydantic errors should be reserved for validation at object creation.
I implemented a class called ErrorCollector which collects errors in a list to be raised at a later point in time.
This is how you use it:

    errors = ErrorCollector()
    for file in required_data_dir.iterdir():
        try:
            RequiredDataValidator.from_file(file).validate_with_definition(dsd)
        except ValueError as error:
            errors.append(error)
    if errors:
        raise ValueError(f"Found error(s) in required data files:\n{errors}")

Output wise it is currently not ideal but I'd say this PR is big enough and I'll address this in a follow up PR.

Each item

Pydantic v2 changed the rather straightforward each_item option for validators. Previously this option would tell pydantic to apply a validator to each item of a list of dictionary.
Now this has to be done via the Annotated keyword in the variable type definition.
I don't really like it in terms of readability but there seems to be no other option.

Various small changes

Optional to |

In a number of type hints, updated Optional to the recommended | None.

Used the new model_dump

In a couple of places the new Basemodel.model_dump() came in handy for shortening code.

Removed unused code

  • DataStructureConfig.check_repository_consistency was always covered by another validator
  • In RegionProcessor._apply_region_processing it is not necessary to check that all the regions are valid. This is already done at initialization of the processor.

@phackstock phackstock added the enhancement New feature or request label Dec 19, 2023
@phackstock phackstock self-assigned this Dec 19, 2023
@phackstock phackstock marked this pull request as ready for review December 20, 2023 11:28
Copy link
Member

@danielhuppmann danielhuppmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the thorough refactoring, looks all good to me!

@phackstock phackstock merged commit 72fec3f into IAMconsortium:main Dec 21, 2023
8 checks passed
@phackstock phackstock deleted the feature/update-to-pydantic2 branch December 21, 2023 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to use latest pydantic Replace usage of pydantic ValidationErrors outside of actual validators
2 participants