-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update to pydantic v2 #297
Merged
phackstock
merged 24 commits into
IAMconsortium:main
from
phackstock:feature/update-to-pydantic2
Dec 21, 2023
Merged
Update to pydantic v2 #297
phackstock
merged 24 commits into
IAMconsortium:main
from
phackstock:feature/update-to-pydantic2
Dec 21, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danielhuppmann
approved these changes
Dec 21, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the thorough refactoring, looks all good to me!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #291, closes #214.
Changes
Pydantic v2 introduced a lot of API changes and deprecated a lot of classes and functions so quite a few changes were necessary.
CustomErrors
Previously we made use of a lot of custom errors. Pydantic v2 no longer allows this. There is a class
PydanticCustomError
but it cannot be subclassed.After consideration I came to the conclusion that it's still useful to make use of these custom errors since this gives us finer control over the formatting of the error messages.
As a fix I ripped out the error folder and replaced it with a single
error.py
module. In this module there's now a dictionary calledpydantic_custom_error_config
that defines the values for our various custom errors. For each error this is a short name and the error message.Custom errors can now be raised like so:
where
value1
andvalue2
are to be interpolated as part of the error message.Error collection
Previously I had (ab)-used pydantic achieve the behavior of collecting a number of errors and then raising them all at once. This is no longer possible since pydantic 2 removed the class I used to do that. It was also never really the cleanest way to implement this mechanic since pydantic errors should be reserved for validation at object creation.
I implemented a class called
ErrorCollector
which collects errors in a list to be raised at a later point in time.This is how you use it:
Output wise it is currently not ideal but I'd say this PR is big enough and I'll address this in a follow up PR.
Each item
Pydantic v2 changed the rather straightforward
each_item
option for validators. Previously this option would tell pydantic to apply a validator to each item of a list of dictionary.Now this has to be done via the
Annotated
keyword in the variable type definition.I don't really like it in terms of readability but there seems to be no other option.
Various small changes
Optional to |
In a number of type hints, updated
Optional
to the recommended| None
.Used the new model_dump
In a couple of places the new
Basemodel.model_dump()
came in handy for shortening code.Removed unused code
DataStructureConfig.check_repository_consistency
was always covered by another validatorRegionProcessor._apply_region_processing
it is not necessary to check that all the regions are valid. This is already done at initialization of the processor.