Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job nested process workflow #748

Merged
merged 24 commits into from
Nov 16, 2024
Merged

Job nested process workflow #748

merged 24 commits into from
Nov 16, 2024

Conversation

@fmigneault fmigneault self-assigned this Oct 30, 2024
@github-actions github-actions bot added ci/doc Issue related to documentation of the package feature/CWL Issue related to CWL support feature/db Related to database or datatype manipulation. feature/oas Issues related to OpenAPI specifications. process/esgf-cwt Issue related to ESGF-CWT components. process/wps1 Issue related to WPS 1.x processes support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support labels Oct 30, 2024
@fmigneault fmigneault added process/OAP-Part3: Workflows OGC API - Processes - Part 3: Workflows/Chaining and removed process/wps1 Issue related to WPS 1.x processes support ci/doc Issue related to documentation of the package feature/CWL Issue related to CWL support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support process/esgf-cwt Issue related to ESGF-CWT components. feature/db Related to database or datatype manipulation. feature/oas Issues related to OpenAPI specifications. labels Oct 30, 2024
@github-actions github-actions bot added ci/doc Issue related to documentation of the package feature/CWL Issue related to CWL support feature/oas Issues related to OpenAPI specifications. process/esgf-cwt Issue related to ESGF-CWT components. process/wps1 Issue related to WPS 1.x processes support ci/operations Related to CI operations (actions, execution, install, builds, etc.) feature/db Related to database or datatype manipulation. process/builtin Issue related to builtin application processes process/wps3 Issue related to WPS 3.x (REST-JSON) processes support ci/tests Tests of the package and features labels Oct 30, 2024
@fmigneault fmigneault marked this pull request as ready for review November 11, 2024 22:54
@fmigneault fmigneault linked an issue Nov 11, 2024 that may be closed by this pull request
@fmigneault fmigneault added project/OGC Related to OGC testbeds or relavant projects. project/CRIM-DEVOPS Project linked to the CRIM project DevOps/Weaver (https://crim-ca.atlassian.net/browse/GD-47). project/OGC-GDC Developments related to OGC GeoDataCube labels Nov 11, 2024
Copy link

codecov bot commented Nov 13, 2024

Codecov Report

Attention: Patch coverage is 96.03960% with 8 lines in your changes missing coverage. Please review.

Project coverage is 87.40%. Comparing base (0f61ab2) to head (edd3abc).
Report is 33 commits behind head on master.

Files with missing lines Patch % Lines
weaver/processes/wps_process_base.py 84.61% 2 Missing ⚠️
weaver/wps_restapi/colander_extras.py 96.92% 0 Missing and 2 partials ⚠️
weaver/database/__init__.py 87.50% 0 Missing and 1 partial ⚠️
weaver/processes/builtin/collection_processor.py 0.00% 1 Missing ⚠️
weaver/processes/execution.py 97.72% 0 Missing and 1 partial ⚠️
weaver/wps_restapi/swagger_definitions.py 97.87% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
+ Coverage   87.07%   87.40%   +0.33%     
==========================================
  Files          80       80              
  Lines       20395    21312     +917     
  Branches     2774     2954     +180     
==========================================
+ Hits        17758    18627     +869     
- Misses       1900     1921      +21     
- Partials      737      764      +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Nazim-crim
Nazim-crim previously approved these changes Nov 13, 2024
Copy link
Contributor

@Nazim-crim Nazim-crim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

weaver/processes/ogc_api_process.py Show resolved Hide resolved
weaver/processes/wps_process_base.py Show resolved Hide resolved
Comment on lines +3965 to +3966
# type: (Dict[str, Any]) -> List[colander.SchemaNode]
self.children = [node.clone() for node in ExecuteProcessParameters().children]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't confusing calling it a bind if it's not a shallow copy ?
Shouldn't it be self.children = list(ExecuteProcessParameters().children)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limitation from the parent colander library that called the method bind. Nested nodes have to be cloned, otherwise you could have by-ref conflicting updates if similar definitions are used simultaneously.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a test in ef93c74 (#748) so you can get a better understanding of the feature and its behavior.

Base automatically changed from job-management-execute to master November 13, 2024 21:51
@fmigneault fmigneault dismissed Nazim-crim’s stale review November 13, 2024 21:51

The base branch was changed.

@fmigneault fmigneault merged commit a9bda01 into master Nov 16, 2024
48 checks passed
@fmigneault fmigneault deleted the job-nested-process-workflow branch November 16, 2024 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci/doc Issue related to documentation of the package ci/operations Related to CI operations (actions, execution, install, builds, etc.) ci/tests Tests of the package and features feature/CWL Issue related to CWL support feature/db Related to database or datatype manipulation. feature/oas Issues related to OpenAPI specifications. process/builtin Issue related to builtin application processes process/esgf-cwt Issue related to ESGF-CWT components. process/OAP-Part3: Workflows OGC API - Processes - Part 3: Workflows/Chaining process/workflow Related to a Workflow process. process/wps1 Issue related to WPS 1.x processes support process/wps3 Issue related to WPS 3.x (REST-JSON) processes support project/CRIM-DEVOPS Project linked to the CRIM project DevOps/Weaver (https://crim-ca.atlassian.net/browse/GD-47). project/OGC Related to OGC testbeds or relavant projects. project/OGC-GDC Developments related to OGC GeoDataCube
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support the Nested Processes as process: <uri> construct
2 participants