✨ Is355/optimizer projects ports #3504

pcrespov · 2022-11-02T17:30:23Z

What do these changes do?

The PR extend the concept of ports of a service to a project: a project can now have input/output ports. The inputs can be read and updated and the latter can be read using the web API. The input ports in a project are defined by parameter nodes and the project outputs by probe nodes.

Highlights

✨ webserver API updates to version 0.10.0
- ✨defines new collections: GET/PATCH projects/{project_id}/inputs and GET projects/{project_id}/outputs: see project_ports_handlers.py
♻️ models-library
- updates fastapi_encoder
- adds utils for service_io (will be used in catalog in coming PR)
🔨api/specs/webserver/scripts/openapi_projects_ports.py helper scripts to produce OAS
🔨 scripts to create ERD of pydantic models

Related issue/s

Implementing S-D22.4 Meta-Modeling optimizer module Y4M12 osparc-issues#355

How to test

For manual testing (with swagger) :

Create a project with parameters and probes. Name them nicely

three inputs in yellow: X, on and Z
three outputs in red Random sleep interval_2, Random sleep interval and Array probe
all inputs are connected
sleeper A has no results and sleeper B does
Array probe is disconnected

Open swagger

get projects/{project_id}/inputs, patch projects/{project_id}/inputs
run projects/{project_id}
get projects/{project_id}/outputs

Checklist

validate input values against schema
make version-*
make openapi.json
cd packages/postgres-database, make setup-commit, sc-pg review -m "my changes"
Unit tests for the changes exist
Runs in the swarm
Documentation reflects the changes
New module? Add your github username to .github/CODEOWNERS

codecov · 2022-11-02T17:34:08Z

Codecov Report

Merging #3504 (bccd8cd) into master (e7eada1) will increase coverage by 0.1%.
The diff coverage is 79.7%.

@@           Coverage Diff            @@
##           master   #3504     +/-   ##
========================================
+ Coverage    83.4%   83.6%   +0.1%     
========================================
  Files         848     824     -24     
  Lines       35779   35376    -403     
  Branches      748     706     -42     
========================================
- Hits        29862   29588    -274     
+ Misses       5726    5603    -123     
+ Partials      191     185      -6

Flag	Coverage Δ
integrationtests	`67.2% <46.6%> (-0.2%)`	⬇️
unittests	`80.6% <79.7%> (+0.1%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ages/models-library/src/models_library/services.py	`92.9% <ø> (ø)`
...vice_webserver/projects/projects_nodes_handlers.py	`79.3% <ø> (ø)`
...models_library/utils/_original_fastapi_encoders.py	`41.7% <41.7%> (ø)`
...vice_webserver/projects/projects_ports_handlers.py	`92.2% <92.2%> (ø)`
...rary/src/servicelib/aiohttp/requests_validation.py	`97.4% <92.3%> (-2.6%)`	⬇️
...r/src/simcore_service_webserver/projects/_ports.py	`93.5% <93.5%> (ø)`
...ls-library/src/models_library/utils/services_io.py	`95.4% <95.4%> (ø)`
...brary/src/models_library/utils/fastapi_encoders.py	`100.0% <100.0%> (+56.5%)`	⬆️
...r/src/simcore_service_webserver/projects/plugin.py	`100.0% <100.0%> (ø)`
.../simcore_service_webserver/projects/projects_db.py	`96.5% <100.0%> (+<0.1%)`	⬆️
... and 34 more

sanderegg · 2022-11-08T08:07:25Z

scripts/erd/Dockerfile

+#
+
+ARG PYTHON_VERSION="3.9.12"
+FROM python:${PYTHON_VERSION}-slim-buster as base


sometimes I wonder if the non-slim version without installing 10 tools is not better than the slim version + 10 tools in term of size and build time.

yep. we could explore that. @mguidon showed me some tools we could use in the new HB registry. It might in addition help us decide more secure base images

sanderegg

Great work!

mguidon · 2022-11-08T08:29:42Z

👍

GitHK

So if I get this correctly the purpose of the PR, is to allow you to read and write input ports and read output ports.
Could you please add that in the description or body. It is not obvious.

Some comments and questions below

packages/models-library/src/models_library/utils/services_io.py

services/web/server/src/simcore_service_webserver/projects/_ports.py

GitHK · 2022-11-08T13:06:17Z

services/web/server/src/simcore_service_webserver/projects/_ports.py

+            and node.key.startswith("simcore/services/frontend/parameter/")
+            and node.outputs
+        ):
+            # invariants


This is not that helpful

I will add extra comments but i refer to a common programming term here: https://en.wikipedia.org/wiki/Invariant-based_programming

services/web/server/src/simcore_service_webserver/projects/_ports.py

services/web/server/src/simcore_service_webserver/projects/projects_ports_handlers.py

GitHK · 2022-11-08T13:29:20Z

services/web/server/tests/unit/with_dbs/02/test_projects_ports_handlers.py

+    mock_catalog_service_api_responses: None,
+    expected: type[web.HTTPException],
+):
+    """This tests implements a minimal workflow to support the MVP


MVP stands for?

https://en.wikipedia.org/wiki/Minimum_viable_product

GitHK · 2022-11-08T13:31:35Z

services/web/server/tests/unit/with_dbs/02/test_projects_ports_handlers.py

+    resp = await client.get(f"/v0/projects/{project_id}/outputs")
+    project_outputs, error = await assert_status(resp, expected_cls=expected)
+
+    if not error:


Why would there be an error here? if this is expected in some cases, please provide it as an expectation parameter to the test. I think this allows to better detect issues.

this is a technique we use a lot. The test parametrization adds a expected value with a status code that might be error or not. In the latter case, we perform some extra checks e.g. on the response

the expectation you refer is already checked in assert_status( resp, expected_cls=expected) . If expected is a non-error, then that assert function will make sure that error is None

GitHK

👍

extra fix in requests_validation

sonarqubecloud · 2022-11-09T09:55:47Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

pcrespov self-assigned this Nov 2, 2022

pcrespov force-pushed the is355/optimizer-projects-ports branch from 56604a1 to fce73bc Compare November 4, 2022 23:15

pcrespov added this to the Katherine Switzer milestone Nov 5, 2022

pcrespov added the a:webserver issue related to the webserver service label Nov 5, 2022

pcrespov mentioned this pull request Nov 6, 2022

S-D22.4 Meta-Modeling optimizer module Y4M12 ITISFoundation/osparc-issues#355

Closed

pcrespov force-pushed the is355/optimizer-projects-ports branch from 55f5003 to b90aacf Compare November 7, 2022 20:09

pcrespov changed the title ~~WIP: ✨ Is355/optimizer projects ports~~ ✨ Is355/optimizer projects ports Nov 7, 2022

pcrespov requested review from sanderegg, GitHK, odeimaiz and mguidon November 7, 2022 20:18

pcrespov force-pushed the is355/optimizer-projects-ports branch from 04c22dd to ac9ac96 Compare November 7, 2022 20:21

pcrespov marked this pull request as ready for review November 7, 2022 20:39

sanderegg reviewed Nov 8, 2022

View reviewed changes

sanderegg approved these changes Nov 8, 2022

View reviewed changes

pcrespov force-pushed the is355/optimizer-projects-ports branch from c5af8f6 to 1330a2a Compare November 8, 2022 08:31

GitHK reviewed Nov 8, 2022

View reviewed changes

pcrespov force-pushed the is355/optimizer-projects-ports branch from e7287f5 to a737c33 Compare November 8, 2022 15:23

pcrespov requested a review from GitHK November 8, 2022 15:25

GitHK approved these changes Nov 8, 2022

View reviewed changes

pcrespov force-pushed the is355/optimizer-projects-ports branch from 40d0175 to 69ca14b Compare November 9, 2022 00:44

pcrespov added 7 commits November 9, 2022 10:53

fixes OAS

bfef7bd

erd tools

118cddd

minor doc

fdc78d5

erdantic recipes

ccaf433

cleanup

e180cf9

drafts port's handlers

8bc35d0

extends parse_request_body

abf06d8

pcrespov added 26 commits November 9, 2022 10:53

Fixes aiohttp.request_validators

88b84bb

extra fix in requests_validation

doc

f601b48

Fixes error in uuid

3dd46d3

modifies replace to update_project_inputs

2aec9f9

adds port-links and handles errors

f8b21bf

drafted test for projects_ports

426eec9

adds port scema validation

f72e2c1

fixing linter and routes tests

193ebe9

convert old to new ports legacy

e7d03a1

adds get json-schema to project-ports

e14d8b3

cleanup test ports

2ce8f99

splits tests in two

a3db936

adds tests on schema

1d2c2b2

activates acceptance test

76a4a65

ignores pylint

be6486a

fixes test

4be5021

refactors fixture

076c3be

updates API

0787c9c

skip CC

436bd64

rm todos

969b914

fixes validator

08654f4

@GitHK review: rename io

3b831e2

@GitHK review: rename

ca3cdc9

@GitHK review: doc

4a7a758

forgot

046421a

@GitHK review: rename

bccd8cd

pcrespov force-pushed the is355/optimizer-projects-ports branch from 69ca14b to bccd8cd Compare November 9, 2022 09:53

pcrespov merged commit 31ac97f into ITISFoundation:master Nov 9, 2022

pcrespov deleted the is355/optimizer-projects-ports branch November 9, 2022 10:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Is355/optimizer projects ports #3504

✨ Is355/optimizer projects ports #3504

pcrespov commented Nov 2, 2022 •

edited

Loading

codecov bot commented Nov 2, 2022 •

edited

Loading

sanderegg Nov 8, 2022

pcrespov Nov 8, 2022

sanderegg left a comment

mguidon commented Nov 8, 2022

GitHK left a comment

GitHK Nov 8, 2022

pcrespov Nov 8, 2022

GitHK Nov 8, 2022

pcrespov Nov 8, 2022

GitHK Nov 8, 2022

pcrespov Nov 8, 2022

GitHK left a comment

sonarqubecloud bot commented Nov 9, 2022

✨ Is355/optimizer projects ports #3504

✨ Is355/optimizer projects ports #3504

Conversation

pcrespov commented Nov 2, 2022 • edited Loading

What do these changes do?

Highlights

Related issue/s

How to test

Checklist

codecov bot commented Nov 2, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderegg left a comment

Choose a reason for hiding this comment

mguidon commented Nov 8, 2022

GitHK left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitHK left a comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Nov 9, 2022

pcrespov commented Nov 2, 2022 •

edited

Loading

codecov bot commented Nov 2, 2022 •

edited

Loading