Add Stateful API testing for Model Registry #132
Conversation
Signed-off-by: lugi0 <[email protected]>
|
The following are automatically added/executed:
Available user actions:
Supported labels{'/wip', '/lgtm', '/hold', '/verified'} |
|
On hold until kubeflow/model-registry#773 is merged |
| # @pytest.fixture(scope="class") | ||
| # def generated_schema_local(model_registry_instance_rest_endpoint): | ||
| # schema = schemathesis.openapi.from_path( | ||
| # path="utilities/manifests/model-registry-api.yaml", |
There was a problem hiding this comment.
please set path in py_config so when a user wants to overwrite they can us tc=:
There was a problem hiding this comment.
Not sure to which scenario this would apply, this needs to be manually enabled and pointing to an updated openapi spec, I don't think it makes sense to have it exposed as a config variable.
|
|
||
| def after_call(self, response: Response, case: Case) -> None: | ||
| LOGGER.info( | ||
| "%s %s -> %d", |
There was a problem hiding this comment.
please use f-string formatting
There was a problem hiding this comment.
Will remove as not needed
|
|
||
|
|
||
| @pytest.fixture | ||
| def state_machine(generated_schema: BaseOpenAPISchema, current_client_token: str) -> APIStateMachine: |
There was a problem hiding this comment.
this can be a regular function; there's no setup / teardown related to the test here
There was a problem hiding this comment.
Schemathesis' documentation recommends using a fixture when loading the schema lazily: https://schemathesis.readthedocs.io/en/stable/stateful.html#schemathesis.stateful.state_machine.APIStateMachine.validate_response
| @pytest.mark.fuzzer | ||
| @pytest.mark.sanity |
There was a problem hiding this comment.
use pytestmark to set all markers in the same place.
| # Once it is updated to support Stateful testing of the API we can enable this to run every time, | ||
| # but for now having it run manually to check the existing failures is more than enough. | ||
| @pytest.mark.skip(reason="Only run manually for now") | ||
| @pytest.mark.fuzzer |
There was a problem hiding this comment.
i do not think that sanity tests should be marked as fuzzer; sanity tests are a set of tests to check basic functionality. fuzzing (or in this case tools which are based on it) is a different level of testing
There was a problem hiding this comment.
proper functioning of the API is basic functionality IMHO. We can exclude the "fuzzer" mark if we don't want to see these failures, that's why I'm setting both.
Description
This adds a test for stateful API testing of Model Registry. It also bumps Schemathesis to V4, which is still an alpha release but appears to work fine for this workload.
I will keep this as a draft until the openAPI spec is updated in the Model Registry repo to support Stateful testing, and we can then decide if we are ok with an alpha release being pulled or if we want to wait / downgrade to V3 (requires some rework due to breaking changes between the two major versions).
I have also added a new mark type, "fuzzer", which I'd use to indicate tests which are bound to fail due to their random nature.
How Has This Been Tested?
Multiple runs made from local with an updated openAPI spec file which have already identified some failures to address in Model Registry.
Merge criteria: