Release 2.4.0 #1157
Merged
Release 2.4.0 #1157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Feature/lot3 output sql endpoint (#858) * Readded sql endpoint * Split raw files for sql processing. Amend sql view to support both. * Added tests for output api. * Update reader and block endpoints when reader does not support SQL. * added signed url generation to storage for data converter * made it possible to load whole sections from the ini conig as a dict * added task to run data conversion * added missing migrations * made file_storage_link the converted file url if the file has been converted * made the converter raise errors rather than logging can completing successfully * added ability to download converted files * added passing validation configs into the conversion process * added conversion time to related files * added custom logger and logfile gathering for data converter * added group access to mapping files and endpoints for managing access * added tests for file conversion permissions * moved storage objects to lot 3 package * added injecting the model storage config into the OasisLMf bash args * added root dir to model storage * added passing model storage config into tasks * fixed issue when no model storage was set * fixed issues connecting to an azure model storge * fixed tests not being able to override filestora settings * Readded sql endpoint (#848) * Readded sql endpoint * update keycloak * Set version 2.2.0 * Updated Package Requirements: django==3.2.20 * retest * Update changelog * Feautre/1323 reorganize branches plat2 (#849) * Update CI plat2 * Update readme title * Fix piwind branch select * Updated Package Requirements: pyYaml==5.3.1 * fix * test CI workflow without CVE error --------- Co-authored-by: awsbuild <[email protected]> * Fix/migrations plat1 to plat2 (#862) * nuke all current migration files * Add in platform 1 migrations (version 1.28.0) * Apply platform2 migrations ontop of plat1 * Move all of ssl connection string to variable * Update deploy script * test data retention -- testing only * Add helper script to support 2.2.0 and below * Add support to migration between plat2 versions * f * fix * tidy * Revert "test data retention -- testing only" This reverts commit a64e0a8. * Updated Package Requirements: pyyaml==6.0.1 * trigger retest * stricter checking for missing migrations Revert "Revert "test data retention -- testing only"" This reverts commit ecf766d. fix Revert "Revert "Revert "test data retention -- testing only""" This reverts commit 8e4474be5a11869571d10b31ab0ca7b6462e6988. --------- Co-authored-by: awsbuild <[email protected]> * Fix to tag piwind repo on publish (#865) * Fix to tag piwind repo on publish Extract prev versions for latest released worker Disable guards to test release script Set version 2.2.1rc2 Add option to set min CVE errors Fix fix Fix ktools tag and latest publish Fix boolean Update changelog Update changelog test push git tag Update changelog test piwind tag from remote workflow test Revert "test" This reverts commit 82f8ae7. f finish pub script * Revert files edited in release testing * switch piwind tag to main branch * Fix cryptography CVE-2023-38325 - platform 2 (#873) * Updated Package Requirements: cryptography==41.0.2 autobahn pyopenssl * Updated Package Requirements: certifi==2023.7.22 * retest --------- Co-authored-by: awsbuild <[email protected]> --------- Co-authored-by: Sam Gamble <[email protected]> Co-authored-by: awsbuild <[email protected]> Co-authored-by: sambles <[email protected]> Co-authored-by: Dan Bate <[email protected]> * added readme about model storage * upped llvmlite version to fix gul tests * linting * updated lot3 install in ci * added passing df reader configuration to lmf commands * fixed failing tests * added configuration for dask cluster to the docker-compose * made the dask worker and scheduler in docker compose use the host network * fixed failing test * updated lot 3 storage classes * fixed issues reversing sql urls * fixed remaining tests * linting * Hook in oasis-data-manager * Fix clashing Serializer ref_names * Update requriments * Hide lot3 endpoints Skip lot3 tests * update debian worker base image * PEP * Fix image build errors * fix missing system package for dask-sql * WIP - workflow hang when writing files * Add portainer to compose * Strip out dask and redo image packages * f * Fix merge issues * Apply json post fix to v1 files * wip - add oasis-data-manager branch build * fix local runs - post merge * pep * Fix model storage * Update S3 localstack example for testing * f * fix * keep run data for debugging * Disable global env fetch and fallback on local storage * test no global ENV for model storage * pep * test with real bucket * Fix bucket root dir * dump params to oasislmf.json in run dir for debugging * fix pre-analysis-hook * DEBUG * Add missing ENV to fail analysis if RI or IL is requested in settings without the required oasis files * Get V2 worker running with remote piwind data * Updated Package Requirements: oasis-data-manager==0.1.3 * pep * fix tests * WIP * PEP * Fix V1 worker tests * Flake8 * Skip all LOT3 SQL endpoint testing (disabled at the moment) * Fix loading diff between V1 and V2 input tar * Fix settings store in gen losses * Tone down Vun scanning - just run on releases * Update deploy scripts for plat testing --------- Co-authored-by: James Outterside <[email protected]> Co-authored-by: Dan Bate <[email protected]> Co-authored-by: awsbuild <[email protected]>
* Move debug compose file * disable debug options and remove src mounts in deafult compose * Updated Package Requirements: aiohttp==3.9.4 * test --------- Co-authored-by: awsbuild <[email protected]>
* Add missing calls to method param func, this checks and aligns any deprecated param names * Fix unit test
* Update keycloak to mount in cert from KeyVault * Test chart install - minikube without azure values
* fix "oed_schema_info" getting removed from task params * Pass oed_custom_spec into OedExposure load * pep * Updated Package Requirements: pymysql==1.1.1 --------- Co-authored-by: awsbuild <[email protected]>
…issue (#1054) * Add last stable 2.3.x image to compatibility checks * Fix failed jobs * f * Add check for raw_output_locations in worker result
* Delete UI compose -- exisits in default * Clear out older compose examples * Delete postgres example -- this is now the default * Remove redis example
Co-authored-by: awsbuild <[email protected]>
* When new run is triggered send entire WS status, not just single analysis * Fix oed_spec issue * pep
* wip * Only read settings JSON on create / partial update * POST settings on model creation working * PATCH model settings working * Expand analyses with Settings on POST/PATCH * PEP8 * Fix model list serializer * Fix missing field * Fix missing run_mode
Groups property is being used on:
- portfolios
- models
- data_files
- analyses
But v1 of the API does not return property groups. Only v2 does.
After checking the swagger, updating to v2 seems better than removing groups from the sanitization of the returned body.
Using API v2 for:
- /models/${PIWIND_VERSION}/chunking_configuration
- /models/${PIWIND_VERSION}/scaling_configuration
version 1 of the API lacks of these methods
Co-authored-by: Leon Alvarez <[email protected]>
* Make sure TB is not none and log any exections within worker * re-raise
…1078) * Fix analyses return types * Add missing data_files to read-only model serializer * Fix model return types
* Fix analysis to return AnalysesList * Fix model serializer for queues
* Updated Package Requirements: fiona==1.10b1 * Updated Package Requirements: fiona==1.10b2 --------- Co-authored-by: awsbuild <[email protected]>
Fix hook params
…ad request. (#1086) * Dont set analyses state as RUN_ERROR on run submit validation issues * Add back raise_validate_errors method ~ needed in V1 * pep
Co-authored-by: awsbuild <[email protected]>
* Fix analysis cancellation if analyses is queued * Allow exection cancel to be re-issued if sub-tasks are marked as running or queued * Revert "Allow exection cancel to be re-issued if sub-tasks are marked as running or queued" This reverts commit bbb2f32. * WIP * Revert "Fix analysis cancellation if analyses is queued" This reverts commit 8ca9da0. * Send Update sub-task ID when creaing chain + kill any task marked as REVOKE * pep * replace prints with logger * Set celery workers to deubg * Fix task-id extraction * handle exceptions - don't fail job if task-id extract fails
Co-authored-by: awsbuild <[email protected]>
* Fix input gen logging issue Track and copy sub-tasks logs to target dir before tar creation Store V1 logs in same locations logs -> log * Fix V2 model with single chunk not following ktools_num_processes * pep * Fix log tar arcname between v1 an v2
* Feature for model storage_link endpoint Draft storage link endpoint for models Add v1 model storage links pep * Name clash fix
* Fix from siobhan - update mapper class in realm config * Add catch for missing groups in claim * Allow admin users to work without group claim * Make sure to update role before groups
test example from github trivy-action
* tmp * retun checks for V2 runs * Only block input generation with a missing loc file on run_mode = V2 * Revert "retun checks for V2 runs" This reverts commit 925474d. * fix * Update error in v2 test * Fix testing and validation * update port val messages
* Remove the conf variable MODEL_SETTINGS_FILE from run params * Fix removal of MODEL_SETTINGS_FILE
* apply index opts * Add status index_DB migration file * Add script to test DB migrations Add basic CI check for DB update miration fetch all commits to parse out last release Fix DB create script adjust permissions limit py env to server requ * Updated Package Requirements: waitress==3.0.1 * Fix CVE-2023-43804 --------- Co-authored-by: awsbuild <[email protected]>
* Fix pip install on based worker images * Updated Package Requirements: waitress==3.0.1 * retest * f * Add to server --------- Co-authored-by: awsbuild <[email protected]>
* In Progress - List contents of input_file and output_file * Filter files with extension in list_tar_file * Implement extracting files using new endpoint * Native tarinfo file check for list file. * Content type and error checking for extract file * Content type and error checking for extract file * Refactor + remove white spaces * Add extract output files * Rename endpoints * Remove accidental database commit * Update gitignore with potential db loc * Add swagger response to tar_list method * Fix pep8 errors * Remove comment
* update worker get version * WIP - Added debugging points to check endpoints * Add ods + oed version to model versions. * Include migration with additional version fields * Add server components version info * Bug/Fix ktools version * Extract components version logic to model * Remove debug tools from Dockerfile * Update tests for server info --------- Co-authored-by: Sam Gamble <[email protected]>
* feature working * clean out breakpoints * pep
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # via s3fs | ||
| aiohappyeyeballs==2.4.3 | ||
| # via aiohttp | ||
| aiohttp==3.10.10 |
Check failure
Code scanning / Trivy
aiohttp: aiohttp memory leak when middleware is enabled when requesting a resource with a non-allowed method High
| # via s3fs | ||
| aiohappyeyeballs==2.4.3 | ||
| # via aiohttp | ||
| aiohttp==3.10.10 |
Check warning
Code scanning / Trivy
aiohttp: aiohttp vulnerable to request smuggling due to incorrect parsing of chunk extensions Medium
| # via virtualenv | ||
| django==3.2.23 | ||
| django==3.2.25 |
Check notice
Code scanning / Trivy
python-django: Potential user email enumeration via response status on password reset Low
* update test_analysis_model * fix test_portfolio * pep
* update ffspec * revert "update ffspec" This reverts commit ad39016. * Add workaround and fix pre-analysis file loading * foo
* validation * Add workaround for COB exception allways raised * f * tidy * V1 update tests * fix v2 tests * ffs
* test update * pep * f * compat with location loading
sambles
force-pushed
the
release/2.4.0
branch
from
75a6f8d to
b2a8ef6
Compare
| aiohttp==3.9.0 | ||
| aiohappyeyeballs==2.4.4 | ||
| # via aiohttp | ||
| aiohttp==3.10.10 |
Check failure
Code scanning / Trivy
aiohttp: aiohttp memory leak when middleware is enabled when requesting a resource with a non-allowed method High
| aiohttp==3.9.0 | ||
| aiohappyeyeballs==2.4.4 | ||
| # via aiohttp | ||
| aiohttp==3.10.10 |
Check warning
Code scanning / Trivy
aiohttp: aiohttp vulnerable to request smuggling due to incorrect parsing of chunk extensions Medium
| # via ipython | ||
| jinja2==3.1.2 | ||
| jinja2==3.1.4 |
Check failure
Code scanning / Trivy
jinja2: Jinja has a sandbox breakout through malicious filenames High
| # via ipython | ||
| jinja2==3.1.2 | ||
| jinja2==3.1.4 |
Check failure
Code scanning / Trivy
jinja2: Jinja has a sandbox breakout through indirect reference to format method High
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release 2.4.0