This repository has been archived by the owner on Sep 16, 2022. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge stable to master: v0.2.0-rc1 (#319)
* Merge test to stable: v0.1.6-rc1 (#315) * CSCMETAX-428: [ADD] File api: Add api POST /files/restore to restore files from state removed=True to removed=False * CSCMETAX-61: [ADD] Improve logging when calculating project directory byte sizes and file counts (log totals once succesfully finished) * CSCMETAX-61: [FIX] File api: Improve dispatch() method to only execute a request without a transaction when using POST to create files. Used to catch all POST requests to /files * CSCMETAX-61: [ADD|REF] Improve logging in some file operations. Save date_modified timestamp to files when deleting * CSCMETAX-428: [ADD] When restoring files using /files/restore, also restore missing directories, when necessary. Improve logging * CSCMETAX-420: [ADD] Log basic request info middleware before, and after a request * CSCMETAX-424:[FIX] contributors content and kernel version. * CSCMETAX-61: [REF] Update requirements.txt * CSCMETAX-430: [REF] research_dataset.access_rights is now compulsory in att and ida dataset schemas. Updated test data accordingly. * CSCMETAX-248: [ADD] License information * CSCMETAX-61: [FIX] Fix broken resumption token logic. * CSCMETAX-434: [FIX] Don't strip whitespace from file or directory paths * CSCMETAX-438: [ADD] Store also information about Internal Server Errors (HTTP 500), and return error_identifier to users * CSCMETAX-442: [FIX|REF] Fix datacite convert errors. Convert metax datasets to datacite xml using 'datacite' library Previous implemenetation of datacite conversion relied on xml transformation, which used our custom 'metax dataset xml' as a starting point, and contained no xml schema validation at the end of it. The end result currently is quite far from the schema, and the xslt is very difficult to get right. As an alternative solution, use python 'datacite' library, which sports a datacite json-schema, and offers conversion from that json to xml representation. The json-way is easier to validate (there is a all-in-one json schema available), and easier to modify (just python code to create datacite json from metax dataset, instead of screwing around with xslt files). Just to be sure, the produced xml was manually tested to validate agains the datacite xml. The 'datacite' library conveniently also includes wrappers for DOI-related api's. * CSCMETAX-61: [FIX] Unhandled exception handling in handle_exception(). If an exception is not converted to framework-recognized exception earlier, rollback is not executed * CSCMETAX-388: [ADD] Better way to execute code at the very end of a request. During a request, enables any code, anywhere to append callable objects to a list of callables that will get executed in order of adding, at the end of the request. No need to deliver data or parameters from some code deepdown all the way to the out-most region in the code, where the request is almost finished, since the callables contain all the information the need, and are self-contained. If needed tho, it would be very easy to pass additional parameters to the callable, such as the request- or the response-object. * CSCMETAX-388: [REF] Publish dataset updates to rabbitmq using the new method of executing code at the end of the request * CSCMETAX-388: [ADD] Stub implementation of REMS connection * CSCMETAX-449: [ADD] Datacite conversion: If publisher is omitted from metax dataset, use first creator * CSCMETAX-437: [FIX] RequestLogging: If no useragent is set, print (no useragent) instead of crashing and burning * CSCMETAX-61: [FIX] When deleting datasets, only publish 'delete' to rabbitmq Before the recent refactoring of publishing to rabbitmq, when deleting a dataset, only 'delete' was published. Now, after the refactoring, it was discovered that when deleting a dataset, what happens is that actually the CR.save() method is executed as well, by the superclass. Previously, it did not cause a rabbitmq publish due to it being done in a different place. This is now fixed by having the superclass (model Common) only save the value of field 'removed', and calling its superclass (not CR.save(), and not common.save()) * CSCMETAX-450: [REF] If restriction grounds are not of open type (codes 1 and 2), then access type code must not be open_access OR If restriction grounds are of open type, then access type code must be open_access * CSCMETAX-61: [REF] Update requirements.txt * CSCMETAX-132: [REF] Do not publish messages or init exchanges if connection to RabbitMQ is not ok. Catch exceptions on publishing messages. On initializing RabbitMQ class, i.e. while connecting to RabbitMQ, increase robustness by 1) retrying connection 15 times waiting 2 seconds in between attempts before failing and 2) choosing random ip address from a list of rabbitmq cluster hosts. * CSCMETAX-61: [REF] Redis config key updates * CSCMETAX-61: [REF] access_type is again compulsory in harvester dataset schema. * CSCMETAX-455: [REF] Add additionalproperties to schemas. Fix some found problems in test data and app code. Note: test data fetch_and_update does not work in this commit. * CSCMETAX-455: [REF] Change some of the additionalProperties: false flags to true since some objects do have properties that are not visible in the json schema. Fix api code reference data autopopulation for the parts that were actually not properly done before. * CSCMETAX-455: [REF] Fix tests * CSCMETAX-455: [REF] Fix tests * Merge test to stable: v0.2.0-rc1 (#318) * CSCMETAX-428: [ADD] File api: Add api POST /files/restore to restore files from state removed=True to removed=False * CSCMETAX-61: [ADD] Improve logging when calculating project directory byte sizes and file counts (log totals once succesfully finished) * CSCMETAX-61: [FIX] File api: Improve dispatch() method to only execute a request without a transaction when using POST to create files. Used to catch all POST requests to /files * CSCMETAX-61: [ADD|REF] Improve logging in some file operations. Save date_modified timestamp to files when deleting * CSCMETAX-428: [ADD] When restoring files using /files/restore, also restore missing directories, when necessary. Improve logging * CSCMETAX-420: [ADD] Log basic request info middleware before, and after a request * CSCMETAX-424:[FIX] contributors content and kernel version. * CSCMETAX-61: [REF] Update requirements.txt * CSCMETAX-430: [REF] research_dataset.access_rights is now compulsory in att and ida dataset schemas. Updated test data accordingly. * CSCMETAX-248: [ADD] License information * CSCMETAX-61: [FIX] Fix broken resumption token logic. * CSCMETAX-434: [FIX] Don't strip whitespace from file or directory paths * CSCMETAX-438: [ADD] Store also information about Internal Server Errors (HTTP 500), and return error_identifier to users * CSCMETAX-442: [FIX|REF] Fix datacite convert errors. Convert metax datasets to datacite xml using 'datacite' library Previous implemenetation of datacite conversion relied on xml transformation, which used our custom 'metax dataset xml' as a starting point, and contained no xml schema validation at the end of it. The end result currently is quite far from the schema, and the xslt is very difficult to get right. As an alternative solution, use python 'datacite' library, which sports a datacite json-schema, and offers conversion from that json to xml representation. The json-way is easier to validate (there is a all-in-one json schema available), and easier to modify (just python code to create datacite json from metax dataset, instead of screwing around with xslt files). Just to be sure, the produced xml was manually tested to validate agains the datacite xml. The 'datacite' library conveniently also includes wrappers for DOI-related api's. * CSCMETAX-61: [FIX] Unhandled exception handling in handle_exception(). If an exception is not converted to framework-recognized exception earlier, rollback is not executed * CSCMETAX-388: [ADD] Better way to execute code at the very end of a request. During a request, enables any code, anywhere to append callable objects to a list of callables that will get executed in order of adding, at the end of the request. No need to deliver data or parameters from some code deepdown all the way to the out-most region in the code, where the request is almost finished, since the callables contain all the information the need, and are self-contained. If needed tho, it would be very easy to pass additional parameters to the callable, such as the request- or the response-object. * CSCMETAX-388: [REF] Publish dataset updates to rabbitmq using the new method of executing code at the end of the request * CSCMETAX-388: [ADD] Stub implementation of REMS connection * CSCMETAX-449: [ADD] Datacite conversion: If publisher is omitted from metax dataset, use first creator * CSCMETAX-437: [FIX] RequestLogging: If no useragent is set, print (no useragent) instead of crashing and burning * CSCMETAX-61: [FIX] When deleting datasets, only publish 'delete' to rabbitmq Before the recent refactoring of publishing to rabbitmq, when deleting a dataset, only 'delete' was published. Now, after the refactoring, it was discovered that when deleting a dataset, what happens is that actually the CR.save() method is executed as well, by the superclass. Previously, it did not cause a rabbitmq publish due to it being done in a different place. This is now fixed by having the superclass (model Common) only save the value of field 'removed', and calling its superclass (not CR.save(), and not common.save()) * CSCMETAX-450: [REF] If restriction grounds are not of open type (codes 1 and 2), then access type code must not be open_access OR If restriction grounds are of open type, then access type code must be open_access * CSCMETAX-61: [REF] Update requirements.txt * CSCMETAX-132: [REF] Do not publish messages or init exchanges if connection to RabbitMQ is not ok. Catch exceptions on publishing messages. On initializing RabbitMQ class, i.e. while connecting to RabbitMQ, increase robustness by 1) retrying connection 15 times waiting 2 seconds in between attempts before failing and 2) choosing random ip address from a list of rabbitmq cluster hosts. * CSCMETAX-61: [REF] Redis config key updates * CSCMETAX-61: [REF] access_type is again compulsory in harvester dataset schema. * CSCMETAX-455: [REF] Add additionalproperties to schemas. Fix some found problems in test data and app code. Note: test data fetch_and_update does not work in this commit. * CSCMETAX-455: [REF] Change some of the additionalProperties: false flags to true since some objects do have properties that are not visible in the json schema. Fix api code reference data autopopulation for the parts that were actually not properly done before. * CSCMETAX-455: [REF] Fix tests * CSCMETAX-455: [REF] Fix tests * CSCMETAX-413: [ADD] event_outcome to dataset schemas, full test datas. Add autopopulating event_outcome reference data and tests for it. * CSCMETAX-61: [ADD] homepage association for agent object in dataset schemas.
- Loading branch information
