From b869b0ea0df0915331e497adcffbf31c87ca1c29 Mon Sep 17 00:00:00 2001 From: Sebastiaan Huber Date: Fri, 14 Oct 2022 11:29:55 +0200 Subject: [PATCH] Release `v2.1.0` --- CHANGELOG.md | 323 +++++++++++++++++++++++++++++++++++++++++++++- aiida/__init__.py | 2 +- 2 files changed, 323 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e88f7a290..67f1674bb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,326 @@ # Changelog +## v2.1.0 - 2022-11-07 + +This feature release comes with a number of new features as well as quite a few fixes of bugs and stability issues. +Further down you will find a complete list of changes, after a short description of some of the most important changes: + +- [Automatic input serialization in calculation and work functions](#automatic-input-serialization-in-calculation-and-work-functions) +- [Improved interface for creating codes](#improved-interface-for-creating-codes) +- [Support for running code in containers](#support-for-running-code-in-containers) +- [Control daemon and processes from the API](#control-daemon-and-processes-from-the-api) +- [REST API can serve multiple profiles](#rest-api-can-serve-multiple-profiles) +- [Pluginable data storage backends](#pluginable-data-storage-backends) +- [Full list of changes](#full-list-of-changes) + +### Automatic input serialization in calculation and work functions + +The inputs to `calcfunction`s and `workfunction`s are now automatically converted to AiiDA data types if they are one of the basic Python types (`bool`, `dict`, `Enum`, `float`, `int`, `list` or `str`). +This means that code that looked like: + +```python +from aiida.engine import calcfunction +from aiida.orm import Bool, Float, Int, Str + +@calcfunction +def function(switch, threshold, count, label): + ... + +function(Bool(True), Float(0.25), Int(10), Str('some-label')) +``` + +can now be simplified to: + +```python +from aiida.engine import calcfunction +from aiida.orm import Bool, Float, Int, Str + +@calcfunction +def function(switch, threshold, count, label): + ... + +function(True, 0.25, 10, 'some-label') +``` + +### Improved interface for creating codes + +The `Code` data plugin was a single class that served two different types of codes: "remote" codes and "local" codes. +These names "remote" and "local" have historically caused a lot of confusion. +Likewise, using a single class `Code` for both implementations also has led to confusing interfaces. + +To address this issue, the functionality has been split into two new classes [`InstalledCode`](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#installedcode) and [`PortableCode`](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#portablecode), that replace the "remote" and "local" code, respectively. +The installed code represents an executable binary that is already pre-installed on some compute resource. +The portable code represents a code (executable plus any additional required files) that are stored in AiiDA's storage and can be automatically transfered to any computer before being executed. + +Creating a new instance of these new code types is easy: +```python +from pathlib import Path +from aiida.orm import InstalledCode, PortableCode + +installed_code = InstalledCode( + label='installed-code', + computer=load_computer('localhost'), + filepath_executable='/usr/bin/bash' +) + +portable_code = PortableCode( + label='portable-code', + filepath_files=Path('/some/path/code'), + filepath_executable='executable.exe' +) +``` + +Codes can also be created through the new `verdi` command `verdi code create`. +To specify the type of code to create, pass the corresponding entry point name as an argument. +For example, to create a new installed code, invoke: +```bash +verdi code create core.code.installed +```` +The options for each subcommand are automatically generated based on the code type, and so only options that are relevant to that code type will be prompted for. + +The new code classes both subclass the `aiida.orm.nodes.data.code.abstract.AbstractCode` base class. +This means that both `InstalledCode`s and `PortableCode`s can be used as the `code` input for `CalcJob`s without problems. + +The old `Code` class remains supported for the time being as well, however, it is deprecated and will be remove at some point. +The same goes for the `verdi code setup` command; please use `verdi code create` instead. +Existing codes will be automatically migrated to either an `InstalledCode` or a `PortableCode`. +It is strongly advised that you update any code that creates new codes to use these new plugin types. + +### Support for running code in containers + +Support is added to run calculation jobs inside a container. +A containerized code can be setup through the CLI: +```bash +verdi code create core.code.containerized \ + --label containerized \ + --image-name docker://alpine:3 \ + --filepath-executable /bin/sh \ + --engine-command "singularity exec --bind $PWD:$PWD {image_name}" +``` +as well as through the API: +```python +from aiida.orm import ContainerizedCode, load_computer +code = ContainerizedCode( + computer=load_computer('some-computer') + filepath_executable='/bin/sh' + image_name='docker://alpine:3', + engine_command='singularity exec --bind $PWD:$PWD {image_name}' +).store() +``` +In the example above we use the [Singularity](https://singularity-docs.readthedocs.io/en/latest/) containerization technology. +For more information on what containerization programs are supported and how to configure them, please refer to the [documentation](https://aiida.readthedocs.io/projects/aiida-core/en/v2.1.0/topics/data_types.html#containerizedcode). + +### Control daemon and processes from the API + +Up till now, the daemon and live processes could only easily be controlled through `verdi daemon` and `verdi process`, respectively. +In this release, modules are added to provide the same functionality through the Python API. + +#### Daemon API + +The daemon can now be started and stopped through the `DaemonClient` which can be obtained through the `get_daemon_client` utility function: +```python +from aiida.engine.daemon.client import get_daemon_client +client = get_daemon_client() +``` +By default, this will give the daemon client for the current default profile. +It is also possible to explicitly specify a profile: +```python +client = get_daemon_client(profile='some-profile') +``` +The daemon can be started and stopped through the client: +```python +client.start_daemon() +assert client.is_daemon_running +client.stop_daemon(wait=True) +``` + +#### Process API + +The functionality of `verdi process` to `play`, `pause` and `kill` is now made available through the `aiida.engine.process.control` module. +Processes can be played, paused or killed through the `play_processes`, `pause_processes`, and `kill_processes`, respectively. +The processes to act upon are defined through their `ProcessNode` which can be loaded using `load_node`. +```python +from aiida.engine.process import control + +processes = [load_node(), load_node()] + +pause_processes(processes) # Pause the processes +play_processes(processes) # Play them again +kill_processes(processes) # Kill the processes +``` +Instead of specifying an explicit list of processes, the functions also take the `all_entries` keyword argument: +```python +pause_processes(all_entries=True) # Pause all running processes +``` + +### REST API can serve multiple profiles + +Before, a single REST API could only serve data of a single profile at a time. +This limitation has been removed and a single REST API instance can now serve data from all profiles of an AiiDA instance. +To maintain backwards compatibility, the new functionality needs to be explicitly enabled through the configuration: +```bash +verdi config set rest_api.profile_switching True +``` +After the REST API is restarted, it will now accept the `profile` query parameter, for example: +```bash +http://127.0.0.1:5000/api/v4/computers?profile=some-profile-name +```` +If the specified is already loaded, the REST API functions exactly as without profile switching enabled. +If another profile is specified, the REST API will first switch profiles before executing the request. + +If the profile parameter is specified in a request and the REST API does not have profile switching enabled, a 400 response is returned. + +### Pluginable data storage backends + +Warning: this is beta functionality. +It is now possible to implement custom storage backends to control where all data of an AiiDA profile is stored. +To provide a data storage plugin, one should implement the `aiida.orm.implementation.storage_backend.StorageBackend` interface. +The default implementation provided by `aiida-core` is the `aiida.storage.psql_dos.backend.PsqlDosBackend` which uses a PostgreSQL database for the provenance graph and a [`disk-objectstore`](#https://pypi.org/project/disk-objectstore/) container for repository files. + +Storage backend plugins should be registered in the new entry point group `aiida.storage`. +The default storage backend `PsqlDosBackend` has the `core.psql_dos` entry point name. + +The storage backend to be used for a profile can be specified using the `--db-backend` option in `verdi setup` and `verdi quicksetup`. +The entry point of the selected backend is stored in the `storage.backend` key of a profile configuration: +```json +{ + "profiles": { + "profile-name": { + "PROFILE_UUID": "", + "storage": { + "backend": "core.psql_dos", + "config": {} + }, + "process_control": {}, + "default_user_email": "aiida@localhost", + "test_profile": false + }, + +} +``` + +At the moment, it is not quite clear if the abstract interface `StorageBackend` properly abstracts everything that is needed to implement any storage backend. +For the time being then, it is advised to subclass the `PsqlDosBackend` and replace parts required for the use-case, such as just replacing the file repository implementation. + + +### Full list of changes + +#### Features +- `Process`: Add hook to customize the `process_label` attribute [[#5713]](https://github.com/aiidateam/aiida-core/pull/5713) +- Add the `ContainerizedCode` data plugin [[#5667]](https://github.com/aiidateam/aiida-core/pull/5667) +- API: Add the `aiida.engine.processes.control` module [[#5630]](https://github.com/aiidateam/aiida-core/pull/5630) +- `PluginVersionProvider`: Add support for entry point strings [[#5662]](https://github.com/aiidateam/aiida-core/pull/5662) +- `verdi setup`: Add the `--profile-uuid` option [[#5673]](https://github.com/aiidateam/aiida-core/pull/5673) +- Process control: Add the `revive_processes` method [[#5677]](https://github.com/aiidateam/aiida-core/pull/5677) +- Process functions: Add the `get_source_code_function` method [[#4554]](https://github.com/aiidateam/aiida-core/pull/4554) +- CLI: Improve the quality of `verdi code list` output [[#5750]](https://github.com/aiidateam/aiida-core/pull/5750) +- CLI: Add the `verdi devel revive` command [[#5677]](https://github.com/aiidateam/aiida-core/pull/5677) +- CLI: `verdi process status --max-depth` [[#5727]](https://github.com/aiidateam/aiida-core/pull/5727) +- CLI: `verdi setup/quicksetup` store autofill user info early [[#5729]](https://github.com/aiidateam/aiida-core/pull/5729) +- CLI: Add the `devel launch-add` command [[#5733]](https://github.com/aiidateam/aiida-core/pull/5733) +- CLI: Make filename in `verdi node repo cat` optional for `SinglefileData` [[#5747]](https://github.com/aiidateam/aiida-core/pull/5747) +- CLI: Add the `verdi devel rabbitmq` command group [[#5718]](https://github.com/aiidateam/aiida-core/pull/5718) +- API: Add function to start the daemon [[#5625]](https://github.com/aiidateam/aiida-core/pull/5625) +- `BaseRestartWorkChain`: add the `get_outputs` hook [[#5618]](https://github.com/aiidateam/aiida-core/pull/5618) +- `CalcJob`: extend `retrieve_list` syntax with `depth=None` [[#5651]](https://github.com/aiidateam/aiida-core/pull/5651) +- `CalcJob`: allow wildcards in `stash.source_list` paths [[#5601]](https://github.com/aiidateam/aiida-core/pull/5601) +- Add global config option `rest_api.profile_switching` [[#5054]](https://github.com/aiidateam/aiida-core/pull/5054) +- REST API: make the profile configurable as request parameter [[#5054]](https://github.com/aiidateam/aiida-core/pull/5054) +- `ProcessFunction`: Automatically serialize Python base type inputs [[#5688]](https://github.com/aiidateam/aiida-core/pull/5688) +- `BaseRestartWorkChain`: allow to override priority in `handler_overrides` [[#5546]](https://github.com/aiidateam/aiida-core/pull/5546) +- ORM: add `entry_point` classproperty to `Node` and `Group` [[#5437]](https://github.com/aiidateam/aiida-core/pull/5437) +- Add the `aiida.storage` entry point group [[#5501]](https://github.com/aiidateam/aiida-core/pull/5501) +- Add the config option `storage.sandbox` [[#5501]](https://github.com/aiidateam/aiida-core/pull/5501) +- Add the `InstalledCode` and `PortableCode` data plugins [[#5510]](https://github.com/aiidateam/aiida-core/pull/5510) +- CLI: Add the `verdi code create` command group [[#5510]](https://github.com/aiidateam/aiida-core/pull/5510) +- CLI: Add the `DynamicEntryPointCommandGroup` command group [[#5510]](https://github.com/aiidateam/aiida-core/pull/5510) +- Add a client to connect to RabbitMQ Manamegement HTTP API [[#5718]](https://github.com/aiidateam/aiida-core/pull/5718) +- `LsfScheduler`: add support for `num_machines` [[#5153]](https://github.com/aiidateam/aiida-core/pull/5153) +- `JobResource`: add the `accepts_default_memory_per_machine` [[#5642]](https://github.com/aiidateam/aiida-core/pull/5642) +- `AbstractCode`: add abstraction methods for command line parameters [[#5664]](https://github.com/aiidateam/aiida-core/pull/5664) +- `ArithmeticAddCalculation`: Add the `metadata.options.sleep` input [[#5663]](https://github.com/aiidateam/aiida-core/pull/5663) +- `DaemonClient`: add the `get_env` method [[#5631]](https://github.com/aiidateam/aiida-core/pull/5631) +- Tests: Make daemon fixtures available to plugin packages [[#5701]](https://github.com/aiidateam/aiida-core/pull/5701) +- `verdi plugin list`: Show which exit codes invalidate cache [[#5710]](https://github.com/aiidateam/aiida-core/pull/5710) +- `verdi plugin list`: Show full help for input and output ports [[#5711]](https://github.com/aiidateam/aiida-core/pull/5711) + +#### Fixes +- `ArrayData`: replace `nan` and `inf` with `None` when dumping to JSON [[#5613]](https://github.com/aiidateam/aiida-core/pull/5613) +- Archive: add missing migration of transport entry points [[#5604]](https://github.com/aiidateam/aiida-core/pull/5604) +- `BaseRestartWorkChain`: fix `handler_overrides` ignoring `enabled=False` [[#5598]](https://github.com/aiidateam/aiida-core/pull/5598) +- CLI: allow setting options for config without profiles [[#5544]](https://github.com/aiidateam/aiida-core/pull/5544) +- CLI: normalize use of colors [[#5547]](https://github.com/aiidateam/aiida-core/pull/5547) +- `Config`: fix bug in downgrade past version 6 [[#5528]](https://github.com/aiidateam/aiida-core/pull/5528) +- `DaemonClient`: close `CircusClient` after call [[#5631]](https://github.com/aiidateam/aiida-core/pull/5631) +- Engine: Do not call serializer for `None` values [[#5694]](https://github.com/aiidateam/aiida-core/pull/5694) +- Engine: Do not let `DuplicateSubcriberError` except a `Process` [[#5715]](https://github.com/aiidateam/aiida-core/pull/5715) +- ORM: raise when trying to pickle instance of `Entity` [[#5549]](https://github.com/aiidateam/aiida-core/pull/5549) +- ORM: Return `None` in `get_function_source_code` instead of excepting [[#5730]](https://github.com/aiidateam/aiida-core/pull/5730) +- Fix `get_entry_point` not raising even for duplicate entry points [[#5531]](https://github.com/aiidateam/aiida-core/pull/5531) +- Fix: reference to command in message for `verdi storage maintain` [[#5558]](https://github.com/aiidateam/aiida-core/pull/5558) +- Fix: `is_valid_cache` setter for `ProcessNode`s [[#5583]](https://github.com/aiidateam/aiida-core/pull/5583) +- Fix exception when importing an archive into a profile with many nodes [[#5740]](https://github.com/aiidateam/aiida-core/pull/5740) +- `Profile`: make definition of daemon filepaths dynamic [[#5631]](https://github.com/aiidateam/aiida-core/pull/5631) +- Fixtures: Fix bug in reset of `empty_config` fixture [[#5717]](https://github.com/aiidateam/aiida-core/pull/5717) +- `PsqlDosBackend`: ensure sqla sessions are garbage-collected on `close` [[#5728]](https://github.com/aiidateam/aiida-core/pull/5728) +- `TrajectoryData`: Fix bug in `get_step_data` [[#5734]](https://github.com/aiidateam/aiida-core/pull/5734) +- `ProfileManager`: restart daemon in `clear_profile` [[#5751]](https://github.com/aiidateam/aiida-core/pull/5751) + +#### Changes +- Mark relevant `Process` exit codes as `invalidates_cache=True`[[#5709]](https://github.com/aiidateam/aiida-core/pull/5709) +- `TemplatereplacerCalculation`: Change exit codes to be in 300 range [[#5709]](https://github.com/aiidateam/aiida-core/pull/5709) +- Add the prefix `core.` to all storage entry points [[#5501]](https://github.com/aiidateam/aiida-core/pull/5501) +- `CalcJob`: Fully abstract interaction with `AbstractCode` in presubmit [[#5666]](https://github.com/aiidateam/aiida-core/pull/5666) +- CLI: make label the default group list order in `verdi group list` [[#5523]](https://github.com/aiidateam/aiida-core/pull/5523) +- Config: add migration to properly prefix storage backend [[#5501]](https://github.com/aiidateam/aiida-core/pull/5501) +- Move query utils from `aiida.cmdline` to `aiida.tools` [[#5630]](https://github.com/aiidateam/aiida-core/pull/5630) +- `SandboxFolder`: decouple the location from the profile [[#5496]](https://github.com/aiidateam/aiida-core/pull/5496) +- `TemplatereplacerDoublerParser`: rename and generalize implementation [[#5669]](https://github.com/aiidateam/aiida-core/pull/5669) +- `Process`: Allow `None` for input ports that are not required [[#5722]](https://github.com/aiidateam/aiida-core/pull/5722) + +#### Dependencies +- RabbitMQ: Remove support for v3.5 and older [[#5718]](https://github.com/aiidateam/aiida-core/pull/5718) +- Relax `wrapt` requirement [[#5607]](https://github.com/aiidateam/aiida-core/pull/5607) +- Set upper limit `werkzeug<2.2` [[#5606]](https://github.com/aiidateam/aiida-core/pull/5606) +- Update requirement `click~=8.1` [[#5504]](https://github.com/aiidateam/aiida-core/pull/5504) + +#### Deprecations +- Deprecate `Profile.repository_path` [[#5516]](https://github.com/aiidateam/aiida-core/pull/5516) +- Deprecate: `verdi code setup` and `CodeBuilder` [[#5510]](https://github.com/aiidateam/aiida-core/pull/5510) +- Deprecate the method `aiida.get_strict_version` [[#5512]](https://github.com/aiidateam/aiida-core/pull/5512) +- Remove use of legacy `Code` [[#5510]](https://github.com/aiidateam/aiida-core/pull/5510) + +#### Documentation +- Add section on basic performance benchmark with automated benchmark script [[#5724]](https://github.com/aiidateam/aiida-core/pull/5724) +- Add `-U` flag to PostgreSQL database backup command [[#5550]](https://github.com/aiidateam/aiida-core/pull/5550) +- Clarify excepted and killed calculations are not cached [[#5525]](https://github.com/aiidateam/aiida-core/pull/5525) +- Correct snippet for workchain context nested keys [[#5551]](https://github.com/aiidateam/aiida-core/pull/5551) +- Plugin package setup add PEP 621 example [[#5626]](https://github.com/aiidateam/aiida-core/pull/5626) +- Remove note on disk space for caching [[#5534]](https://github.com/aiidateam/aiida-core/pull/5534) +- Remove explicit release tag in Docker image name [[#5671]](https://github.com/aiidateam/aiida-core/pull/5671) +- Remove example REST API extension with POST requests [[#5737]](https://github.com/aiidateam/aiida-core/pull/5737) +- Resubmit a `Process` from a `ProcessNode` [[#5579]](https://github.com/aiidateam/aiida-core/pull/5579) + +#### Devops +- Add a notification for nightly workflow on fail [[#5605]](https://github.com/aiidateam/aiida-core/pull/5605) +- CI: Remove `--use-feature` flag in `pip install` of CI [[#5703]](https://github.com/aiidateam/aiida-core/pull/5703) +- Fixtures: Add `started_daemon_client` and `stopped_daemon_client` [[#5631]](https://github.com/aiidateam/aiida-core/pull/5631) +- Fixtures: Add the `entry_points` fixture to dynamically add and remove entry points [[#5745]](https://github.com/aiidateam/aiida-core/pull/5745) +- Refactor: `Process` extract `CalcJob` specific input handling from `Process` [[#5539]](https://github.com/aiidateam/aiida-core/pull/5539) +- Refactor: remove unnecessary use of `tempfile.mkdtemp` [[#5639]](https://github.com/aiidateam/aiida-core/pull/5639) +- Refactor: Remove internal use of various deprecated resources [[#5716]](https://github.com/aiidateam/aiida-core/pull/5716) +- Refactor: Turn `aiida.manage.external.rmq` into a package [[#5718]](https://github.com/aiidateam/aiida-core/pull/5718) +- Tests: remove legacy `tests/utils/configuration.py` [[#5500]](https://github.com/aiidateam/aiida-core/pull/5500) +- Tests: fix the RPN work chains for the nightly build [[#5529]](https://github.com/aiidateam/aiida-core/pull/5529) +- Tests: Manually stop daemon after `verdi devel revive` test [[#5689]](https://github.com/aiidateam/aiida-core/pull/5689) +- Tests: Add verbose info if `submit_and_wait` times out [[#5689]](https://github.com/aiidateam/aiida-core/pull/5689) +- Tests: Do not set default memory for `localhost` fixture [[#5689]](https://github.com/aiidateam/aiida-core/pull/5689) +- Tests: Suppress RabbitMQ and developer version warnings [[#5689]](https://github.com/aiidateam/aiida-core/pull/5689) +- Tests: Add the `EntryPointManager` exposed as `entry_points` fixture [[#5656]](https://github.com/aiidateam/aiida-core/pull/5656) +- Tests: Only reset database connection at end of suite [[#5641]](https://github.com/aiidateam/aiida-core/pull/5641) +- Tests: Suppress logging and warnings from temporary profile fixture [[#5702]](https://github.com/aiidateam/aiida-core/pull/5702) + + ## v2.0.4 - 2022-09-22 [Full changelog](https://github.com/aiidateam/aiida-core/compare/v2.0.3...v2.0.4) @@ -36,7 +357,7 @@ Update of the Dockerfile base image (`aiidateam/aiida-prerequisites`) to version [Full changelog](https://github.com/aiidateam/aiida-core/compare/v2.0.0...v2.0.1) ### Dependencies -- Fix incompatibility with `click>=8.1` and require `click==8.1` as a minimum by @sphuber in [#5504](https://github.com/aiidateam/aiida-core/pull/5504) +- Fix incompatibility with `click>=8.1` and require `click==8.1` as a minimum by @sphuber in [[#5504]](https://github.com/aiidateam/aiida-core/pull/5504) ## v2.0.0 - 2022-04-27 diff --git a/aiida/__init__.py b/aiida/__init__.py index d7cf67370a..c6532bb542 100644 --- a/aiida/__init__.py +++ b/aiida/__init__.py @@ -28,7 +28,7 @@ 'For further information please visit http://www.aiida.net/. All rights reserved.' ) __license__ = 'MIT license, see LICENSE.txt file.' -__version__ = '2.0.4.post0' +__version__ = '2.1.0' __authors__ = 'The AiiDA team.' __paper__ = ( 'S. P. Huber et al., "AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and '