Skip to content

Commit

Permalink
[major] Refactor the Executor interface (#548)
Browse files Browse the repository at this point in the history
* [major] Refactor the Executor interface

Replace the backend parameter from the initialization of the Executor class with individual executor classes for the individual backends:
* `Executor(backend="local")` becomes `from executorlib import LocalExecutor§
* `Executor(backend="flux_allocation")` becomes `from executorlib import FluxAllocationExecutor§
* `Executor(backend="flux_submission")` becomes `from executorlib import FluxSubmissionExecutor§
* `Executor(backend="slurm_allocation")` becomes "SlurmAllocationExecutor"
* `Executor(backend="slurm_submission")` becomes `from executorlib import SlurmSubmissionExecutor`

This has two advantages: On the one hand it is less error prone to mistyping the backend name, as the user can use auto completion to import the right module. On the other hand it is more consistent with the standard library which defines the `ProcessPoolExecutor` and the `ThreadPoolExecutor`, rather than a `backend` parameter.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

* update benchmark

* move interfaces to individual modules

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove create

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove redundant functionality

* Move graph tests

* remove core check

* clean up

* more tests

* fix import

* more fixes

* Rename LocalExecutor to SingleNodeExecutor

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test block allocation

* fix

* flux fix

* Rename interfaces

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix docstrings

* add docstrings to create_*_executor() function

* fix test

* merge

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
  • Loading branch information
jan-janssen and pre-commit-ci[bot] authored Feb 3, 2025
1 parent 11d44cc commit d95d6ca
Show file tree
Hide file tree
Showing 24 changed files with 1,820 additions and 773 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/unittest-flux-mpich.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,4 @@ jobs:
timeout-minutes: 5
run: >
flux start
python -m unittest tests/test_flux_executor.py tests/test_executor_backend_flux.py tests/test_cache_executor_pysqa_flux.py;
python -m unittest tests/test_flux_executor.py tests/test_executor_backend_flux.py tests/test_cache_executor_pysqa_flux.py tests/test_plot_dependency_flux.py;
2 changes: 1 addition & 1 deletion .github/workflows/unittest-flux-openmpi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
timeout-minutes: 5
run: >
flux start
coverage run -a --omit="executorlib/_version.py,tests/*" -m unittest tests/test_flux_executor.py tests/test_executor_backend_flux.py tests/test_cache_executor_pysqa_flux.py;
coverage run -a --omit="executorlib/_version.py,tests/*" -m unittest tests/test_flux_executor.py tests/test_executor_backend_flux.py tests/test_cache_executor_pysqa_flux.py tests/test_plot_dependency_flux.py;
coverage xml
env:
PYMPIPOOL_PMIX: "pmix"
Expand Down
260 changes: 14 additions & 246 deletions executorlib/__init__.py

Large diffs are not rendered by default.

295 changes: 0 additions & 295 deletions executorlib/interactive/create.py

This file was deleted.

Empty file.
Loading

0 comments on commit d95d6ca

Please sign in to comment.