forked from securefederatedai/openfl
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Automation - OpenFL Docker implementation for TaskRunner API (securef…
…ederatedai#1201) * First commit from vRA machine Signed-off-by: noopur <[email protected]> * From azure machine, e2e tested Signed-off-by: noopur <[email protected]> * From azure machine, e2e tested Signed-off-by: noopur <[email protected]> * Install dependencies on collaborators Signed-off-by: noopur <[email protected]> * Workflow to run docker Signed-off-by: noopur <[email protected]> * Workflow to run docker Signed-off-by: noopur <[email protected]> * Workflow to run docker Signed-off-by: noopur <[email protected]> * Step to build openfl image Signed-off-by: noopur <[email protected]> * Better logging Signed-off-by: noopur <[email protected]> * After task runner test run Signed-off-by: noopur <[email protected]> * Revert docker run in workflow yaml Signed-off-by: noopur <[email protected]> * Sequential collab func execution Signed-off-by: noopur <[email protected]> * Handle csr error for parallel run Signed-off-by: noopur <[email protected]> * Multiple changes Signed-off-by: noopur <[email protected]> * Changes Signed-off-by: noopur <[email protected]> * Changes Signed-off-by: noopur <[email protected]> * Correction from tls to use_tls Signed-off-by: noopur <[email protected]> * Docker job Signed-off-by: noopur <[email protected]> * Copy cols.yaml with sudo Signed-off-by: noopur <[email protected]> * Do not include data folder in artifacts Signed-off-by: noopur <[email protected]> * Printenv for testing Signed-off-by: noopur <[email protected]> * 7 min timeout Signed-off-by: noopur <[email protected]> * Write result xml to correct path Signed-off-by: noopur <[email protected]> * Pass no proxy to container creation cmd Signed-off-by: noopur <[email protected]> * Pass no proxy to container creation cmd Signed-off-by: noopur <[email protected]> * Pass no proxy to container creation cmd Signed-off-by: noopur <[email protected]> * Revert 7 to 15min Signed-off-by: noopur <[email protected]> * Fetch non-empty lines from log files Signed-off-by: noopur <[email protected]> * Run docker with keras Signed-off-by: noopur <[email protected]> * Correct file check logic Signed-off-by: noopur <[email protected]> * Exclude cert and data folders from tar Signed-off-by: noopur <[email protected]> * Final changes Signed-off-by: noopur <[email protected]> * Updated README.md file Signed-off-by: noopur <[email protected]> * Temp change Signed-off-by: noopur <[email protected]> * Custom exceptions added Signed-off-by: noopur <[email protected]> * Remove unwanted logging Signed-off-by: noopur <[email protected]> * Update cols.yaml from scratch Signed-off-by: noopur <[email protected]> * Comment updated Signed-off-by: noopur <[email protected]> * Added Docker workflow Signed-off-by: noopur <[email protected]> * Run tls, non-tls and no client auth with diff python versions Signed-off-by: noopur <[email protected]> --------- Signed-off-by: noopur <[email protected]>
- Loading branch information
1 parent
f622efa
commit 8ab9e3c
Showing
18 changed files
with
1,554 additions
and
580 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,233 @@ | ||
--- | ||
#--------------------------------------------------------------------------- | ||
# Workflow to run Task Runner E2E tests via Docker | ||
# Authors - Noopur, Payal Chaurasiya | ||
#--------------------------------------------------------------------------- | ||
name: Task Runner E2E via Docker | ||
|
||
on: | ||
workflow_dispatch: | ||
inputs: | ||
num_rounds: | ||
description: "Number of rounds to train" | ||
required: false | ||
default: "5" | ||
type: string | ||
num_collaborators: | ||
description: "Number of collaborators" | ||
required: false | ||
default: "2" | ||
type: string | ||
|
||
permissions: | ||
contents: read | ||
|
||
# Environment variables common for all the jobs | ||
env: | ||
NUM_ROUNDS: ${{ inputs.num_rounds || '5' }} | ||
NUM_COLLABORATORS: ${{ inputs.num_collaborators || '2' }} | ||
|
||
jobs: | ||
test_with_tls_docker: | ||
name: tr_tls_docker | ||
runs-on: ubuntu-22.04 | ||
timeout-minutes: 15 | ||
strategy: | ||
matrix: | ||
# There are open issues for some of the models, so excluding them for now: | ||
# model_name: [ "torch_cnn_mnist", "keras_cnn_mnist", "torch_cnn_histology" ] | ||
model_name: ["keras_cnn_mnist"] | ||
python_version: ["3.9"] | ||
fail-fast: false # do not immediately fail if one of the combinations fail | ||
|
||
env: | ||
MODEL_NAME: ${{ matrix.model_name }} | ||
PYTHON_VERSION: ${{ matrix.python_version }} | ||
|
||
steps: | ||
- name: Checkout OpenFL repository | ||
id: checkout_openfl | ||
uses: actions/[email protected] | ||
with: | ||
fetch-depth: 2 # needed for detecting changes | ||
submodules: "true" | ||
token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Python | ||
id: setup_python | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: ${{ env.PYTHON_VERSION }} | ||
|
||
- name: Install dependencies | ||
id: install_dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
pip install -r test-requirements.txt | ||
- name: Run Task Runner E2E tests with TLS | ||
id: run_tests | ||
run: | | ||
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \ | ||
-m docker --model_name ${{ env.MODEL_NAME }} \ | ||
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} | ||
echo "Task runner end to end test run completed" | ||
- name: Print test summary | ||
id: print_test_summary | ||
if: ${{ always() }} | ||
run: | | ||
export PYTHONPATH="$PYTHONPATH:." | ||
python tests/end_to_end/utils/summary_helper.py | ||
echo "Test summary printed" | ||
- name: Create Tar (exclude cert and data folders) | ||
id: tar_files | ||
if: ${{ always() }} | ||
run: | | ||
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results | ||
- name: Upload Artifacts | ||
id: upload_artifacts | ||
uses: actions/upload-artifact@v4 | ||
if: ${{ always() }} | ||
with: | ||
name: tr_tls_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }} | ||
path: result.tar | ||
|
||
test_with_non_tls_docker: | ||
name: tr_non_tls_docker | ||
runs-on: ubuntu-22.04 | ||
timeout-minutes: 15 | ||
strategy: | ||
matrix: | ||
# Testing non TLS scenario only for keras_cnn_mnist model and python 3.10 | ||
# If required, this can be extended to other models and python versions | ||
model_name: ["keras_cnn_mnist"] | ||
python_version: ["3.10"] | ||
fail-fast: false # do not immediately fail if one of the combinations fail | ||
|
||
env: | ||
MODEL_NAME: ${{ matrix.model_name }} | ||
PYTHON_VERSION: ${{ matrix.python_version }} | ||
|
||
steps: | ||
- name: Checkout OpenFL repository | ||
id: checkout_openfl | ||
uses: actions/[email protected] | ||
with: | ||
fetch-depth: 2 # needed for detecting changes | ||
submodules: "true" | ||
token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Python | ||
id: setup_python | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: ${{ env.PYTHON_VERSION }} | ||
|
||
- name: Install dependencies | ||
id: install_dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
pip install -r test-requirements.txt | ||
- name: Run Task Runner E2E tests without TLS | ||
id: run_tests | ||
run: | | ||
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \ | ||
-m docker --model_name ${{ env.MODEL_NAME }} \ | ||
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} --disable_tls | ||
echo "Task runner end to end test run completed" | ||
- name: Print test summary | ||
id: print_test_summary | ||
if: ${{ always() }} | ||
run: | | ||
export PYTHONPATH="$PYTHONPATH:." | ||
python tests/end_to_end/utils/summary_helper.py | ||
echo "Test summary printed" | ||
- name: Create Tar (exclude cert and data folders) | ||
id: tar_files | ||
if: ${{ always() }} | ||
run: | | ||
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results | ||
- name: Upload Artifacts | ||
id: upload_artifacts | ||
uses: actions/upload-artifact@v4 | ||
if: ${{ always() }} | ||
with: | ||
name: tr_non_tls_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }} | ||
path: result.tar | ||
|
||
test_with_no_client_auth_docker: | ||
name: tr_no_client_auth_docker | ||
runs-on: ubuntu-22.04 | ||
timeout-minutes: 15 | ||
strategy: | ||
matrix: | ||
# Testing non TLS scenario only for keras_cnn_mnist model and python 3.10 | ||
# If required, this can be extended to other models and python versions | ||
model_name: ["keras_cnn_mnist"] | ||
python_version: ["3.11"] | ||
fail-fast: false # do not immediately fail if one of the combinations fail | ||
|
||
env: | ||
MODEL_NAME: ${{ matrix.model_name }} | ||
PYTHON_VERSION: ${{ matrix.python_version }} | ||
|
||
steps: | ||
- name: Checkout OpenFL repository | ||
id: checkout_openfl | ||
uses: actions/[email protected] | ||
with: | ||
fetch-depth: 2 # needed for detecting changes | ||
submodules: "true" | ||
token: ${{ secrets.GITHUB_TOKEN }} | ||
|
||
- name: Set up Python | ||
id: setup_python | ||
uses: actions/setup-python@v3 | ||
with: | ||
python-version: ${{ env.PYTHON_VERSION }} | ||
|
||
- name: Install dependencies | ||
id: install_dependencies | ||
run: | | ||
python -m pip install --upgrade pip | ||
pip install . | ||
pip install -r test-requirements.txt | ||
- name: Run Task Runner E2E tests without TLS | ||
id: run_tests | ||
run: | | ||
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \ | ||
-m docker --model_name ${{ env.MODEL_NAME }} \ | ||
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} --disable_client_auth | ||
echo "Task runner end to end test run completed" | ||
- name: Print test summary | ||
id: print_test_summary | ||
if: ${{ always() }} | ||
run: | | ||
export PYTHONPATH="$PYTHONPATH:." | ||
python tests/end_to_end/utils/summary_helper.py | ||
echo "Test summary printed" | ||
- name: Create Tar (exclude cert and data folders) | ||
id: tar_files | ||
if: ${{ always() }} | ||
run: | | ||
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results | ||
- name: Upload Artifacts | ||
id: upload_artifacts | ||
uses: actions/upload-artifact@v4 | ||
if: ${{ always() }} | ||
with: | ||
name: tr_no_client_auth_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }} | ||
path: result.tar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,6 @@ | ||
docker | ||
lxml==5.3.0 | ||
paramiko | ||
pytest==8.3.4 | ||
pytest-asyncio==0.24.0 | ||
pytest-mock==3.14.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.