Skip to content

Commit

Permalink
Automation - OpenFL Docker implementation for TaskRunner API (securef…
Browse files Browse the repository at this point in the history
…ederatedai#1201)

* First commit from vRA machine

Signed-off-by: noopur <[email protected]>

* From azure machine, e2e tested

Signed-off-by: noopur <[email protected]>

* From azure machine, e2e tested

Signed-off-by: noopur <[email protected]>

* Install dependencies on collaborators

Signed-off-by: noopur <[email protected]>

* Workflow to run docker

Signed-off-by: noopur <[email protected]>

* Workflow to run docker

Signed-off-by: noopur <[email protected]>

* Workflow to run docker

Signed-off-by: noopur <[email protected]>

* Step to build openfl image

Signed-off-by: noopur <[email protected]>

* Better logging

Signed-off-by: noopur <[email protected]>

* After task runner test run

Signed-off-by: noopur <[email protected]>

* Revert docker run in workflow yaml

Signed-off-by: noopur <[email protected]>

* Sequential collab func execution

Signed-off-by: noopur <[email protected]>

* Handle csr error for parallel run

Signed-off-by: noopur <[email protected]>

* Multiple changes

Signed-off-by: noopur <[email protected]>

* Changes

Signed-off-by: noopur <[email protected]>

* Changes

Signed-off-by: noopur <[email protected]>

* Correction from tls to use_tls

Signed-off-by: noopur <[email protected]>

* Docker job

Signed-off-by: noopur <[email protected]>

* Copy cols.yaml with sudo

Signed-off-by: noopur <[email protected]>

* Do not include data folder in artifacts

Signed-off-by: noopur <[email protected]>

* Printenv for testing

Signed-off-by: noopur <[email protected]>

* 7 min timeout

Signed-off-by: noopur <[email protected]>

* Write result xml to correct path

Signed-off-by: noopur <[email protected]>

* Pass no proxy to container creation cmd

Signed-off-by: noopur <[email protected]>

* Pass no proxy to container creation cmd

Signed-off-by: noopur <[email protected]>

* Pass no proxy to container creation cmd

Signed-off-by: noopur <[email protected]>

* Revert 7 to 15min

Signed-off-by: noopur <[email protected]>

* Fetch non-empty lines from log files

Signed-off-by: noopur <[email protected]>

* Run docker with keras

Signed-off-by: noopur <[email protected]>

* Correct file check logic

Signed-off-by: noopur <[email protected]>

* Exclude cert and data folders from tar

Signed-off-by: noopur <[email protected]>

* Final changes

Signed-off-by: noopur <[email protected]>

* Updated README.md file

Signed-off-by: noopur <[email protected]>

* Temp change

Signed-off-by: noopur <[email protected]>

* Custom exceptions added

Signed-off-by: noopur <[email protected]>

* Remove unwanted logging

Signed-off-by: noopur <[email protected]>

* Update cols.yaml from scratch

Signed-off-by: noopur <[email protected]>

* Comment updated

Signed-off-by: noopur <[email protected]>

* Added Docker workflow

Signed-off-by: noopur <[email protected]>

* Run tls, non-tls and no client auth with diff python versions

Signed-off-by: noopur <[email protected]>

---------

Signed-off-by: noopur <[email protected]>
  • Loading branch information
noopurintel authored Dec 12, 2024
1 parent f622efa commit 8ab9e3c
Show file tree
Hide file tree
Showing 18 changed files with 1,554 additions and 580 deletions.
233 changes: 233 additions & 0 deletions .github/workflows/task_runner_docker_e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
---
#---------------------------------------------------------------------------
# Workflow to run Task Runner E2E tests via Docker
# Authors - Noopur, Payal Chaurasiya
#---------------------------------------------------------------------------
name: Task Runner E2E via Docker

on:
workflow_dispatch:
inputs:
num_rounds:
description: "Number of rounds to train"
required: false
default: "5"
type: string
num_collaborators:
description: "Number of collaborators"
required: false
default: "2"
type: string

permissions:
contents: read

# Environment variables common for all the jobs
env:
NUM_ROUNDS: ${{ inputs.num_rounds || '5' }}
NUM_COLLABORATORS: ${{ inputs.num_collaborators || '2' }}

jobs:
test_with_tls_docker:
name: tr_tls_docker
runs-on: ubuntu-22.04
timeout-minutes: 15
strategy:
matrix:
# There are open issues for some of the models, so excluding them for now:
# model_name: [ "torch_cnn_mnist", "keras_cnn_mnist", "torch_cnn_histology" ]
model_name: ["keras_cnn_mnist"]
python_version: ["3.9"]
fail-fast: false # do not immediately fail if one of the combinations fail

env:
MODEL_NAME: ${{ matrix.model_name }}
PYTHON_VERSION: ${{ matrix.python_version }}

steps:
- name: Checkout OpenFL repository
id: checkout_openfl
uses: actions/[email protected]
with:
fetch-depth: 2 # needed for detecting changes
submodules: "true"
token: ${{ secrets.GITHUB_TOKEN }}

- name: Set up Python
id: setup_python
uses: actions/setup-python@v3
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install dependencies
id: install_dependencies
run: |
python -m pip install --upgrade pip
pip install .
pip install -r test-requirements.txt
- name: Run Task Runner E2E tests with TLS
id: run_tests
run: |
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \
-m docker --model_name ${{ env.MODEL_NAME }} \
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }}
echo "Task runner end to end test run completed"
- name: Print test summary
id: print_test_summary
if: ${{ always() }}
run: |
export PYTHONPATH="$PYTHONPATH:."
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: tr_tls_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }}
path: result.tar

test_with_non_tls_docker:
name: tr_non_tls_docker
runs-on: ubuntu-22.04
timeout-minutes: 15
strategy:
matrix:
# Testing non TLS scenario only for keras_cnn_mnist model and python 3.10
# If required, this can be extended to other models and python versions
model_name: ["keras_cnn_mnist"]
python_version: ["3.10"]
fail-fast: false # do not immediately fail if one of the combinations fail

env:
MODEL_NAME: ${{ matrix.model_name }}
PYTHON_VERSION: ${{ matrix.python_version }}

steps:
- name: Checkout OpenFL repository
id: checkout_openfl
uses: actions/[email protected]
with:
fetch-depth: 2 # needed for detecting changes
submodules: "true"
token: ${{ secrets.GITHUB_TOKEN }}

- name: Set up Python
id: setup_python
uses: actions/setup-python@v3
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install dependencies
id: install_dependencies
run: |
python -m pip install --upgrade pip
pip install .
pip install -r test-requirements.txt
- name: Run Task Runner E2E tests without TLS
id: run_tests
run: |
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \
-m docker --model_name ${{ env.MODEL_NAME }} \
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} --disable_tls
echo "Task runner end to end test run completed"
- name: Print test summary
id: print_test_summary
if: ${{ always() }}
run: |
export PYTHONPATH="$PYTHONPATH:."
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: tr_non_tls_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }}
path: result.tar

test_with_no_client_auth_docker:
name: tr_no_client_auth_docker
runs-on: ubuntu-22.04
timeout-minutes: 15
strategy:
matrix:
# Testing non TLS scenario only for keras_cnn_mnist model and python 3.10
# If required, this can be extended to other models and python versions
model_name: ["keras_cnn_mnist"]
python_version: ["3.11"]
fail-fast: false # do not immediately fail if one of the combinations fail

env:
MODEL_NAME: ${{ matrix.model_name }}
PYTHON_VERSION: ${{ matrix.python_version }}

steps:
- name: Checkout OpenFL repository
id: checkout_openfl
uses: actions/[email protected]
with:
fetch-depth: 2 # needed for detecting changes
submodules: "true"
token: ${{ secrets.GITHUB_TOKEN }}

- name: Set up Python
id: setup_python
uses: actions/setup-python@v3
with:
python-version: ${{ env.PYTHON_VERSION }}

- name: Install dependencies
id: install_dependencies
run: |
python -m pip install --upgrade pip
pip install .
pip install -r test-requirements.txt
- name: Run Task Runner E2E tests without TLS
id: run_tests
run: |
python -m pytest -s tests/end_to_end/test_suites/docker_tests.py \
-m docker --model_name ${{ env.MODEL_NAME }} \
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} --disable_client_auth
echo "Task runner end to end test run completed"
- name: Print test summary
id: print_test_summary
if: ${{ always() }}
run: |
export PYTHONPATH="$PYTHONPATH:."
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
uses: actions/upload-artifact@v4
if: ${{ always() }}
with:
name: tr_no_client_auth_docker_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }}
path: result.tar
15 changes: 9 additions & 6 deletions .github/workflows/task_runner_e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,11 @@ jobs:
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Tar files
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: tar -cvf result.tar results
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
Expand Down Expand Up @@ -151,10 +152,11 @@ jobs:
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Tar files
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: tar -cvf result.tar results
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
Expand Down Expand Up @@ -218,10 +220,11 @@ jobs:
python tests/end_to_end/utils/summary_helper.py
echo "Test summary printed"
- name: Tar files
- name: Create Tar (exclude cert and data folders)
id: tar_files
if: ${{ always() }}
run: tar -cvf result.tar results
run: |
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" $HOME/results
- name: Upload Artifacts
id: upload_artifacts
Expand Down
2 changes: 2 additions & 0 deletions test-requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
docker
lxml==5.3.0
paramiko
pytest==8.3.4
pytest-asyncio==0.24.0
pytest-mock==3.14.0
10 changes: 9 additions & 1 deletion tests/end_to_end/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,18 @@ python -m pytest -s tests/end_to_end/test_suites/task_runner_tests.py --num_roun

```
results
├── <workspace_name> # Based on the workspace name provided during test run.
├── <workspace_name> # Same as model name used for testing.
├── aggregator
├── workspace # containing aggregator specific files and folders
├── collaborator1
├── workspace # containing collaborator1 specific files and folders
├── ....
├── collaborator<n>
├── workspace # containing collaborator<n> specific files and folders
├── results.xml # Output file in JUNIT.
└── deployment.log # Log file containing step by step test progress.
```
Folders excluded for all the participants - cert and data.

## Contribution

Expand Down
Loading

0 comments on commit 8ab9e3c

Please sign in to comment.