Merge branch 'securefederatedai:develop' into straggler_handling_update

securefederatedai · Dec 2, 2024 · 03f2bc1 · 03f2bc1
2 parents e89959b + c25d144
commit 03f2bc1
Show file tree

Hide file tree

Showing 359 changed files with 5,250 additions and 1,974 deletions.
diff --git a/.github/workflows/experimental_workflow_tests.yml b/.github/workflows/experimental_workflow_tests.yml
@@ -24,63 +24,63 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -r tests/github/experimental/requirements_experimental_localruntime_tests.txt
+        pip install -r tests/github/experimental/workflow/LocalRuntime/requirements_experimental_localruntime_tests.txt
         pip install .
     - name: Workflow - Exclude Test (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_exclude.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_exclude.py
     - name: Workflow - Exclude Test (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_exclude.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_exclude.py ray
     - name: Workflow - Include Test (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_include.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_include.py
     - name: Workflow - Include Test (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_include.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_include.py ray
     - name: Workflow - Include / Exclude Test (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_include_exclude.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_include_exclude.py
     - name: Workflow - Include / Exclude Test (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_include_exclude.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_include_exclude.py ray
 
     - name: Workflow - Internal Loop Test (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_internalloop.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_internalloop.py
     - name: Workflow - Internal Loop Test (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_internalloop.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_internalloop.py ray
     - name: Workflow - Private Attributes (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_privateattributes.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_privateattributes.py
     - name: Workflow - Private Attributes (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_privateattributes.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_privateattributes.py ray
     - name: Workflow - References (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_reference.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference.py
     - name: Workflow - References (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_reference.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference.py ray
     - name: Workflow - References with Exclude (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_reference_with_exclude.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference_with_exclude.py
     - name: Workflow - References with Exclude (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_reference_with_exclude.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference_with_exclude.py ray
     - name: Workflow - References with Include (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_reference_with_include.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference_with_include.py
     - name: Workflow - References with Include (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_reference_with_include.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_reference_with_include.py ray
     - name: Workflow - Collaborator Subset (Single Process Backend)
       run: |
-        python tests/github/experimental/testflow_subset_of_collaborators.py
+        python tests/github/experimental/workflow/LocalRuntime/testflow_subset_of_collaborators.py
     - name: Workflow - Collaborator Subset (Ray Backend)
       run: |
-        python tests/github/experimental/testflow_subset_of_collaborators.py ray
+        python tests/github/experimental/workflow/LocalRuntime/testflow_subset_of_collaborators.py ray
     - name: Test Experimental Aggregator Based Workflow API
       run: |
-        python -m tests.github.experimental.workspace.test_experimental_agg_based_workflow --custom_template tests/github/experimental/workspace/testcase_datastore_cli --fed_workspace aggregator --col col1 --col col2  --rounds-to-train 1
+        python -m tests.github.experimental.workflow.AggregatorBasedWorkflow.test_experimental_agg_based_workflow --custom_template tests/github/experimental/workflow/AggregatorBasedWorkflow/testcase_datastore_cli --fed_workspace aggregator --col col1 --col col2  --rounds-to-train 1
diff --git a/.github/workflows/task_runner_e2e.yml b/.github/workflows/task_runner_e2e.yml
@@ -163,3 +163,70 @@ jobs:
         with:
           name: tr_non_tls_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }}
           path: result.tar
+
+  test_with_no_client_auth:
+    name: tr_no_client_auth
+    runs-on: ubuntu-22.04
+    timeout-minutes: 120 # 2 hours
+    strategy:
+      matrix:
+        # Testing non TLS scenario only for torch_cnn_mnist model and python 3.10
+        # If required, this can be extended to other models and python versions
+        model_name: ["keras_cnn_mnist"]
+        python_version: ["3.10"]
+      fail-fast: false # do not immediately fail if one of the combinations fail
+
+    env:
+      MODEL_NAME: ${{ matrix.model_name }}
+      PYTHON_VERSION: ${{ matrix.python_version }}
+
+    steps:
+      - name: Checkout OpenFL repository
+        id: checkout_openfl
+        uses: actions/[email protected]
+        with:
+          fetch-depth: 2 # needed for detecting changes
+          submodules: "true"
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Python
+        id: setup_python
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Install dependencies
+        id: install_dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install .
+          pip install -r test-requirements.txt
+
+      - name: Run Task Runner E2E tests without TLS
+        id: run_tests
+        run: |
+          python -m pytest -s tests/end_to_end/test_suites/task_runner_tests.py \
+          -m ${{ env.MODEL_NAME }} --model_name ${{ env.MODEL_NAME }} \
+          --num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }} --disable_client_auth
+          echo "Task runner end to end test run completed"
+
+      - name: Print test summary
+        id: print_test_summary
+        if: ${{ always() }}
+        run: |
+          export PYTHONPATH="$PYTHONPATH:."
+          python tests/end_to_end/utils/summary_helper.py
+          echo "Test summary printed"
+
+      - name: Tar files
+        id: tar_files
+        if: ${{ always() }}
+        run: tar -cvf result.tar results
+
+      - name: Upload Artifacts
+        id: upload_artifacts
+        uses: actions/upload-artifact@v4
+        if: ${{ always() }}
+        with:
+          name: tr_no_client_auth_${{ env.MODEL_NAME }}_python${{ env.PYTHON_VERSION }}_${{ github.run_id }}
+          path: result.tar
diff --git a/.github/workflows/workflow_interface_101_mnist.yml b/.github/workflows/workflow_interface_101_mnist.yml
@@ -36,11 +36,11 @@ jobs:
 
     - name: Run Notebook
       run: |
-        jupyter nbconvert --execute --to notebook ./openfl-tutorials/experimental/101_MNIST.ipynb
+        jupyter nbconvert --execute --to notebook ./openfl-tutorials/experimental/workflow/101_MNIST.ipynb
         echo "Notebook run completed"
 
     - name: Tar files
-      run: tar -cvf notebook.tar ./openfl-tutorials/experimental/101_MNIST.nbconvert.ipynb
+      run: tar -cvf notebook.tar ./openfl-tutorials/experimental/workflow/101_MNIST.nbconvert.ipynb
 
     - name: Upload Artifacts
       uses: actions/upload-artifact@v4

diff --git a/README.md b/README.md
@@ -40,7 +40,7 @@ OpenFL supports two APIs to set up a Federated Learning experiment:
 Define an experiment and distribute it manually. All participants can verify model code and [FL plan](https://openfl.readthedocs.io/en/latest/about/features_index/taskrunner.html#federated-learning-plan-fl-plan-settings) prior to execution. The federation is terminated when the experiment is finished. This API is meant for enterprise-grade FL experiments, including support for mTLS-based communication channels and TEE-ready nodes (based on Intel® SGX).
 
 - [Workflow API](https://openfl.readthedocs.io/en/latest/about/features_index/workflowinterface.html) ([*experimental*](https://openfl.readthedocs.io/en/latest/developer_guide/experimental_features.html)):
-Create complex experiments that extend beyond traditional horizontal federated learning. This API enables an experiment to be simulated locally, then seamlessly scaled to a federated setting. See the [experimental tutorials](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/) to learn how to coordinate [aggregator validation after collaborator model training](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/102_Aggregator_Validation.ipynb), [perform global differentially private federated learning](https://github.com/psfoley/openfl/tree/experimental-workflow-interface/openfl-tutorials/experimental/Global_DP), measure the amount of private information embedded in a model after collaborator training with [privacy meter](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/Privacy_Meter/readme.md), or [add a watermark to a federated model](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/301_MNIST_Watermarking.ipynb).
+Create complex experiments that extend beyond traditional horizontal federated learning. This API enables an experiment to be simulated locally, then seamlessly scaled to a federated setting. See the [experimental tutorials](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/) to learn how to coordinate [aggregator validation after collaborator model training](https://github.com/securefederatedai/openfl/tree/develop/openfl-tutorials/experimental/workflow/102_Aggregator_Validation.ipynb), [perform global differentially private federated learning](https://github.com/psfoley/openfl/tree/experimental-workflow-interface/openfl-tutorials/experimental/workflow/Global_DP), measure the amount of private information embedded in a model after collaborator training with [privacy meter](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/Privacy_Meter/readme.md), or [add a watermark to a federated model](https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/301_MNIST_Watermarking.ipynb).
 
 The quickest way to test OpenFL is to follow the [online documentation](https://openfl.readthedocs.io/en/latest/index.html) to launch your first federation.<br/>
 Read the [blog post](https://medium.com/openfl/from-centralized-machine-learning-to-federated-learning-with-openfl-b3e61da52432) explaining steps to train a model with OpenFL. <br/>

diff --git a/docs/about/features_index/privacy_meter.rst b/docs/about/features_index/privacy_meter.rst
@@ -22,7 +22,7 @@ In this threat model, each party can audit the privacy loss of the local and glo
 
 Workflow
 -----------------------------------------------
-We provide a demo code in `cifar10_PM.py <https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/Privacy_Meter/cifar10_PM.py>`_. Here, we briefly describe its workflow.
+We provide a demo code in `cifar10_PM.py <https://github.com/securefederatedai/openfl/blob/develop/openfl-tutorials/experimental/workflow/Privacy_Meter/cifar10_PM.py>`_. Here, we briefly describe its workflow.
 In each round of FL, parties train, starting with the current global model as initialization, using their local dataset. Then, the current global model and updated local model will be passed to the privacy auditing module (See `audit` function in `cifar10_PM.py`) to produce a privacy loss report. The local model update will then be shared to the server and all such updates aggregated to form the next global model. Though this is a simulation so that no network sharing of models is involved, these reports could be used in a fully distributed setting to trigger actions when the loss is too high. These actions could include not sharing local updates to the aggregator, not 
 allowing the FL system to release the model to other outside entities, or potentially re-running local training in a differentially private mode and re-auditing in an attempt to reduce the leakage before sharing occurs.
 
@@ -36,4 +36,4 @@ To measure the success of the attack (privacy loss), we generate the ROC of the
 
 Examples
 -----------------------------------------------
-`Here <https://github.com/securefederatedai/openfl/tree/f1657abe88632d542504d6d71ca961de9333913f/openfl-tutorials/experimental/Privacy_Meter>`_, we give a few commands and the results for each of them. 
+`Here <https://github.com/securefederatedai/openfl/tree/f1657abe88632d542504d6d71ca961de9333913f/openfl-tutorials/experimental/workflow/Privacy_Meter>`_, we give a few commands and the results for each of them.