Merge branch 'securefederatedai:develop' into serialization_removal

.github/workflows/scans.yml

@@ -0,0 +1,30 @@
+# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Security Scans
+
+on:
+  pull_request:
+    branches: [ develop ]
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v3
+      with:
+        python-version: "3.8"
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install .
+    - name: Hadolint Dockerfile Scan
+      run: |
+        docker run -v ${PWD}/openfl-docker:/openfl-docker --rm -i hadolint/hadolint hadolint -t error /openfl-docker/Dockerfile.base

.github/workflows/trivy.yml

@@ -0,0 +1,29 @@
+name: build
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+jobs:
+  build:
+    name: Build
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Build an image from Dockerfile
+        run: |
+          docker build -t docker.io/securefederatedai/openfl:${{ github.sha }} -f openfl-docker/Dockerfile.base .
+
+      - name: Run Trivy vulnerability scanner
+        uses: aquasecurity/trivy-action@master
+        with:
+          image-ref: 'docker.io/securefederatedai/openfl:${{ github.sha }}'
+          format: 'sarif'
+          output: 'trivy-results.sarif'
+
+      - name: Upload Trivy scan results to GitHub Security tab
+        uses: github/codeql-action/upload-sarif@v2
+        with:
+          sarif_file: 'trivy-results.sarif'

README.md

@@ -12,8 +12,12 @@
 [<img src="https://img.shields.io/badge/[email protected]?logo=slack">](https://join.slack.com/t/openfl/shared_invite/zt-ovzbohvn-T5fApk05~YS_iZhjJ5yaTw) 
 [![License](https://img.shields.io/badge/License-Apache%202.0-brightgreen.svg)](https://opensource.org/licenses/Apache-2.0)
 [![Citation](https://img.shields.io/badge/cite-citation-brightgreen)](https://arxiv.org/abs/2105.06413)
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/interactive_api/numpy_linear_regression/workspace/SingleNotebook.ipynb)
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/6599/badge)](https://bestpractices.coreinfrastructure.org/projects/6599)
+<a href="https://scan.coverity.com/projects/securefederatedai-openfl">
+  <img alt="Coverity Scan Build Status"
+       src="https://scan.coverity.com/projects/29040/badge.svg"/>
+</a>
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/intel/openfl/blob/develop/openfl-tutorials/interactive_api/numpy_linear_regression/workspace/SingleNotebook.ipynb)
 
 OpenFL is a Python 3 framework for Federated Learning. OpenFL is designed to be a _flexible_, _extensible_ and _easily learnable_ tool for data scientists. OpenFL is hosted by The Linux Foundation, aims to be community-driven, and welcomes contributions back to the project. 
 

ROADMAP.md

@@ -36,13 +36,13 @@ In the process of thinking about federated workflows, and the properties that ar
 7. Don't reinvent unless absolutely necessary
 
 ### 1.2 Security, Privacy, and Governance
-OpenFL is designed for security and privacy, and later this year we will be releasing some exciting extensions that build on running [OpenFL experiments within SGX enclaves](https://github.com/intel/openfl/blob/develop/openfl-gramine/MANUAL.md).   
+OpenFL is designed for security and privacy, and soon we will be releasing some exciting extensions that build on running [OpenFL experiments within SGX enclaves](https://github.com/intel/openfl/blob/develop/openfl-gramine/MANUAL.md).   
 
-### 1.4 Decoupling interface from infrastructure
+### 1.3 Decoupling interface from infrastructure
 The task runner interface is coupled with the the single experiment aggregator / collaborator infrastructure, and the interactive API is tied to the director / envoy infrastructure. 
 The interactive API was originally designed to be a high-level API for OpenFL, but for the cases when more control is required by users, access to lower level interfaces is necessary.
 
-### 1.3 Consolidating interfaces
+### 1.4 Consolidating interfaces
 Today we support three interfaces: TaskRunner, native Python API, and interactive API. These are all distinct APIs, and are not particularly interoperable.
 By the time we reach OpenFL 2.0, our intention is to deprecate the original native [Python API](https://openfl.readthedocs.io/en/latest/source/workflow/running_the_federation.notebook.html) used for simulations, 
 bring consistency to the remaining interfaces with a high level, middle level, and low level API that are **fully interoperable**. This will result in being able to use the interface you're most comfortable with for a simulation,
@@ -58,20 +58,18 @@ This causes community fragmentation and distracts from some of the bigger proble
 
 ## Upcoming OpenFL releases
 
-### OpenFL 1.6 (Q2 2023)
+### OpenFL 1.6 (Q4 2023)
 1. Use the OpenFL Workflow Interface on distributed infrastructure with the [FederatedRuntime](https://openfl.readthedocs.io/en/latest/workflow_interface.html#runtimes-future-plans)
-2. New use cases enabled by custom workflows
+2. LLM Support
+3. New use cases enabled by custom workflows
     * Standard ML Models (i.e. Tree-based algorithms)
-3. Federated evaluation documentation and examples
-4. Well defined aggregator / collaborator interfaces intended for building higher level projects on top of OpenFL
-5. Significantly improved documentation
-6. New OpenFL Security Repo that extends OpenFL to provide governance, and end-to-end security for federated learning experiments
+4. Federated evaluation documentation and examples
+6. Significantly improved documentation
 
-### OpenFL 2.0 (2023)
+### OpenFL 2.0 (2024)
 1. Interface Cohesion
     * High level interface: Interactive API
-    * Mid level interface: Workflow API
-    * Low level interface: Redesigned TaskRunner API
+    * Low level interface: Workflow API
 2. Decoupling interfaces from infrastructure
-3. Updates to OpenFL Security 
+3. Well defined interfaces intended for building higher level projects on top of OpenFL
 

openfl/experimental/utilities/metaflow_utils.py

@@ -25,7 +25,8 @@
 from metaflow.task import MetaDatum
 import fcntl
 import hashlib
-from dill.source import getsource
+from dill.source import getsource  # nosec
+# getsource only used to determine structure of FlowGraph
 from typing import TYPE_CHECKING
 if TYPE_CHECKING:
     from openfl.experimental.interface import FLSpec
@@ -62,7 +63,9 @@ def __init__(self, name):
         self.name = name
 
     def __enter__(self):
-        lock_id = hashlib.new('md5', self.name.encode("utf8"), usedforsecurity=False).hexdigest()
+        lock_id = hashlib.new('md5', self.name.encode("utf8"),
+                              usedforsecurity=False).hexdigest()  # nosec
+        # MD5sum used for concurrency purposes, not security
         self.fp = open(f"/tmp/.lock-{lock_id}.lck", "wb")
         fcntl.flock(self.fp.fileno(), fcntl.LOCK_EX)
 

openfl/interface/tutorial.py

@@ -30,7 +30,7 @@ def start(ip, port):
     """Start the Jupyter Lab from the tutorials directory."""
     from os import environ
     from os import sep
-    from subprocess import check_call
+    from subprocess import check_call  # nosec
     from sys import executable
 
     from openfl.interface.cli_helper import TUTORIALS

openfl/interface/workspace.py

@@ -3,7 +3,7 @@
 """Workspace module."""
 
 import os
-import subprocess
+import subprocess  # nosec
 import sys
 from pathlib import Path
 from typing import Tuple, Union
@@ -86,7 +86,7 @@ def create_(prefix, template):
 def create(prefix, template):
     """Create federated learning workspace."""
     from os.path import isfile
-    from subprocess import check_call
+    from subprocess import check_call  # nosec
     from sys import executable
 
     from openfl.interface.cli_helper import print_tree
@@ -196,7 +196,7 @@ def import_(archive):
     from os.path import basename
     from os.path import isfile
     from shutil import unpack_archive
-    from subprocess import check_call
+    from subprocess import check_call  # nosec
     from sys import executable
 
     archive = Path(archive).absolute()

openfl/plugins/interface_serializer/dill_serializer.py

@@ -2,7 +2,7 @@
 # SPDX-License-Identifier: Apache-2.0
 """Dill serializer plugin."""
 
-import dill
+import dill  # nosec
 
 from .serializer_interface import Serializer
 
@@ -24,4 +24,4 @@ def serialize(object_, filename):
     def restore_object(filename):
         """Load and deserialize an object."""
         with open(filename, 'rb') as f:
-            return dill.load(f)
+            return dill.load(f)  # nosec

openfl/transport/grpc/aggregator_server.py

@@ -83,7 +83,7 @@ def validate_collaborator(self, request, context):
             if not self.aggregator.valid_collaborator_cn_and_id(
                     common_name, collaborator_common_name):
                 # Random delay in authentication failures
-                sleep(5 * random())
+                sleep(5 * random())  # nosec
                 context.abort(
                     StatusCode.UNAUTHENTICATED,
                     f'Invalid collaborator. CN: |{common_name}| '

openfl/utilities/ca/ca.py

@@ -9,11 +9,11 @@
 import sys
 import shutil
 import signal
-import subprocess
+import subprocess  # nosec
 import time
 from logging import getLogger
 from pathlib import Path
-from subprocess import check_call
+from subprocess import check_call  # nosec
 
 from click import confirm
 

openfl/utilities/ca/downloader.py

@@ -67,5 +67,6 @@ def _download(url, prefix, confirmation):
     if confirmation:
         confirm('CA binaries will be downloaded now', default=True, abort=True)
     name = url.split('/')[-1]
-    urllib.request.urlretrieve(url, f'{prefix}/{name}')
+    # nosec: private function definition with static urls
+    urllib.request.urlretrieve(url, f'{prefix}/{name}')  # nosec
     shutil.unpack_archive(f'{prefix}/{name}', f'{prefix}/step')

openfl/utilities/workspace.py

@@ -10,7 +10,7 @@
 import time
 from contextlib import contextmanager
 from pathlib import Path
-from subprocess import check_call
+from subprocess import check_call  # nosec
 from sys import executable
 from typing import Optional
 from typing import Tuple