Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FIPS switch #19179

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
12da4e1
Add FIPS workflow file
dkirov-dd Dec 3, 2024
a815513
Add Windows steps
dkirov-dd Dec 3, 2024
f7d5b4e
Experiment with download from S3 for Windows
dkirov-dd Dec 3, 2024
a441988
Revert to building openssl
dkirov-dd Dec 3, 2024
fa42ba9
Switch Windows steps to download from S3
dkirov-dd Dec 5, 2024
7e0040d
Remove unnecessary steps
dkirov-dd Dec 5, 2024
6dc5ae4
Add FIPS_MODULE_VERSION for Linux
dkirov-dd Dec 5, 2024
ff1b42e
Finish handling Python in setup
dkirov-dd Dec 6, 2024
08942db
Remove unnecessary steps
dkirov-dd Dec 6, 2024
c8393cd
Add md5 tests
dkirov-dd Dec 6, 2024
4a74187
Make md5 tests pass
dkirov-dd Dec 6, 2024
c1ffd27
Try separating FIPS and non-FIPS md5 tests
dkirov-dd Dec 6, 2024
2a85b00
Add e2e tests for TLS FIPS
dkirov-dd Dec 9, 2024
fe18202
Make TLS E2E tests pass
dkirov-dd Dec 10, 2024
470bca7
Switch from env vars to C bindings
dkirov-dd Dec 13, 2024
e7d4f3c
Revert to using env vars
dkirov-dd Dec 16, 2024
413ca9d
Add option for e2e env vars in workflow
dkirov-dd Dec 17, 2024
c624089
Remove unnecessary comments from start-server.sh
dkirov-dd Dec 17, 2024
f98c4d1
Rework enable_fips for user env var overwrite
dkirov-dd Dec 18, 2024
6ceb4d2
Disable FIPS tests by default in master
dkirov-dd Dec 18, 2024
8c2ba4c
Add changelogs
dkirov-dd Dec 18, 2024
944dbd1
Fix license headers
dkirov-dd Dec 18, 2024
e07659e
Remove unfinished tests
dkirov-dd Dec 18, 2024
8837acd
Remove openssl.cnf workaround
dkirov-dd Dec 18, 2024
6ac9008
Remove unused compose file
dkirov-dd Dec 18, 2024
7c62aef
Fix license headers
dkirov-dd Dec 18, 2024
31a90a2
Bring back integration tests
dkirov-dd Dec 18, 2024
2d3b7d2
Experiment with integration tests
dkirov-dd Dec 19, 2024
c2976fa
Remove integration test files
dkirov-dd Dec 20, 2024
efb8109
Restore pr.yml and test-target.yml
dkirov-dd Dec 20, 2024
2ac8ada
Move FIPS workflow to test-fips.yml
dkirov-dd Dec 20, 2024
8cec105
Fix pytest "not fips" args
dkirov-dd Dec 20, 2024
e54f3fa
Update test-fips.yml
dkirov-dd Dec 20, 2024
2594fcd
Fix unvalid workflow
dkirov-dd Dec 20, 2024
db66922
Modify JOB_NAME env var
dkirov-dd Dec 20, 2024
77508be
Re-introduce experimental integration tests
dkirov-dd Dec 20, 2024
a2061aa
Merge e2e tests and clean test-fips workflow
dkirov-dd Dec 20, 2024
285fa3b
Merge integration tests and use monkeypatch in setup fixture
dkirov-dd Dec 20, 2024
7eae96f
Attemp to fix experimental workflow
dkirov-dd Dec 20, 2024
fda181f
Replace ddev with pytest in experimental workflow
dkirov-dd Dec 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 43 additions & 3 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,48 @@ concurrency:
cancel-in-progress: true

jobs:
test:
uses: ./.github/workflows/pr-test.yml
test-linux:
uses: ./.github/workflows/test-fips.yml
with:
repo: core
platform: linux
runner: '["ubuntu-22.04"]'
zip_url: 'https://agent-ints-python-build-sandbox.s3.eu-north-1.amazonaws.com/python-windows-combined-v3.12.6-openssl-3.0.15-openssl-3.0.9-amd64.zip'
secrets: inherit

test-windows:
uses: ./.github/workflows/test-fips.yml
with:
platform: windows
runner: '["windows-2022"]'
zip_url: 'https://agent-ints-python-build-sandbox.s3.eu-north-1.amazonaws.com/python-windows-combined-v3.12.6-openssl-3.0.15-openssl-3.0.9-amd64.zip'
secrets: inherit

test-tls-linux-inactive:
uses: ./.github/workflows/test-target.yml
with:
job-name: TLS
target: tls
platform: linux
runner: '["ubuntu-22.04"]'
repo: "core"
standard: true
pytest-args: "-s -k test_fips_inactive_e2e"
traces-artifact-name: traces-tls-linux-no-fips
test-results-name: test-tls-linux-no-fips
secrets: inherit

test-tls-linux-active:
uses: ./.github/workflows/test-target.yml
with:
job-name: TLS
target: tls
platform: linux
runner: '["ubuntu-22.04"]'
repo: "core"
standard: true
pytest-args: "-s -k test_fips_active_e2e"
traces-artifact-name: traces-tls-linux-fips
test-results-name: test-tls-linux-fips
agent-image: "datadog/agent-dev:main-fips"
e2e-env-vars: "{'GOFIPS':'1'}"
secrets: inherit
216 changes: 216 additions & 0 deletions .github/workflows/test-fips.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
name: Test FIPS

on:
workflow_call:
inputs:
platform:
required: true
type: string
runner:
required: true
type: string
zip_url:
required: true
type: string

defaults:
run:
shell: bash

jobs:
run:
name: FIPS test on "${{ inputs.platform }}"
runs-on: ${{ fromJson(inputs.runner) }}

env:
FORCE_COLOR: "1"
DEBIAN_FRONTEND: "noninteractive"
OPENSSL_FIPS: 1
PYTHON_VERSION: "3.12"
OPENSSL_VERSION: "3.0.15"
FIPS_MODULE_VERSION: "3.0.9"

steps:

- uses: actions/checkout@v4

- name: Install System Dependencies
if: runner.os == 'Linux'
run: |
sudo apt update
sudo apt install -y --no-install-recommends \
wget \
build-essential \
gcc \
make \
perl \
libc6-dev

- name: Build FIPS Module
if: runner.os == 'Linux'
run: |
wget https://www.openssl.org/source/openssl-${{ env.FIPS_MODULE_VERSION }}.tar.gz \
&& tar -xvzf openssl-${{ env.FIPS_MODULE_VERSION }}.tar.gz \
&& cd openssl-${{ env.FIPS_MODULE_VERSION }} \
&& ./Configure enable-fips \
&& make \
&& sudo make install

- name: Build OpenSSL
if: runner.os == 'Linux'
run: |
wget https://www.openssl.org/source/openssl-${{ env.OPENSSL_VERSION }}.tar.gz \
&& tar -xvzf openssl-${{ env.OPENSSL_VERSION }}.tar.gz \
&& cd openssl-${{ env.OPENSSL_VERSION }} \
&& ./Configure enable-fips \
&& make \
&& sudo make install

- name: Build Python from Source with Custom OpenSSL
if: runner.os == 'Linux'
run: |

# Install dependencies for building Python
sudo apt-get update && sudo apt-get install -y \
build-essential \
zlib1g-dev \
libffi-dev \
libssl-dev \
libncurses5-dev \
libsqlite3-dev \
libreadline-dev \
libbz2-dev \
liblzma-dev \
tk-dev \
uuid-dev \
libgdbm-dev \
wget

# Download and extract Python source
wget https://www.python.org/ftp/python/${{ env.PYTHON_VERSION }}/Python-${{ env.PYTHON_VERSION }}.tgz
tar -xvzf Python-${{ env.PYTHON_VERSION }}.tgz -C python_dir
cd python_dir

# Configure and build Python with custom OpenSSL
./configure --enable-optimizations --with-openssl=$(pwd)/../openssl-${{ env.OPENSSL_VERSION }}
make -j$(nproc)
sudo make altinstall

- name: Download python-windows-combined
if: runner.os == 'Windows'
shell: powershell
run: |
Invoke-WebRequest -Uri '${{ inputs.zip_url }}' -OutFile 'python_combined.zip'

- name: Unzip python_combined.zip
if: runner.os == 'Windows'
shell: powershell
run: |
Expand-Archive -Path python_combined.zip -DestinationPath .\python_dir

- name: Run fipsintall.exe
if: runner.os == 'Windows'
working-directory: .\python_dir
shell: powershell
run: |
.\openssl.exe fipsinstall -module .\ossl-modules\fips.dll -out fipsmodule.cnf

- name: Configure OpenSSL for FIPS
if: runner.os == 'Windows'
working-directory: .\python_dir
shell: powershell
run: |
# Create openssl.cnf to enable FIPS mode
$OpenSSLConf = @"
config_diagnostics = 1
openssl_conf = openssl_init

.include fipsmodule.cnf

[openssl_init]
providers = provider_sect
alg_section = algorithm_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1

[algorithm_sect]
default_properties = fips=yes
"@
$OpenSSLConf | Set-Content -Path ".\openssl.cnf"

- name: Verify OpenSSL
if: runner.os == 'Windows'
working-directory: .\python_dir
shell: powershell
run: |
.\openssl.exe version -a
.\openssl.exe list -providers

- name: Verify OpenSSL with FIPS ENV vars
if: runner.os == 'Windows'
working-directory: .\python_dir
shell: powershell
run: |
$env:OPENSSL_MODULES = ".\ossl-modules"
$env:OPENSSL_CONF = ".\openssl.cnf"
.\openssl.exe list -providers

- name: Add Python to PATH Windows
if: runner.os == 'Windows'
shell: powershell
run: |
Add-Content -Path $env:GITHUB_ENV -Value "PATH=$(pwd)\python_dir;$(pwd)\python_dir\Scripts;$env:PATH"

- name: Add Python to PATH Linux
if: runner.os == 'Linux'
run: |
echo "PATH=$(pwd)/python_dir:$PATH" >> $GITHUB_ENV

- name: Install pip
run: |
python -m ensurepip

- name: Restore cache
uses: actions/cache/restore@v4
with:
path: ${{ runner.os == 'Windows' && '~\AppData\Local\pip\Cache' || '~/.cache/pip' }}
key: >-
${{ format(
'v01-python-{0}-{1}-{2}-{3}',
env.pythonLocation,
hashFiles('datadog_checks_base/pyproject.toml'),
hashFiles('datadog_checks_dev/pyproject.toml'),
hashFiles('ddev/pyproject.toml')
)}}
restore-keys: |-
v01-python-${{ env.pythonLocation }}

- name: Install ddev from local folder
run: |
python -m pip install -e ./datadog_checks_dev[cli]
python -m pip install -e ./ddev

- name: Configure ddev
run: |
ddev config set repos.core .
ddev config set repo core

- name: Test
if: runner.os == 'Windows'
working-directory: .\python_dir
shell: powershell
run: |
$env:PATH_TO_OPENSSL_CONF = "$(pwd)\openssl.cnf"
$env:PATH_TO_OPENSSL_MODULES = "$(pwd)\ossl-modules"
$env:OPENSSL_CONF = "$(pwd)\openssl.cnf"
$env:OPENSSL_MODULES = "$(pwd)\ossl-modules"
.\openssl.exe list -providers
.\openssl.exe md5
ddev test datadog_checks_base -- -k before_fips
ddev test datadog_checks_base -- -k after_fips
python -c "import ssl; ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT).set_ciphers('MD5')"
Loading
Loading