Strategy and Steps to Create a SBOM Pilot Demo for the "volttron-core" Repository #191

kefeimo · 2024-06-17T21:16:40Z

kefeimo
Jun 17, 2024
Collaborator

SBOM basics write up

Strategy and Steps to Create a SBOM Pilot Demo for the "volttron-core" Repository

Significance of Using SBOM

A Software Bill of Materials (SBOM) is a comprehensive inventory of all components, libraries, and modules that are included in a piece of software. SBOMs are significant because they:

Enhance Security: Identify and mitigate vulnerabilities by knowing exactly what components are in use.
Ensure Compliance: Maintain compliance with open source licenses and regulatory requirements.
Improve Maintenance: Facilitate easier updates and maintenance by understanding dependencies.
Support Incident Response: Quickly assess and respond to security incidents by knowing affected components.

Implementation Strategy

The strategy involves creating an SBOM for the "volttron-core" repository both locally and via GitHub Actions for continuous integration. The process includes:

Local Setup: Develop and test the SBOM generation process on a local machine using WSL Ubuntu 22.
GitHub Actions Integration: Automate the SBOM generation process using GitHub Actions to ensure continuous monitoring and generation on code changes.

Local Setup Steps

Install WSL and Ubuntu 22:
- Ensure WSL is installed and set up with Ubuntu 22.
- Update packages: sudo apt update && sudo apt upgrade

Install Node.js and npm:

curl -fsSL <https://deb.nodesource.com/setup_20.x> | sudo -E bash -
sudo apt-get install -y nodejs

Generated files: None

Install Python and pip:

sudo apt-get install -y python3 python3-pip

Generated files: None

Install cdxgen and depscan:

sudo npm install -g @cyclonedx/cdxgen
sudo pip install owasp-depscan

Generated files: None

Clone the volttron-core repository:

git clone <https://github.com/kefeimo/volttron-core.git>
cd volttron-core

Generated files: None

Generate SBOM and Vulnerability Data Report:
```
cdxgen -t python .
depscan --bom bom.json
mv ./bom.vdr.json ./reports/bom.vdr.json || echo "No VDR was generated."
```
- Generated files:
  - bom.json: SBOM file generated by cdxgen.
  - bom.vdr.json: Vulnerability Data Report (VDR) file generated by depscan.

GitHub Actions Integration

Create Workflow File: Add a new GitHub Actions workflow in .github/workflows/sbom.yml.
GitHub Actions Workflow Configuration:

# .github/workflows/sbom.yml
name: Generate SBOM and VDR

on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  generate-sbom:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install cdxgen
        run: npm install -g @cyclonedx/cdxgen

      - name: Install depscan
        run: pip install owasp-depscan

      - name: Generate SBOM
        run: cdxgen -t python .
        # Generated files: bom.json

      - name: Generate Vulnerability Data Report
        run: |
          depscan --bom bom.json
          TIMEOUT=10
          while [ $TIMEOUT -gt 0 ]; do
            if [ -f "./bom.vdr.json" ]; then
              mv ./bom.vdr.json ./reports/bom.vdr.json
              echo "VDR file generated and moved to reports directory."
              break
            fi
            echo "Waiting for bom.vdr.json..."
            sleep 1
            TIMEOUT=$((TIMEOUT-1))
          done
          if [ $TIMEOUT -eq 0 ]; then
            echo "Timeout reached. No VDR file generated."
          fi
        # Generated files: bom.vdr.json (if VDR is generated)

      - name: Upload SBOM Artifact
        uses: actions/upload-artifact@v4
        with:
          name: sbom
          path: ./bom.json

      - name: Upload VDR Artifact
        uses: actions/upload-artifact@v4
        with:
          name: vdr
          path: ./reports

Running the Demo Locally

Navigate to the Project Directory:
```
cd path/to/volttron-core
```

Run SBOM and VDR Generation Commands:

cdxgen -t python .  # generate bom.json
depscan --bom bom.json  # generate bom.vdr.json
mv ./bom.vdr.json ./reports/bom.vdr.json || echo "No VDR was generated."

Verify Outputs: Check for the bom.json and bom.vdr.json files in the project directory.

Example output log

$ depscan --bom bom.json  # generate bom.vdr.json

██████╗ ███████╗██████╗ ███████╗ ██████╗ █████╗ ███╗   ██╗
██╔══██╗██╔════╝██╔══██╗██╔════╝██╔════╝██╔══██╗████╗  ██║
██║  ██║█████╗  ██████╔╝███████╗██║     ███████║██╔██╗ ██║
██║  ██║██╔══╝  ██╔═══╝ ╚════██║██║     ██╔══██║██║╚██╗██║
██████╔╝███████╗██║     ███████║╚██████╗██║  ██║██║ ╚████║
╚═════╝ ╚══════╝╚═╝     ╚══════╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═══╝

INFO [2024-06-16 19:04:47,766] Performing regular scan for /home/kefei/project/volttron-core using plugin bom

                                                            Dependency Scan Results (BOM)                                                             
╔════════════════════════════════════════════════════════════════╤════════════════════════════════════╤════════════════════╤═══════════════╤═════════╗
║ Dependency Tree                                                │ Insights                           │ Fix Version        │ Severity      │   Score ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ CVE-2023-50782                           │ 🧾 Vendor Confirmed                │ 42.0.2             │ HIGH          │     7.5 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ CVE-2023-49083                           │ 🧾 Vendor Confirmed                │ 41.0.6             │ MEDIUM        │     5.9 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected]                                                   │ 📓 Indirect dependency             │                    │ HIGH          │     7.5 ║
║ └── [email protected] ⬅ CVE-2022-42969                                 │                                    │                    │               │         ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ CVE-2023-0286                            │ 🧾 Vendor Confirmed                │ 39.0.1             │ HIGH          │     7.4 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ GHSA-5cpq-8wj7-hf2v                      │                                    │ 41.0.0             │ LOW           │     2.0 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected]                                          │ 📓 Indirect dependency             │ 23.9.0             │ CRITICAL      │     9.8 ║
║ └── [email protected] ⬅ CVE-2023-41419                            │                                    │                    │               │         ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ GHSA-jm77-qphf-c4w8                      │                                    │ 41.0.3             │ LOW           │     2.0 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ CVE-2023-23931                           │                                    │ 39.0.1             │ MEDIUM        │     6.5 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ GHSA-v8gr-m533-ghj9                      │                                    │ 41.0.4             │ LOW           │     2.0 ║
╟────────────────────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┼───────────────┼─────────╢
║ [email protected] ⬅ CVE-2024-0727                            │ 🧾 Vendor Confirmed                │ 42.0.2             │ MEDIUM        │     5.5 ║
╚════════════════════════════════════════════════════════════════╧════════════════════════════════════╧════════════════════╧═══════════════╧═════════╝

Integration with GitHub Actions

Commit and Push Workflow:

git add .github/workflows/sbom.yml
git commit -m "Add SBOM generation workflow"
git push origin main

Verify GitHub Actions: Check the Actions tab in the GitHub repository to ensure the workflow runs on every push or pull request to the main branch.

Example Action Log (i.e., of Generate SBOM and VDR)

Run depscan --bom bom.json
  depscan --bom bom.json
  TIMEOUT=10
  while [ $TIMEOUT -gt 0 ]; do
    if [ -f "./bom.vdr.json" ]; then
      mv ./bom.vdr.json ./reports/bom.vdr.json
      echo "VDR file generated and moved to reports directory."
      break
    fi
    echo "Waiting for bom.vdr.json..."
    sleep 1
    TIMEOUT=$((TIMEOUT-1))
  done
  if [ $TIMEOUT -eq 0 ]; then
    echo "Timeout reached. No VDR file generated."
  fi
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.10.14/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib
╭─────────────────────── Donate to the OWASP Foundation ───────────────────────╮
│ OWASP foundation relies on donations to fund our projects.                   │
│ Please donate at:                                                            │
│ https://owasp.org/donate/?reponame=www-project-dep-scan&title=OWASP+depscan  │
╰──────────────────────────────────────────────────────────────────────────────╯
INFO [2024-06-17 00:02:07,315] About to download the vulnerability database from ghcr.io/appthreat/vdbgz:v5. This might take a while ...
INFO [2024-06-17 00:02:43,343] Performing regular scan for /home/runner/work/volttron-core/volttron-core using plugin bom

██████╗ ███████╗██████╗ ███████╗ ██████╗ █████╗ ███╗   ██╗
██╔══██╗██╔════╝██╔══██╗██╔════╝██╔════╝██╔══██╗████╗  ██║
██║  ██║█████╗  ██████╔╝███████╗██║     ███████║██╔██╗ ██║
██║  ██║██╔══╝  ██╔═══╝ ╚════██║██║     ██╔══██║██║╚██╗██║
██████╔╝███████╗██║     ███████║╚██████╗██║  ██║██║ ╚████║
╚═════╝ ╚══════╝╚═╝     ╚══════╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═══╝

                         Dependency Scan Results (BOM)                          
╔══════════════════════╤══════════════════════╤═════════════╤══════════╤═══════╗
║ Dependency Tree      │ Insights             │ Fix Version │ Severity │ Score ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 42.0.2      │ MEDIUM   │   5.5 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅ CVE-2024-0727  │ 🧾 Vendor Confirmed  │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 42.0.2      │ HIGH     │   7.5 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅ CVE-2023-50782 │ 🧾 Vendor Confirmed  │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 41.0.0      │ LOW      │   2.0 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅                │                      │             │          │       ║
║     GHSA-5cpq-8wj7-… │                      │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ [email protected]… │ 📓 Indirect          │ 23.9.0      │ CRITICAL │   9.8 ║
║ └── [email protected] ⬅ │ dependency           │             │          │       ║
║     CVE-2023-41419   │                      │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ [email protected].… │ 📓 Indirect          │ 65.5.1      │ MEDIUM   │   5.9 ║
║ └── [email protected]… │ dependency           │             │          │       ║
║     ⬅ CVE-2022-40897 │ 🧾 Vendor Confirmed  │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 41.0.4      │ LOW      │   2.0 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅                │                      │             │          │       ║
║     GHSA-v8gr-m533-… │                      │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 41.0.6      │ MEDIUM   │   5.9 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅ CVE-2023-49083 │ 🧾 Vendor Confirmed  │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 41.0.3      │ LOW      │   2.0 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅                │                      │             │          │       ║
║     GHSA-jm77-qphf-… │                      │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 39.0.1      │ HIGH     │   7.4 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅ CVE-2023-0286  │ 🧾 Vendor Confirmed  │             │          │       ║
╟──────────────────────┼──────────────────────┼─────────────┼──────────┼───────╢
║ volttron@latest      │ 📓 Indirect          │ 39.0.1      │ MEDIUM   │   6.5 ║
║ └── cryptography@36… │ dependency           │             │          │       ║
║     ⬅ CVE-2023-23931 │                      │             │          │       ║
╚══════════════════════╧══════════════════════╧═════════════╧══════════╧═══════╝
VDR file generated and moved to reports directory.

This approach ensures a robust, automated process for generating and maintaining SBOMs, enhancing security and compliance for the volttron-core repository.

kefeimo · 2024-06-17T21:16:52Z

kefeimo
Jun 17, 2024
Collaborator Author

Question:

How does the cdxgen -t python . command generate the BOM file? Specifically, does it retrieve dependency information from the virtual environment or the pyproject.toml file? Given that the cdxgen tool can obtain dependency information from various sources, what is the order of precedence for these sources?

Response:

The cdxgen -t python . command generates a Bill of Materials (BOM) file for a Python project by analyzing its dependencies. It can retrieve dependency information from several sources, following a specific precedence order:

Primary Sources for Dependencies:
- requirements.txt: Lists the exact versions of required packages and is given the highest precedence.
- setup.py: Contains dependency specifications, often used in older Python projects.
- Pipfile: Used by Pipenv for advanced dependency management.
- pyproject.toml: The modern standard for Python projects, often used with tools like Poetry.
- Virtual Environment: As a fallback, the tool can inspect the active virtual environment to determine installed packages and their versions.
Precedence Order:
- 1. requirements.txt: If present, this file is prioritized because it provides a clear list of dependencies.
- 2. setup.py: Used if requirements.txt is not found.
- 3. Pipfile: Checked next if neither requirements.txt nor setup.py are present.
- 4. pyproject.toml: Used if none of the previous files are found.
- 5. Virtual Environment: Inspected only if none of the specific dependency files are available, as this might include packages not explicitly required by the project.
Process:
- When the cdxgen command is run, it scans the project directory for the presence of these files.
- It parses the file with the highest precedence found, extracts the list of dependencies along with their versions, and compiles this information into a BOM file.
- The BOM file is typically in JSON or XML format and follows standards like CycloneDX to ensure compatibility with various tools and systems.

By following this precedence order, cdxgen ensures it accurately captures the project's dependencies, preferring explicit declarations in configuration files over inferred data from the virtual environment. This process results in a reliable BOM file that can be used for security analysis, compliance, and other purposes.

0 replies

kefeimo · 2024-06-19T22:13:49Z

kefeimo
Jun 19, 2024
Collaborator Author

How to Include BOM Files into a Wheel

1. Preparation: Collect the Files

First, collect the files of interest (i.e., bom.json, bom.vdr.json) into a target path.

mkdir -p ./bom_reports && cp bom.* ./bom_reports/ && cp ./reports/* ./bom_reports/

Note: The ./bom_reports directory should include:

bom.json (generated by cdxgen -t python .)
bom.vdr.json (generated by depscan --bom bom.json)
depscan-bom.json and depscan.html (both generated by depscan --bom bom.json under ./reports/)

2. Modify `pyproject.toml` and Build the Wheel

Add the bom_reports directory to the list of included files in the pyproject.toml file for Poetry.

[tool.poetry]
name = "your_project_name"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

# Add this section
include = [
    "bom_reports/**/*"
]

Run the following command to build the wheel:

poetry build -vvv

Example output:

Using virtualenv: /home/user/.cache/pypoetry/virtualenvs/your_project-xxxxx-py3.10
Building your_project (0.1.0)
  - Building sdist
    Ignoring: dist/your_project-0.1.0.tar.gz
    Ignoring: poetry.lock
  ...
  - Adding: /home/user/project/your_project/bom_reports/bom.json
  - Adding: /home/user/project/your_project/bom_reports/bom.vdr.json
  - Adding: /home/user/project/your_project/bom_reports/depscan-bom.json
  - Adding: /home/user/project/your_project/bom_reports/depscan.html
  - Adding: LICENSE.md
  - Adding: pyproject.toml
  - Adding: README.md
  - Built your_project-0.1.0.tar.gz
  - Building wheel

3. Update the GitHub Action Workflow

.github/workflows/run-sbom.yml

name: Generate SBOM and VDR

on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  generate-sbom:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v3

      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install cdxgen
        run: npm install -g @cyclonedx/cdxgen

      - name: Install depscan
        run: pip install owasp-depscan

      - name: Generate SBOM
        run: cdxgen -t python .
        # Generated files: bom.json

      - name: Generate Vulnerability Data Report
        run: |
          depscan --bom bom.json
          # Generated files: bom.vdr.json and depscan reports under ./reports/

      - name: Collect SBOM Artifacts
        run: |
          TIMEOUT=10
          while [ $TIMEOUT -gt 0 ]; do
            if [ -f "./bom.vdr.json" ]; then
              mkdir -p ./bom_reports 
              cp ./bom.* ./bom_reports/ 
              cp ./reports/* ./bom_reports/
              echo "VDR file generated and moved to bom_reports/ directory."
              break
            fi
            echo "Waiting for bom.vdr.json..."
            sleep 1
            TIMEOUT=$((TIMEOUT-1))
          done
          if [ $TIMEOUT -eq 0 ]; then
            echo "Timeout reached. No VDR file generated."
          fi

      - name: Install Poetry
        run: |
          curl -sSL https://install.python-poetry.org | python3 -
          export PATH="$HOME/.local/bin:$PATH"
      
      - name: Install Dependencies
        run: |
          poetry install --no-interaction --no-ansi

      - name: Build Wheel
        run: poetry build -vvv 

      - name: Upload SBOM and VDR Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: bom-artifacts
          path: ./bom_reports/
      
      - name: Upload dist Artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: ./dist/

Note

Currently, there are complexities involved in uploading and downloading artifacts (e.g., wheel files) between different workflows. To simplify, the "Generate SBOM and VDR" workflow combines SBOM generation and wheel build steps. Consider updating your GitHub Action workflows to improve reusability, especially as tools like actions/upload-artifact and actions/download-artifact continue to improve support for sharing artifacts across workflows.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strategy and Steps to Create a SBOM Pilot Demo for the "volttron-core" Repository #191

{{title}}

Replies: 2 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Strategy and Steps to Create a SBOM Pilot Demo for the "volttron-core" Repository #191

kefeimo Jun 17, 2024 Collaborator

SBOM basics write up