Merge pull request #57 from trailofbits/mschwager-rule-ci-testing

Add Semgrep and CodeQL sections on testing in CI

content/docs/static-analysis/codeql/10-advanced.md

@@ -488,6 +488,36 @@ to ensure that it is correct and if it is, rename `MemcpyCall.actual` to
 For more information about testing CodeQL queries, see the
 [official documentation](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/testing-custom-queries).
 
+## Testing custom queries in CI
+
+### GitHub Actions
+
+The following workflow can be used to test custom CodeQL queries in GitHub Actions:
+
+```yml
+name: Test CodeQL queries
+
+on: [push, pull_request]
+
+jobs:
+  codeql-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - id: init
+        uses: github/codeql-action/init@v3
+      - uses: actions/cache@v4
+        with:
+          path: ~/.codeql
+          key: ${{ runner.os }}-${{ runner.arch }}-${{ steps.init.outputs.codeql-version }}
+      - name: Run tests
+        run: |
+          ${{ steps.init.outputs.codeql-path }} test run ./path/to/query/tests/
+```
+
+This workflow also speeds up subsequent runs by caching query extraction and
+compilation, and pack dependency installation.
+
 ## Editor support for CodeQL
 
 The CodeQL CLI includes a server for the language-server protocol (LSP)

content/docs/static-analysis/semgrep/10-advanced.md

@@ -1053,6 +1053,35 @@ test file:
 4. **Evaluate the rule against real-world code**: Test the rule against actual code from your projects,
 open-source repositories, or other codebases to assess its effectiveness in real-life scenarios.
 
+## Testing custom rules in CI
+
+### GitHub Actions
+
+The following workflow can be used to test custom Semgrep rules in GitHub Actions:
+
+```yml
+name: Test Semgrep rules
+
+on: [push, pull_request]
+
+jobs:
+  semgrep-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+          cache: "pip"
+      - run: python -m pip install -r requirements.txt
+      - run: semgrep --test --test-ignore-todo ./path/to/rules/
+```
+
+Make sure to include `semgrep` in your `requirements.txt` (or [`poetry` or `pipenv` equivalents](https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages))
+file to speed up workflow runs by caching the dependency. Note, we include
+`--test-ignore-todo` here so we do not fail CI runs on [TODO tests](https://semgrep.dev/docs/writing-rules/testing-rules),
+which are a valuable form of documentation for future rule improvements.
+
 ## Autofix feature
 
 The autofix feature can automatically correct identified vulnerabilities, potential errors, or coding standard violations.