Overhaul CI documentation

docs/continuous_integration.rst

@@ -0,0 +1,60 @@
+.. _ci_tests:
+
+Continuous Integration
+===============================
+
+The SecureDrop project uses `GitHub Actions <https://docs.github.com/en/actions>`_ for
+running automated continuous integration on code changes. You can get an overview of what
+each project does by reviewing the ``Makefile`` and files in the ``.github/workflows`` folder of the project's repository.
+
+Basics
+------
+
+Our CI runs a mixture of linters, tests and build processes to validate code submissions.
+
+Most tasks have a corresponding ``make`` target that will run the same command locally. Common
+targets across all our projects include:
+
+* ``make lint``: run linters
+* ``make fix``: apply automated fixes from formatters and linters
+* ``make test``: run automated tests
+
+In CI, these are run in a container using the corresponding Linux distribution (e.g. Debian or Fedora),
+which can also be used to reproduce CI results locally. Some projects, like ``securedrop`` (server) and
+``securedrop-workstation``, automatically run commands in containers.
+
+Pull requests
+-------------
+
+Most CI jobs are `triggered <https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows>`_ by both ``push`` and ``pull_request`` events. The former is run against
+your branch, while the latter is run against your branch merged into ``main`` (or ``develop``).
+
+
+Special branch prefixes
+-----------------------
+
+In the ``securedrop`` repository, some slower jobs are only triggered if a specific branch prefix
+is used when creating the pull request. Currently these are:
+
+* ``stg-*``: runs a staging build in Google Cloud, see :ref:`Configuration Tests<config_tests>`
+* ``l10n-*``: runs localization tests across all 20+ supported languages
+
+Nightlies
+---------
+
+For ``securedrop-workstation`` and ``securedrop-client``, packages are built for every merged
+commit as well as every night. These packages are published to the test yum and apt repositories.
+
+A "nightlies" workflow runs in each repository that builds the respective packages. The workflow
+uses an authenticated token for the ``sdcibot`` GitHub account to push the packages and build metadata
+to ``build-logs``, ``securedrop-apt-test`` and ``securedrop-yum-test``.
+
+The ``securedrop-apt-test`` and ``securedrop-yum-test`` repositories have a workflow that automatically
+prunes older packages, also using a token for ``sdcibot`` to push to themselves.
+
+Workstation CI
+--------------
+
+For testing ``securedrop-workstation``, we run a special CI job that virtualizes Qubes OS inside
+of VMWare. Documentation for this is available in the `securedrop-workstation-ci <https://github.com/freedomofpress/securedrop-workstation-ci>`_
+repository.

docs/index.rst

@@ -18,6 +18,7 @@ administrators <https://docs.securedrop.org/>`_.
    contributor_guidelines
    tips_and_tricks
    supported_languages
+   continuous_integration
    dependency_updates
 
 .. toctree::
@@ -38,7 +39,6 @@ administrators <https://docs.securedrop.org/>`_.
    testing_securedrop
    testing_application_tests
    testing_configuration_tests
-   testing_continuous_integration
    demo
    apt_repo
    updating_ossec

docs/testing_application_tests.rst

@@ -3,66 +3,30 @@
 Testing: Application Tests
 ==========================
 
-The application test suite uses:
-
-  * Pytest_
-  * Selenium_
+The application test suite uses pytest_, selenium_, and other Python tools
+to comprehensively test the SecureDrop server.
 
 The application tests consist of unit tests for the Python application code
 and functional tests that verify the functionality of the application code
 from the perspective of the user through a web browser.
 
-The functional tests use an outdated version of Firefox chosen specifically
-for compatibility with Selenium 2, and a rough approximation of the most
-recent Tor Browser.
-
-.. note:: We're working on running the Selenium tests in Tor Browser.
-          See `GitHub #1629`_ for more info.
-
-.. _`GitHub #1629`: https://github.com/freedomofpress/securedrop/pull/1629
-
-.. _Pytest: https://docs.pytest.org/en/latest/
-.. _Selenium: https://www.selenium.dev/documentation/
-
-Installation
-------------
-
-The application tests are installed automatically in the development
-and app-staging VMs, based on the contents of
-``securedrop/requirements/test-requirements.txt``.
-If you wish to change the dependencies, see :ref:`updating_pip_dependencies`.
+.. _pytest: https://docs.pytest.org/en/latest/
+.. _selenium: https://www.selenium.dev/documentation/
 
 Running the Application Tests
 -----------------------------
 
-The tests can be run inside the development VM:
+The tests are written to be run inside the development container:
 
 .. code:: sh
 
     make test
 
-Or the app-staging VM:
-
-.. code:: sh
-
-    vagrant ssh app-staging
-    sudo bash
-    cd /var/www/securedrop
-    pytest -v tests
-    chown -R www-data /var/lib/securedrop /var/www/securedrop
-
-.. warning:: The ``chown`` is necessary because running the tests as
-             root will change ownership of some files, creating
-             problems with the source and journalist interfaces.
-
-For explanation of the difference between these machines, see
-:doc:`virtual_environments`.
-
 If you just want to run the functional tests, you can use:
 
 .. code:: sh
 
-    securedrop/bin/dev-shell bin/run-test -v tests/functional
+    make test-functional
 
 Similarly, if you want to run a single test, you can specify it through the
 file, class, and test name:
@@ -72,13 +36,6 @@ file, class, and test name:
     securedrop/bin/dev-shell bin/run-test \
         tests/test_journalist.py::TestJournalistApp::test_invalid_credentials
 
-The `gnupg
-<https://pythonhosted.org/python-gnupg>`_ library can be quite verbose in its
-output. The default log level applied to this package is ``ERROR`` but this can
-be controlled via the ``GNUPG_LOG_LEVEL`` environment variable. It can have values
-such as ``INFO`` or ``DEBUG`` if some particular test case or test run needs
-greater verbosity.
-
 Page Layout Tests
 ~~~~~~~~~~~~~~~~~
 
@@ -87,12 +44,11 @@ language using the page layout tests. These will generate screenshots of
 each page and can be used for example to update the SecureDrop user guides
 when modifications are made to the UI.
 
-You can run all tests, including the page layout tests with the `--page-layout`
-option:
+To run just these tests:
 
 .. code:: sh
 
-    securedrop/bin/dev-shell bin/run-test --page-layout tests
+    make test-pageslayout
 
 
 Updating the Application Tests

docs/testing_configuration_tests.rst

@@ -3,9 +3,9 @@
 Testing: Configuration Tests
 ============================
 
-Testinfra_ tests verify the end state of the staging VMs. Any
-changes to the Ansible configuration should have a corresponding
-spectest.
+Testinfra_ tests verify the end state of a full SecureDrop server, whether on
+physical hardware or in staging VMs. Any changes to the Ansible configuration
+should have a corresponding test.
 
 .. _Testinfra: https://testinfra.readthedocs.io/en/latest/
 
@@ -91,13 +91,35 @@ mirroring the Ansible configuration. Prior to the reorganization of
 the Ansible layout, the tests are rather tightly coupled to hosts. The
 layout of config tests is therefore subject to change.
 
-Config Testing Strategy
------------------------
+Running the CI Staging Environment
+----------------------------------
 
-The config tests currently emphasize testing implementation rather than
-functionality. This was a temporary measure to increase the testing
-baseline for validating the Ansible provisioning flow, which aided in migrating
-to a current version of Ansible (v2+). Now that the Ansible version is current,
-the config tests can be improved to validate behavior, such as confirming
-ports are blocked via external network calls, rather than simply checking
-that the iptables rules are formatted as expected.
+The staging environment can also run via CI in Google Cloud (GCE). These tests are
+run every night or if a member of the ``freedomofpress`` Github Organization
+pushes to a branch that starts with ``stg-``. Please ask in your PR if you'd like
+someone to run the tests for you.
+
+The tests can also be run manually with a Google Cloud Platform account and Docker
+installed locally:
+
+Source the setup script using the following command:
+
+.. code:: sh
+
+    source ./devops/gce-nested/ci-env.sh
+
+You will be prompted for the values of the required environment variables. There
+are some defaults set that you may want to change. You will need to export
+``GOOGLE_CREDENTIALS`` with `authentication details <https://cloud.google.com/docs/authentication/use-cases>`_
+for your GCP account, which is outside the scope of this guide. Some parameters
+are specific to FPF's GCE setup and may need adjusting if you are running elsewhere.
+
+Then to run the tests locally:
+
+.. code:: sh
+
+    make ci-go
+
+You can use ``./devops/gce-nested/ci-runner.sh`` to provision the remote hosts
+while making changes, including rebuilding the Debian packages used in the
+Staging environment. See :doc:`virtual_environments` for more information.