From 18cc68c314fffdb0ed0ca21a96c061d575410c4c Mon Sep 17 00:00:00 2001 From: Kunal Mehta Date: Thu, 22 Aug 2024 18:51:32 -0400 Subject: [PATCH] Overhaul CI documentation Broadly update the docs for: * switch to GitHub Actions * special branch prefixes in server repository * workstation and client nightlies * pointer to SDW CI * remove outdated information regarding server tests * simplify staging docs, not super practical to run outside our CI Refs #181. --- docs/continuous_integration.rst | 60 ++++++++ docs/index.rst | 2 +- docs/testing_application_tests.rst | 60 ++------ docs/testing_configuration_tests.rst | 46 ++++-- docs/testing_continuous_integration.rst | 177 ------------------------ 5 files changed, 103 insertions(+), 242 deletions(-) create mode 100644 docs/continuous_integration.rst delete mode 100644 docs/testing_continuous_integration.rst diff --git a/docs/continuous_integration.rst b/docs/continuous_integration.rst new file mode 100644 index 00000000..ef34c728 --- /dev/null +++ b/docs/continuous_integration.rst @@ -0,0 +1,60 @@ +.. _ci_tests: + +Continuous Integration +=============================== + +The SecureDrop project uses `GitHub Actions `_ for +running automated continuous integration on code changes. You can get an overview of what +each project does by reviewing the ``Makefile`` and files in the ``.github/workflows`` folder. + +Basics +------ + +Our CI runs a mixture of linters, tests and build processes to validate code submissions. + +Most tasks have a corresponding ``make`` target that will run the same command locally. Common +targets across all our projects include: + +* ``make lint``: run linters +* ``make fix``: apply automated fixes from formatters and linters +* ``make test``: run automated tests + +In CI, these are run in a container using the corresponding Linux distribution (e.g. Debian or Fedora), +which can also be used to reproduce CI results locally. Some projects, like ``securedrop`` (server) and +``securedrop-workstation``, automatically run commands in containers. + +Pull requests +------------- + +Most CI jobs are triggered by both ``push`` and ``pull_request`` events. The former is run against +your branch, while the latter is run against your branch merged into ``main`` (or ``develop``). + + +Special branch prefixes +----------------------- + +In the ``securedrop`` repository, some slower jobs are only triggered if a specific branch prefix +is used when creating the pull request. Currently these are: + +* ``stg-*``: runs a staging build in GCE, see :ref:`Configuration Tests` +* ``l10n-*``: runs localization tests across all 20+ supported languages + +Nightlies +--------- + +For ``securedrop-workstation`` and ``securedrop-client``, we build packages on every merged +commit and every night and publish them to our test yum and apt repositories respectively. + +A "nightlies" workflow runs in each repository that builds the respective packages. The workflow +uses an authenticated token for the ``sdcibot`` GitHub account to push the packages and build metadata +to ``build-logs``, ``securedrop-apt-test`` and ``securedrop-yum-test``. + +The ``securedrop-apt-test`` and ``securedrop-yum-test`` repositories have a workflow that automatically +prunes older packages, also using a token for ``sdcibot`` to push to themselves. + +Workstation CI +-------------- + +For testing ``securedrop-workstation``, we run a special CI job that virtualizes Qubes OS inside +of VMWare. Documentation for this is available in the `securedrop-workstation-ci `_ +repository. diff --git a/docs/index.rst b/docs/index.rst index 709eebc4..0b422abc 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,6 +18,7 @@ administrators `_. contributor_guidelines tips_and_tricks supported_languages + continuous_integration dependency_updates .. toctree:: @@ -38,7 +39,6 @@ administrators `_. testing_securedrop testing_application_tests testing_configuration_tests - testing_continuous_integration apt_repo updating_ossec apparmor_profiles diff --git a/docs/testing_application_tests.rst b/docs/testing_application_tests.rst index 0aa17641..6ecfcff4 100644 --- a/docs/testing_application_tests.rst +++ b/docs/testing_application_tests.rst @@ -3,66 +3,30 @@ Testing: Application Tests ========================== -The application test suite uses: - - * Pytest_ - * Selenium_ +The application test suite uses pytest_, selenium_, and other Python tools +to comprehensively test the SecureDrop server. The application tests consist of unit tests for the Python application code and functional tests that verify the functionality of the application code from the perspective of the user through a web browser. -The functional tests use an outdated version of Firefox chosen specifically -for compatibility with Selenium 2, and a rough approximation of the most -recent Tor Browser. - -.. note:: We're working on running the Selenium tests in Tor Browser. - See `GitHub #1629`_ for more info. - -.. _`GitHub #1629`: https://github.com/freedomofpress/securedrop/pull/1629 - -.. _Pytest: https://docs.pytest.org/en/latest/ -.. _Selenium: https://www.selenium.dev/documentation/ - -Installation ------------- - -The application tests are installed automatically in the development -and app-staging VMs, based on the contents of -``securedrop/requirements/test-requirements.txt``. -If you wish to change the dependencies, see :ref:`updating_pip_dependencies`. +.. _pytest: https://docs.pytest.org/en/latest/ +.. _selenium: https://www.selenium.dev/documentation/ Running the Application Tests ----------------------------- -The tests can be run inside the development VM: +The tests are written to be run inside the development container: .. code:: sh make test -Or the app-staging VM: - -.. code:: sh - - vagrant ssh app-staging - sudo bash - cd /var/www/securedrop - pytest -v tests - chown -R www-data /var/lib/securedrop /var/www/securedrop - -.. warning:: The ``chown`` is necessary because running the tests as - root will change ownership of some files, creating - problems with the source and journalist interfaces. - -For explanation of the difference between these machines, see -:doc:`virtual_environments`. - If you just want to run the functional tests, you can use: .. code:: sh - securedrop/bin/dev-shell bin/run-test -v tests/functional + make test-functional Similarly, if you want to run a single test, you can specify it through the file, class, and test name: @@ -72,13 +36,6 @@ file, class, and test name: securedrop/bin/dev-shell bin/run-test \ tests/test_journalist.py::TestJournalistApp::test_invalid_credentials -The `gnupg -`_ library can be quite verbose in its -output. The default log level applied to this package is ``ERROR`` but this can -be controlled via the ``GNUPG_LOG_LEVEL`` environment variable. It can have values -such as ``INFO`` or ``DEBUG`` if some particular test case or test run needs -greater verbosity. - Page Layout Tests ~~~~~~~~~~~~~~~~~ @@ -87,12 +44,11 @@ language using the page layout tests. These will generate screenshots of each page and can be used for example to update the SecureDrop user guides when modifications are made to the UI. -You can run all tests, including the page layout tests with the `--page-layout` -option: +To run just these tests: .. code:: sh - securedrop/bin/dev-shell bin/run-test --page-layout tests + make test-pageslayout Updating the Application Tests diff --git a/docs/testing_configuration_tests.rst b/docs/testing_configuration_tests.rst index 223cc40a..9bbf6509 100644 --- a/docs/testing_configuration_tests.rst +++ b/docs/testing_configuration_tests.rst @@ -3,9 +3,9 @@ Testing: Configuration Tests ============================ -Testinfra_ tests verify the end state of the staging VMs. Any -changes to the Ansible configuration should have a corresponding -spectest. +Testinfra_ tests verify the end state of a full SecureDrop server, whether on +physical hardware or in staging VMs. Any changes to the Ansible configuration +should have a corresponding test. .. _Testinfra: https://testinfra.readthedocs.io/en/latest/ @@ -91,13 +91,35 @@ mirroring the Ansible configuration. Prior to the reorganization of the Ansible layout, the tests are rather tightly coupled to hosts. The layout of config tests is therefore subject to change. -Config Testing Strategy ------------------------ +Running the CI Staging Environment +---------------------------------- -The config tests currently emphasize testing implementation rather than -functionality. This was a temporary measure to increase the testing -baseline for validating the Ansible provisioning flow, which aided in migrating -to a current version of Ansible (v2+). Now that the Ansible version is current, -the config tests can be improved to validate behavior, such as confirming -ports are blocked via external network calls, rather than simply checking -that the iptables rules are formatted as expected. +The staging environment can also run via CI, running in GCE. These tests are +run every night or if you push to a branch that starts with ``stg-``. Currently +this can only be done by members of the ``freedomofpress`` GitHub organization, +please ask if you'd like someone to run the tests for you. + +You can also run them yourself if you have a Google Cloud Platform account and Docker +installed locally. + +Source the setup script using the following command: + +.. code:: sh + + source ./devops/gce-nested/ci-env.sh + +You will be prompted for the values of the required environment variables. There +are some defaults set that you may want to change. You will need to export +``GOOGLE_CREDENTIALS`` with authentication details for your GCP account, +which is outside the scope of this guide. Some parameters are specific to FPF's +GCE setup and may need adjusting if you are running elsewhere. + +Then to run the tests locally: + +.. code:: sh + + make ci-go + +You can use ``./devops/gce-nested/ci-runner.sh`` to provision the remote hosts +while making changes, including rebuilding the Debian packages used in the +Staging environment. See :doc:`virtual_environments` for more information. diff --git a/docs/testing_continuous_integration.rst b/docs/testing_continuous_integration.rst deleted file mode 100644 index 137950e4..00000000 --- a/docs/testing_continuous_integration.rst +++ /dev/null @@ -1,177 +0,0 @@ -.. _ci_tests: - -Testing: CI -=========== - -The SecureDrop project uses CircleCI_ for running automated test suites on code changes. - -.. _CircleCI: https://circleci.com/gh/freedomofpress/securedrop - -The relevant files for configuring the CI tests are the ``Makefile`` in -the main repo, the configuration file at ``.circleci/config.yml``, and -the scripts in ``devops/``. You may want to consult the -`CircleCI Configuration Reference `__ -to interpret the configuration file. Review the ``workflows`` section of the -configuration file to understand which jobs are run by CircleCI. - -The files under ``devops/`` are used to create a libvirt-compatible environment on GCE. -The GCE host is used as the Ansible controller, mimicking a developer's laptop, -to provision the machines and run the :ref:`tests ` against them. - -.. note:: We skip unnecessary jobs, such as the staging run, for pull requests that only - affect the documentation; to do so, we check whether the branch name begins with - ``docs-``. These checks are enforced in different parts of the configuration, - mainly within the ``Makefile``. - -.. warning:: In CI, we rebase branches in PRs on HEAD of the target branch. - This rebase does not occur for branches that are not in PRs. - When a branch is pushed to the shared ``freedomofpress`` remote, CI will run, - a rebase will not occur, and since opening a - `PR does not trigger a re-build `_, - the CI build results are not shown rebased on the latest of the target branch. - This is important to maintain awareness of if your branch is behind the target - branch. Once your branch is in a PR, you can rebuild, push an additional - commit, or manually rebase your branch to update the CI results. - -Running the CI Staging Environment ----------------------------------- - -The staging environment tests will run automatically in CircleCI, when -changes are submitted by Freedom of the Press Foundation staff (i.e. members -of the ``freedomofpress`` GitHub organization). The tests also perform -basic linting and validation, like checking for formatting errors in the -Sphinx documentation. - -.. tip:: You will need a Google Cloud Platform account to proceed. - See the `Google Cloud Platform Getting Started Guide`_ for detailed instructions. - -.. _Google Cloud Platform Getting Started Guide: https://cloud.google.com/getting-started/ - -In addition to a GCP account, you will need a working `Docker installation`_ in -order to run the container that builds the deb packages. - -You can verify that your Docker installation is working by running -``docker run hello-world`` and confirming you see "Hello from Docker" in the -output as shown below: - -.. code:: sh - - $ docker run hello-world - - Hello from Docker! - This message shows that your installation appears to be working correctly. - ... - -.. _Docker installation: https://docs.docker.com/install/ - -Setup Environment Parameters -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Source the setup script using the following command: - -.. code:: sh - - source ./devops/gce-nested/ci-env.sh - -You will be prompted for the values of the required environment variables. There -are some defaults set that you may want to change. You will need to export -``GOOGLE_CREDENTIALS`` with authentication details for your GCP account, -which is outside the scope of this guide. - -Use Makefile to Provision Hosts -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Run ``make help`` to see the full list of CI commands in the Makefile: - -.. code:: sh - - $ make help - Makefile for developing and testing SecureDrop. - Subcommands: - ci-go Creates, provisions, tests, and destroys GCE host for testing staging environment. - ci-lint Runs linting in linting container. - ci-teardown Destroys GCE host for testing staging environment. - -To run the tests locally: - -.. code:: sh - - make ci-go - -You can use ``./devops/gce-nested/ci-runner.sh`` to provision the remote hosts -while making changes, including rebuilding the Debian packages used in the -Staging environment. See :doc:`virtual_environments` for more information. - -Debugging CI Issues and Connecting to Remote Instances -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For the staging tests, a container will be spawned on CircleCI, which will then -create a Google Compute instance with nested virtualization and will set up the -virtual environment and run the playbooks on that remote. - -Cloud instances are deleted after the test run is completed, whether a test run -passes or fails. In order to debug the state of the remote instance, we must first -ensure that the instance is not automatically destroyed. Note that there is also -a cron job that destroys instances daily as well. The following is an example -of a commit to apply to a branch in order disable the deletion for the Focal staging job: - -.. code:: Diff - - diff --git a/.circleci/config.yml b/.circleci/config.yml - index 4d61769f1..af74672bc 100644 - --- a/.circleci/config.yml - +++ b/.circleci/config.yml - @@ -251,13 +251,6 @@ jobs: - make ci-go - no_output_timeout: 35m - - - - run: - - name: Ensure environment torn down - - # Always report true, since env should will destroyed already - - # if all tests passed. - - command: make ci-teardown || true - - when: always - - - - store_test_results: - path: ~/sd/junit - - diff --git a/devops/gce-nested/ci-go.sh b/devops/gce-nested/ci-go.sh - index ff80aa107..65bbcd7b9 100755 - --- a/devops/gce-nested/ci-go.sh - +++ b/devops/gce-nested/ci-go.sh - @@ -16,4 +16,3 @@ export BASE_OS="${BASE_OS:-focal}" - - ./devops/gce-nested/gce-start.sh - ./devops/gce-nested/gce-runner.sh - -./devops/gce-nested/gce-stop.sh - diff --git a/devops/scripts/create-staging-env b/devops/scripts/create-staging-env - index 8b296be94..df8a4d674 100755 - --- a/devops/scripts/create-staging-env - +++ b/devops/scripts/create-staging-env - @@ -32,7 +32,7 @@ printf "Creating staging environment via '%s'...\\n" "${securedrop_staging_scena - virtualenv_bootstrap - # Are we in CI? Then lets do full testing post install! - if [ "$USER" = "sdci" ]; then - - molecule test -s "${securedrop_staging_scenario}" - + molecule test --destroy=never -s "${securedrop_staging_scenario}" - else - molecule "${MOLECULE_ACTION:-converge}" -s "${securedrop_staging_scenario}" "${EXTRA_ANSIBLE_ARGS[@]}" - fi - -Once that commit is pushed, run the ``staging-test-with-rebase`` job -with ssh using with CircleCI. Once logged into that container, you can ssh into the -Google Compute host: - -.. code:: sh - - ssh -i /tmp/gce-nested/gce sdci@ - -Once on the GCP host, the SecureDrop source is in ``/home/sdci/securedrop-source`` -and you may activate the virtualenv, list the molecule instances and connect to -VM instances: - -.. code:: sh - - cd securedrop-source - source .venv/bin/activate - molecule list - molecule login -s libvirt-staging-focal --host app-staging