reference: Document how to configure/modify CA certificate trust stores

This information is a distillation of guidance and assistance I've
provided to CDC and other downstream users over the years.  I extracted
it from my various notes and recollections and verified some specific
behaviour anew.  An official reference page will be useful for pointing
users to in the future, and serves as a collection point for future
recommendations/information uncovered in troubleshooting sessions.

Related-to: <nextstrain/ncov#1055>
Related-to: <nextstrain/nextclade#726>

src/conf.py

@@ -83,7 +83,8 @@
     'auspice': ('https://docs.nextstrain.org/projects/auspice/page/', None),
     'cli': ('https://docs.nextstrain.org/projects/cli/page/', None),
     'nextclade': ('https://docs.nextstrain.org/projects/nextclade/page/', None),
-    'ncov': ('https://docs.nextstrain.org/projects/ncov/page/', None)
+    'ncov': ('https://docs.nextstrain.org/projects/ncov/page/', None),
+    'python': ('https://docs.python.org/3/', None),
 }
 
 

src/index.rst

@@ -96,6 +96,7 @@ team and other Nextstrain users provide assistance.  For private inquiries,
     reference/data-formats
     reference/data-files
     FAQ <reference/faq>
+    reference/ca-certificates
     reference/snakemake-style-guide
     reference/documentation-style-guide
     reference/governance

src/reference/ca-certificates.rst

@@ -0,0 +1,179 @@
+===========================
+CA certificate trust stores
+===========================
+
+Secure network connections using TLS (or SSL), e.g. ``https://`` URLs, are
+based on *certificates* issued to server operators by *certificate authorities*
+(CAs).  Each program which makes these secure connections ultimately needs to
+have a list of CAs that are to be trusted and be able to trace the issuance of
+a server's certificate back to a trusted CA's root certificate.
+
+This set of trusted CA root certificates is called a *trust store*.  Every
+computer system has its own OS-level trust store and usually also has several
+other application-specific trust stores on the system.  Typically these are
+automatically managed and do not need configuration or modification.
+
+When your network environment includes a proxy, egress firewall, or
+institutional CA used internally, however, it may be necessary to configure the
+trust store used by an application or modify the trust store contents (e.g.
+adding a institutional CA certificate).
+
+This document focuses on configuration and modification of trust stores used by
+software that is in turn used by Nextstrain's software.  It focuses only on
+changes that can be made without modification of source or program code, e.g.
+thru environment variables or config files.
+
+.. contents::
+   :local:
+   :depth: 3
+
+
+Operating system
+================
+
+Linux uses :ref:`OpenSSL <openssl>` for its system-wide trust store.  macOS and
+Windows use their own platform-specific stores, Keychain (managed by
+Keychain.app) and Certificate Store (managed via ``certmgr.msc``),
+respectively.  OpenSSL and some other software on macOS and Windows systems is
+sometimes configured by default to use those platform-specific stores, but not
+always.
+
+If you need to include an additional CA certificate in the trust store and that
+certificate applies to *all* software on the computer (e.g. your web browser),
+not just Nextstrain's, then consider adding the CA certificate to the
+system-wide trust store and configuring Nextstrain's software to use the
+system's trust store instead of any application-specific ones when necessary.
+This blanket approach is generally more reliable over time than
+application-by-application configuration.
+
+
+Specific software
+=================
+
+Most software provides a way to override the trust store by providing a path to
+a file containing the the entire set of CA certificates to trust.  Thus, if you
+only need to include an additional CA certificate in the trust store, you
+should ensure that the file you provide contains a standard set of CA
+certificates as well as your addition.  An exception here is if you know for
+certain that all connections will use your CA certificate and only yours (e.g.
+a mandatory egress proxy that removes (and adds back) TLS for inspection and
+policy purposes).
+
+.. _openssl:
+
+OpenSSL library
+---------------
+
+OpenSSL is the most common library used to provide TLS/SSL support in other
+software.  Its `default locations of trusted CA certificates
+<https://docs.openssl.org/3.0/man3/SSL_CTX_load_verify_locations/>`__ can be
+overridden by setting the ``SSL_CERT_FILE`` and/or ``SSL_CERT_DIR`` environment
+variables.
+
+Its final trust store is built from certificates in all default locations, so
+to *comprehensively* override the defaults, all locations must be overridden.
+This is typically unnecessary unless you need to ensure some CAs in the
+defaults *aren't* trusted.
+
+.. note::
+   Each Conda environment or Docker container has its own isolated copy
+   of OpenSSL and its own isolated default CA certificate trust store.
+
+
+.. _node.js:
+
+Node.js
+-------
+
+*Used by nextstrain.org and Auspice's standalone server component.*
+
+Node.js may use either its own bundled snapshot of `Mozilla's CA trust store`_
+or the :ref:`OpenSSL <openssl>` default CA trust store, with the default choice
+depending on how it was built.
+
+This typically means that ``node`` from your official OS package repositories
+defaults to the OpenSSL default CA trust store while ``node`` from `nodejs.org
+<https://nodejs.org>`__ (e.g. via ``nvm``), `conda-forge
+<https://anaconda.org/conda-forge/nodejs>`__, and other third-party places
+defaults to the bundled CA trust store.  You can be explicit about the
+preferred CA trust store by including |--use-openssl-ca|_ or
+|--use-bundled-ca|_ in the |NODE_OPTIONS| environment variable or by passing
+those options directly in your ``node`` invocation.
+
+To override the trust store, set the |NODE_OPTIONS|_ environment variable to
+include ``--use-openssl-ca`` and then set OpenSSL's ``SSL_CERT_FILE`` or
+``SSL_CERT_DIR`` environment variables.
+
+To add CA certificates to the trust store, set the |NODE_EXTRA_CA_CERTS|_
+environment variable.
+
+.. |--use-openssl-ca| replace:: ``--use-openssl-ca``
+.. _--use-openssl-ca: https://nodejs.org/api/cli.html#--use-bundled-ca---use-openssl-ca
+
+.. |--use-bundled-ca| replace:: ``--use-bundled-ca``
+.. _--use-bundled-ca: https://nodejs.org/api/cli.html#--use-bundled-ca---use-openssl-ca
+
+.. |NODE_OPTIONS| replace:: ``NODE_OPTIONS``
+.. _NODE_OPTIONS: https://nodejs.org/api/cli.html#node_optionsoptions
+
+.. |NODE_EXTRA_CA_CERTS| replace:: ``NODE_EXTRA_CA_CERTS``
+.. _NODE_EXTRA_CA_CERTS: https://nodejs.org/api/cli.html#node_extra_ca_certsfile
+
+
+.. _aws:
+
+AWS CLI and SDK
+---------------
+
+*Used by Nextstrain CLI, some pathogen workflows, and nextstrain.org.*
+
+The AWS CLI, via `Botocore`_, uses `Mozilla's CA trust store`_ via the
+|certifi|_ Python package when it's available, otherwise it falls back to `its
+own bundled CA trust store <botocore-cacerts_>`_.
+
+The JavaScript SDKs default to the :ref:`Node.js <node.js>` trust store.
+
+Set the |AWS_CA_BUNDLE|_ environment variable or |ca_bundle|_ profile
+configuration to override.
+
+.. note::
+   If an AWS-specific CA bundle isn't configured, the AWS CLI (but not the
+   SDKs) falls back to the |REQUESTS_CA_BUNDLE|_ environment variable (if set).
+
+.. _Botocore: https://pypi.org/project/botocore/
+.. _Mozilla's CA trust store: https://wiki.mozilla.org/CA/Included_Certificates
+.. _botocore-cacerts: https://github.com/boto/botocore/blob/master/botocore/cacert.pem
+
+.. |certifi| replace:: ``certifi``
+.. _certifi: https://pypi.org/project/certifi/
+
+.. |AWS_CA_BUNDLE| replace:: ``AWS_CA_BUNDLE``
+.. _AWS_CA_BUNDLE: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list-AWS_CA_BUNDLE
+
+.. |ca_bundle| replace:: ``ca_bundle``
+.. _ca_bundle: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-config-ca_bundle
+
+
+.. _python:
+
+Python
+------
+
+Uses the :ref:`OpenSSL library <openssl>` and its defaults.  See
+:py:meth:`python:ssl.SSLContext.load_default_certs` and
+:py:meth:`python:ssl.SSLContext.set_default_verify_paths`.
+
+
+.. _python-requests:
+
+``requests`` module
+~~~~~~~~~~~~~~~~~~~
+
+*Used by Nextstrain CLI and some pathogen workflows.*
+
+Uses `Mozilla's CA trust store`_ via the |certifi|_ Python package.
+
+Set the |REQUESTS_CA_BUNDLE|_ environment variable to override.
+
+.. |REQUESTS_CA_BUNDLE| replace:: ``REQUESTS_CA_BUNDLE``
+.. _REQUESTS_CA_BUNDLE: https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification