docker-compose: tweaks to work on an M1 MacBook

* Install PostgreSQL v12 instead of latest v15.
* Introduce SAUCE_OS variable to download compatible Sauce Connect.
* Extend installation docs on Docker Compose to reflect experience.
* Build psycopg2 from source instead of using psycopg2-binary.

Dockerfile

@@ -16,6 +16,7 @@ ENTRYPOINT [ "python3" ]
 CMD [ "--version" ]
 
 ARG APP_ENVIRONMENT
+ARG SAUCE_OS
 
 RUN curl -sL https://deb.nodesource.com/setup_14.x | bash -
 RUN apt-get update
@@ -42,8 +43,8 @@ RUN hepdata collect -v  && \
 
 RUN bash -c "echo $APP_ENVIRONMENT"
 
-RUN bash -c "set -x; [[ ${APP_ENVIRONMENT:-prod} = local-web ]] && (cd /usr/local/var && wget https://saucelabs.com/downloads/sc-4.8.1-linux.tar.gz && \
-  tar -xvf sc-4.8.1-linux.tar.gz) || echo 'Not installing SC on prod or worker build'"
+RUN bash -c "set -x; [[ ${APP_ENVIRONMENT:-prod} = local-web ]] && (cd /usr/local/var && wget https://saucelabs.com/downloads/sc-4.8.1-${SAUCE_OS:-linux}.tar.gz && \
+  tar -xvf sc-4.8.1-${SAUCE_OS:-linux}.tar.gz) || echo 'Not installing SC on prod or worker build'"
 
 WORKDIR /code
 

INSTALL.rst

@@ -326,26 +326,33 @@ on the Kubernetes cluster at CERN.
 
 For local development you can use the ``docker-compose.yml`` file to run the HEPData Docker image and its required services.
 
-First, ensure you have installed `Docker <https://docs.docker.com/install/>`_ and `Docker Compose <https://docs.docker.com/compose/install/>`__.
+First, ensure you have installed `Docker <https://docs.docker.com/install/>`_ and `Docker Compose <https://docs.docker.com/compose/install/>`_.
 
 Copy the file ``config_local.docker_compose.py`` to ``config_local.py``.
 
 In order to run the tests via Sauce Labs, ensure you have the variables ``$SAUCE_USERNAME`` and ``$SAUCE_ACCESS_KEY``
 set in your environment (see :ref:`running-the-tests`) **before** starting the containers.
 
+If using an M1 MacBook, also add ``export SAUCE_OS=linux-arm64`` to your bash or zsh profile. This is necessary to
+download the correct `Sauce Connect Proxy
+<https://docs.saucelabs.com/secure-connections/sauce-connect/installation/#downloading-sauce-connect-proxy>`_
+client.
+
 Start the containers:
 
 .. code-block:: console
 
    $ docker-compose up
 
-(This starts containers for all the 5 necessary services. See :ref:`docker-compose-tips` if you only want to run some containers.)
+(This starts containers for all the 6 necessary services. See :ref:`docker-compose-tips` if you only want to run some containers.)
 
 In another terminal, initialise the database:
 
 .. code-block:: console
 
+   $ docker-compose exec web bash -c "hepdata utils reindex -rc True"  # ignore error "hepsubmission" does not exist
    $ docker-compose exec web bash -c "mkdir -p /code/tmp; ./scripts/initialise_db.sh [email protected] password"
+   $ docker-compose exec db bash -c "psql hepdata -U hepdata -c 'update accounts_user set confirmed_at=NOW() where id=1;'"
 
 Now open http://localhost:5000/ and HEPData should be up and running. (It may take a few minutes for Celery to process
 the sample records.)
@@ -354,16 +361,19 @@ To run the tests:
 
 .. code-block:: console
 
-   $ docker-compose exec web bash -c "/usr/local/var/sc-4.8.1-linux/bin/sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY --region eu-central & ./run-tests.sh"
+   $ docker-compose exec web bash -c "/usr/local/var/sc-4.8.1-${SAUCE_OS:-linux}/bin/sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY --region eu-central & ./run-tests.sh"
 
+On an M1 MacBook, the end-to-end tests failed with ``WARNING: security - Error with Permissions-Policy header:
+Unrecognized feature: 'interest-cohort'.`` and the ``assert len(log) == 0`` line in ``tests/e2e/conftest.py`` had to be
+commented out for all tests to pass.
 
 .. _docker-compose-tips:
 
 Tips
 ====
 
 * If you see errors about ports already being allocated, ensure you're not running any of the services another way (e.g. hepdata-converter via Docker).
-* If you want to run just some of the containers, specify their names in the docker-compose command. For example, to just run the web server, database and OpenSearch, run:
+* If you want to run just some of the containers, specify their names in the ``docker-compose`` command. For example, to just run the web server, database and OpenSearch, run:
 
   .. code-block:: console
 
@@ -372,9 +382,12 @@ Tips
   See ``docker-compose.yml`` for the names of each service. Running a subset of containers could be useful in the following cases:
 
    * You want to use the live converter service, i.e.  ``CFG_CONVERTER_URL = 'https://converter.hepdata.net'`` instead of running the converter locally.
-   * You want to run the container for the web service by pulling an image from DockerHub instead of building an image locally.
+   * You want to run the container for the web service by pulling an image from Docker Hub instead of building an image locally.
    * You want to run containers for all services apart from web (and maybe converter) then use a non-Docker web service.
 
+  If using Docker Desktop, you need to use ``host.docker.internal`` instead of ``localhost`` when `connecting from a
+  container to a service on the host <https://docs.docker.com/desktop/networking/#use-cases-and-workarounds-for-all-platforms>`_.
+
 * To run the containers in the background, run:
 
   .. code-block:: console

docker-compose.yml

@@ -5,6 +5,7 @@ services:
       context: .
       args:
         - "APP_ENVIRONMENT=local-web"
+        - "SAUCE_OS=${SAUCE_OS:-linux}"
     command: hepdata run -h 0.0.0.0 -p 5000 --debugger --reload
     ports:
     - "5000:5000"

docker/db/Dockerfile

@@ -1,4 +1,4 @@
-FROM postgres
+FROM postgres:12
 
 # Custom initialization scripts
 COPY ./create_test_db.sh     /docker-entrypoint-initdb.d/20-create_test_db.sh

hepdata/version.py

@@ -28,4 +28,4 @@
 and parsed by ``setup.py``.
 """
 
-__version__ = "0.9.4dev20221110"
+__version__ = "0.9.4dev20221111"

requirements.txt

@@ -32,7 +32,7 @@ lxml==4.9.1
 MarkupSafe==2.1.1
 msgpack==1.0.4
 pluggy==0.13.1    # Fixed version to make sure it works with jsonresolve and pytest
-psycopg2-binary==2.9.5
+psycopg2==2.9.5
 python-dateutil==2.8.2
 pyyaml==6.0
 requests==2.28.1

tests/e2e/conftest.py

@@ -54,7 +54,6 @@
 class SQLAlchemy(InvenioSQLAlchemy):
     def apply_pool_defaults(self, app, options):
         # See https://github.com/pallets/flask-sqlalchemy/issues/589#issuecomment-361075700
-        # Will need updating if we move to flask-sqlalchemy >= 2.5
         options = super().apply_pool_defaults(app, options)
         options["pool_pre_ping"] = True
         return options