ci: enable VNC over ssh

.circleci/config.yml

@@ -88,6 +88,12 @@ aliases:
         git checkout -B "$CIRCLE_BRANCH" "$CIRCLE_SHA1"
       fi
 
+      # Piggyback on this alias to enable VNC connections
+      # The if statement will only be true on node-browsers executors
+      if [ "${SHELL}" != "/bin/bash" ]; then
+        cat ${HOME}/project/.circleci/scripts/enable-vnc.sh >> ~/.bashrc
+      fi
+
   # Check if MMI Optional tests should run
   - &check-mmi-optional
     name: Check if MMI Optional tests should run

.circleci/scripts/enable-vnc.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+# This script is based on the documentation from CircleCI, which does not work as written
+# https://circleci.com/docs/browser-testing/#interacting-with-the-browser-over-vnc
+
+set -e
+set -u
+set -o pipefail
+set -x
+
+cd "${HOME}/project"
+
+# Install a VNC server
+readonly LOCK_FILE="installed.lock"
+if [ ! -f "${LOCK_FILE}" ]; then
+  sudo apt update
+  sudo apt install -y x11vnc
+
+  touch "${LOCK_FILE}"
+fi
+
+# Start VNC server
+if ! pgrep x11vnc > /dev/null; then
+  x11vnc -display "$DISPLAY" -bg -forever -nopw -quiet -listen localhost -xkb -rfbport 5901 -passwd password
+fi

.circleci/scripts/test-run-e2e.sh

@@ -11,11 +11,20 @@ then
     exit 1
 fi
 
+# Skip running e2e tests if we're doing "Rerun job with SSH" and we're not on the first node
+if netstat -tnlp | grep -q 'circleci-agent' && [ "$CIRCLE_NODE_INDEX" -ne 0 ]
+then
+    printf '"Rerun job with SSH" and not on the first node, shutting down\n'
+    circleci-agent step halt
+    exit 1
+fi
+
 # Run the actual test command from the parameters
 timeout 20m "$@" --retries 1
 
 # Error code 124 means the command timed out
-if [ $? -eq 124 ]; then
+if [ $? -eq 124 ]
+then
   # Once deleted, if someone tries to "Rerun failed tests" the result will be
   # "Error: can not rerun failed tests: no failed tests could be found"
   echo 'Timeout error, deleting the test results'

docs/ssh-to-circleci.md

@@ -0,0 +1,33 @@
+# Debugging E2E tests by connecting to CircleCI over SSH and VNC
+
+Developers often say "I can't reproduce this CI failure locally, but it fails on CircleCI."
+
+If you find yourself in that situation, one option is to use Codespaces, which can reproduce some of these failures, and is a little bit easier to use.
+
+The other NEW option is to SSH into CircleCI and use VNC.
+
+1. You must be logged into CircleCI and have access to metamask-extension
+2. "Rerun job with SSH" [Documentation](https://circleci.com/docs/ssh-access-jobs/)
+3. Look for this instruction inside the `Enable SSH` section
+
+   ```
+   You can now SSH into this box if your SSH public key is added:
+       $ ssh -p xxxxx xxx.xxx.xxx.xxx
+   ```
+
+4. Copy the command that CircleCI gives you and append `-L 5901:localhost:5901` (this will tunnel the VNC connection over SSH)
+5. Enter this in a terminal, for example `ssh -p xxxxx xxx.xxx.xxx.xxx -L 5901:localhost:5901`
+6. When you login to SSH, it automatically executes `/.circleci/scripts/enable-vnc.sh` to set up the connection
+7. Use your favorite VNC viewer on your local machine to connect to `localhost:5901`
+   - Mac: In the Terminal, run `open vnc://localhost:5901`
+   - Linux: `tigervnc-viewer` is a good package to match the server
+   - Windows: [RealVNC Viewer](https://www.realvnc.com/en/connect/download/viewer/windows/) or [TightVNC](https://www.tightvnc.com/download.php)
+8. The VNC password is simply `password`
+9. The normal E2E tests will already be running, and you can watch them run
+10. If you want to stop the normal tests and run your own tests, run `pkill timeout` (this works because .circleci/scripts/test-run-e2e.sh runs the `timeout` command)
+
+### Notes:
+
+- This procedure was based on the documentation from CircleCI [here](https://circleci.com/docs/browser-testing/#interacting-with-the-browser-over-vnc). The way they wrote it doesn't really work correctly, but we fixed it.
+- If you run "Rerun job with SSH" on a job that has `parallelism: 24`, it will rerun all 24 VMs, but quickly shut down all but the first one.
+- **Advanced Usage:** If you don't want to run the `.bashrc` when you connect, append this to the SSH command `-t bash --norc --noprofile`