remote docker daemon with self-hosted runners?

[brianjmurrell]

I have been experimenting with self-hosted runners.

To leverage a global docker cache it seemed to make the most sense to have a single docker daemon running on a (big enough) host and then have all of my runners use DOCKER_HOST=$docker_daemon_host to do their docker operations on the single big docker daemon. Docker cache resuse would be maximized that way vs. having a docker daemon and cache on each runner node.

But it seems it's impractical due to GHA's use of docker bind mounts (i.e. docker run ... -v) when running docker containers. I.e.:

/usr/bin/docker run --name dada28a6e70c8df6de4d7aa0ec60a3641b76c4_1ad61d --label dada28 --workdir /github/workspace --rm -e "DEPS_JOBS" -e "COMPILER" -e "BASE_DISTRO" -e "DOCKER_BASE" -e "INPUT_GITHUB_TOKEN" -e "INPUT_FILES" -e "INPUT_COMMENT_MODE" -e "INPUT_FAIL_ON" -e "INPUT_GITHUB_RETRIES" -e "INPUT_COMMIT" -e "INPUT_CHECK_NAME" -e "INPUT_COMMENT_TITLE" -e "INPUT_TIME_UNIT" -e "INPUT_REPORT_INDIVIDUAL_RUNS" -e "INPUT_DEDUPLICATE_CLASSES_BY_FILE_NAME" -e "INPUT_IGNORE_RUNS" -e "INPUT_HIDE_COMMENTS" -e "INPUT_COMMENT_ON_PR" -e "INPUT_JOB_SUMMARY" -e "INPUT_COMPARE_TO_EARLIER_COMMIT" -e "INPUT_PULL_REQUEST_BUILD" -e "INPUT_EVENT_FILE" -e "INPUT_EVENT_NAME" -e "INPUT_TEST_CHANGES_LIMIT" -e "INPUT_CHECK_RUN_ANNOTATIONS" -e "INPUT_CHECK_RUN_ANNOTATIONS_BRANCH" -e "INPUT_SECONDS_BETWEEN_GITHUB_READS" -e "INPUT_SECONDS_BETWEEN_GITHUB_WRITES" -e "INPUT_JSON_FILE" -e "INPUT_JSON_THOUSANDS_SEPARATOR" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/lcl_home/gha_runner/actions-runner/_work/_temp/_github_home":"/github/home" -v "/lcl_home/gha_runner/actions-runner/_work/_temp/_github_workflow":"/github/workflow" -v "/lcl_home/gha_runner/actions-runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/lcl_home/gha_runner/actions-runner/_work/daos/daos":"/github/workspace" dada28:a6e70c8df6de4d7aa0ec60a3641b76c4

and in particular the bind mounts from the above:

-v "/lcl_home/gha_runner/actions-runner/_work/_temp/_github_home":"/github/home"
-v "/lcl_home/gha_runner/actions-runner/_work/_temp/_github_workflow":"/github/workflow"
-v "/lcl_home/gha_runner/actions-runner/_work/_temp/_runner_file_commands":"/github/file_commands"
-v "/lcl_home/gha_runner/actions-runner/_work/daos/daos":"/github/workspace"

as those bind-mount destination paths don't have the same content as they have on the GHA runner since they are bind mounted from the docker daemon host, not the GHA runner.

It seems it might be worthwhile for the runner to issue a warning at startup if it detects that remote docker daemon options (i.e. DOCKER_HOST=$hostname as on example) are being used.

Thoughts? Is my analysis above correct or am I missing something?