Exec error for multi-platform builds of feature layers

[MartinLoeper]

I just enabled the multi-platform build feature in my ci/cd pipeline which runs on x86. I encountered an exec format error during a feature layer build as follows:

------
 > [linux/arm64 dev_containers_target_stage 4/5] RUN --mount=type=bind,from=dev_containers_feature_content_source,source=cosign_0,target=/tmp/build-features-src/cosign_0     cp -ar /tmp/build-features-src/cosign_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/cosign_0  && cd /tmp/dev-container-features/cosign_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/cosign_0:
#0 0.192 Feature       : Cosign (via Github Releases)
#0 0.192 Description   : Cosign is a command line utility that can sign and verify software artifact, such as container images and blobs.
#0 0.192 Id            : ghcr.io/devcontainers-extra/features/cosign
#0 0.192 Version       : 1.0.9
#0 0.192 Documentation : http://github.com/devcontainers-contrib/features/tree/main/src/cosign
#0 0.192 Options       :
#0 0.192     VERSION="latest"
#0 0.192 ===========================================================================
#0 0.210 Error while loading /tmp/dev-container-features/cosign_0/./install.sh: Exec format error
#0 0.212 ERROR: Feature "Cosign (via Github Releases)" (ghcr.io/devcontainers-extra/features/cosign) failed to install! Look at the documentation at http://github.com/devcontainers-contrib/features/tree/main/src/cosign for help troubleshooting this error.
------

My github actions workflow is configured as follows:

jobs:
  docker-build:
    runs-on: nodejs-dind
    steps:
      - name: Checkout (GitHub)
        uses: actions/checkout@v3
      - name: Set up QEMU for multi-architecture builds
        uses: docker/setup-qemu-action@v3
      - name: Setup Docker buildx for multi-architecture builds
        uses: docker/setup-buildx-action@v3
        with:
          use: true
          platforms: linux/amd64,linux/arm64
      - name: Login to Harbor Container Registry
        uses: docker/login-action@v2
        with:
          registry: harbor.xxxxxx.org
          username: ${{ vars.HARBOR_USER }}
          password: ${{ secrets.HARBOR_PASSWORD }}
      - name: Pre-build image
        uses: devcontainers/[email protected]
        with:
          imageName: harbor.xxxxxxx.org/yyyyy/devcontainers/core
          push: always
          platform: linux/amd64,linux/arm64
          configFile: ./.devcontainer/build/devcontainer.json

devcontainer.json:

{
	"name": "Core (build)",
	"build": {
		"dockerfile": "../Dockerfile"
	},
	"features": {
		"ghcr.io/devcontainers/features/docker-outside-of-docker:1": {
			"moby": false,
			"installDockerBuildx": true,
			"installDockerComposeSwitch": true,
			"version": "23.0.1",
			"dockerDashComposeVersion": "v2"
		},
		"ghcr.io/devcontainers-extra/features/cosign:1": {}
	}
}

Dockerfile:

FROM mcr.microsoft.com/devcontainers/typescript-node:1-22-bookworm

RUN corepack enable yarn

USER node
RUN corepack prepare [email protected] --activate

Has anybody an idea why this might happen? I would expect qemu to hook into the execution of install.sh above. So no idea what is going on there.

[Kaniska244]

Hello @MartinLoeper ,

As per the action yml, the docker-build job is running on a nodejs-dind image. Is this a custom image locally created? If this is same as the one from below GitHub repo (very old one) then its built on node 10 with utilities such as git, libxml2-utils, jq, curl, libudev1, iptables along with installed docker (version 17.05.0-ce) and EntryKit (version 0.4.0), finally sets up an entry point for Docker daemon using Entrykit.
https://github.com/making/nodejs-dind/tree/master

If I further examine the above mentioned repo, I see the docker & EntryKit installers used are x86_64 versions & the point to be noted is that the very first arm64 build in sequence which was for cosign, failed with exec format error.

Now if I replace the "nodejs-dind" with "ubuntu-latest" in "runs-on" tag in the yml, the build goes through for me for the same configuration as yours along with cosign and docker-outside-of-docker for both amd64 and arm64.

So the problem could be with this nodejs-dind unless its a custom image locally created.

With Regards,
Kaniska

[MartinLoeper]

Thanks for your very detailed answer @Kaniska244. Much appreciated!

The image I am using is the one from the official gitea act runner repo (an alternative to github runner): https://gitea.com/gitea/act_runner/src/branch/main/Dockerfile

I built multi-architecture docker images with the aforementioned nodejs-dind image by gitea. That's why I thought it might be an issue with the devcontainers cli or the devcontainer features used in my devcontainer.json that make my multi-arch devcontainer image build fail.

If you have time, feel free to take a look into the gitea image. I am afraid, I will not figure out what is wrong with it.