error 413 request entity too large while pushing an image

[waveywaves]

I am getting an error while pushing to ttl.sh, 413 request entity too large.
https://github.com/waveywaves/backstage/actions/runs/3103852916/jobs/5027642820#step:16:661

Based on the docker documentation I understand that the following configuration is necessary to get around this issue
https://docs.docker.com/registry/recipes/nginx/#setting-things-up

# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;

[z4ce]

How big is the layer you're trying to copy? Do you have this image on another registry like docker hub so we can try to reproduce?

[waveywaves]

Image information and size of each layer

> Run docker image ls ttl.sh/0f030c5a-7565-4821-af90-5e83ca59b0cf:2h
REPOSITORY                                    TAG       IMAGE ID       CREATED          SIZE
ttl.sh/0f030c5a-7565-4821-af90-5e83ca59b0cf   2h        656890bddeb1   45 seconds ago   999MB
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
656890bddeb1   45 seconds ago       CMD ["/docker-entrypoint.sh"]                   0B        buildkit.dockerfile.v0
<missing>      45 seconds ago       COPY app-config.yaml ./ # buildkit              3.77kB    buildkit.dockerfile.v0
<missing>      45 seconds ago       RUN /bin/sh -c tar xzf bundle.tar.gz && rm b…   35.4MB    buildkit.dockerfile.v0
<missing>      45 seconds ago       COPY /app/packages/backend/dist/bundle.tar.g…   8.74MB    buildkit.dockerfile.v0
<missing>      46 seconds ago       RUN /bin/sh -c tar xzf skeleton.tar.gz && rm…   484MB     buildkit.dockerfile.v0
<missing>      About a minute ago   COPY /app/yarn.lock /app/package.json /app/p…   903kB     buildkit.dockerfile.v0
<missing>      3 minutes ago        ENV NODE_ENV=production                         0B        buildkit.dockerfile.v0
<missing>      3 minutes ago        WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      3 minutes ago        USER node                                       0B        buildkit.dockerfile.v0
<missing>      3 minutes ago        RUN /bin/sh -c chmod +x docker-entrypoint.sh…   280MB     buildkit.dockerfile.v0
<missing>      4 minutes ago        COPY packages/backend/docker-entrypoint.sh .…   245B      buildkit.dockerfile.v0
<missing>      13 days ago          /bin/sh -c #(nop)  CMD ["node"]                 0B        
<missing>      13 days ago          /bin/sh -c #(nop)  ENTRYPOINT ["docker-entry…   0B        
<missing>      13 days ago          /bin/sh -c #(nop) COPY file:4d192565a7220e13…   388B      
<missing>      13 days ago          /bin/sh -c set -ex   && savedAptMark="$(apt-…   9.5MB     
<missing>      13 days ago          /bin/sh -c #(nop)  ENV YARN_VERSION=1.22.19     0B        
<missing>      13 days ago          /bin/sh -c ARCH= && dpkgArch="$(dpkg --print…   100MB     
<missing>      13 days ago          /bin/sh -c #(nop)  ENV NODE_VERSION=16.17.0     0B        
<missing>      13 days ago          /bin/sh -c groupadd --gid 1000 node   && use…   333kB     
<missing>      13 days ago          /bin/sh -c #(nop)  CMD ["bash"]                 0B        
<missing>      13 days ago          /bin/sh -c #(nop) ADD file:5bd53bff884e470b3…   80.5MB    

the largest layer is 484MB.

You can check the github action where I am running the above command over here. I also attempt to push the image to ttl.sh after, after which it fails.

[waveywaves]

I found a related issue where the user is facing the same problem. docker/build-push-action#634 (comment)
I see that there is a cloudflare folder present in the ttl.sh repo as well. So I believe this requests are gtting proxied through cloudflare. The following excerpt from the above comment might be of some help to fix this.

The harbor domain was proxied through Cloudflare, disabling it finally let the 
action push the image successfully. I guess it make sense for CF to return 
413 because I never got it in my nginx logs. It feels weird though, why did 
building and pushing through the cli succeed? Did I just stumble into an 
edge case where manual build produced a layer within CF's limits while 
the github builder went over?

[z4ce]

Can you try this from another location? I suspect your error is coming from some kind of proxy between you and the ttl.sh service. When I try to push a 1gb layer it seems to work fine.

─ docker image list
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
<none>       <none>    6821543d4f57   53 seconds ago   1.05GB
╰─ docker tag 6821543d4f57 ttl.sh/iz-test:1hr
╰─ docker push ttl.sh/iz-test:1hr
The push refers to repository [ttl.sh/iz-test]
23d2a9cb459a: Pushed
7fb796040f6b: Pushed
1hr: digest: sha256:ace2775a9f8d29121c9ba814e8b3a36bf7567fcfbb999baa335619cbdc07befe size: 740
╰─ docker history ttl.sh/iz-test:1hr
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
6821543d4f57   7 minutes ago   COPY ./bigfile /var/bigfile # buildkit          1.05GB    buildkit.dockerfile.v0
<missing>      12 days ago     /bin/sh -c #(nop)  CMD ["sh"]                   0B
<missing>      12 days ago     /bin/sh -c #(nop) ADD file:e0416d710749b8a1d…   1.38MB

Here is my Dockerfile if you want to try to reproduce:

from busybox
copy ./bigfile /var/bigfile

I used dd if=/dev/urandom of=./bigfile bs=1M count=1000 to make the 1gb file.

[waveywaves]

I was able to push a bigger image as well. Did you check ? #104 (comment)

Can you share the url to the heroku instance where you have ttl.sh deployed ? I would like to go ahead and push there directly to bypass cloudflare.

[kaovilai]

I am hitting this issue in a repeatable GitHub actions. Feel free to try there.

[kaovilai]

@z4ce please check https://github.com/richstokes/k8s-fah/actions/runs/3157318749/jobs/5138045405#step:11:263

[z4ce]

Hi @kaovilai I spoke with @waveywaves on Kubernetes slack. I believe this because the buildx plugin is using the the push driver from containerd which apparently does not support transfer-encoding: chunked see:
https://github.com/containerd/containerd/blob/main/remotes/docker/pusher.go#L265

The best workaround for now is just to use regular docker push instead of using buildx to push your image. I do see some other code in the buildx code for using a moby driver which might as well, but I'm not sure how to enable it through the github action.

[z4ce]

I did some testing, and it's definitely the containerd push driver that is not chunking. I can reproduce by doing

./ctr i pull docker.io/library/openjdk:20-jdk-slim-buster --all-platforms
./ctr image tag docker.io/library/openjdk:20-jdk-slim-buster ttl.sh/iz-test-123:1hr
./ctr --debug image push ttl.sh/iz-test-123:1hr

Result:

DEBU[0001] unexpected response                           body="<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" digest="sha256:8c3a8eecab48f4c2c2053b014c1ce723acdbee8fb8d8831002f1672bb3db6843" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip resp="&{413 Request Entity Too Large 413 HTTP/1.1 1 1 map[Alt-Svc:[h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400] Cf-Ray:[752dbafd2a9c872f-ORD] Connection:[keep-alive] Content-Type:[text/html] Date:[Fri, 30 Sep 2022 14:43:21 GMT] Nel:[{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}] Report-To:[{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=iipZHB6T8KSIk4ebIOT5oRyCzUx2hFu1xZhuURFrx0rx9nW6hkSkLNvQuKslhpwHSqI27EHj27O53lNSpvhdFHWZPIXsCIBrdbPUkHF6eSumf4rUpuRX5cg%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}] Server:[cloudflare]] 0x14000702040 -1 [chunked] false false map[] 0x14000208000 0x1400053a0b0}" size=197212069

[z4ce]

I submitted a PR to fix this in the upstream buildx uses:
containerd/containerd#7459