ERROR: failed to prepare sha256:***: open /var/lb/docker/overlay2/.../.tmp-committed3822505730: no such file or directory

[pjonsson]

Sometimes, when building multiple images concurrently, the build fails with an error message along the lines of ERROR: failed to prepare sha256:b658d8ac8aea7292c464218dbf62c5bcc46468f8b7e017c0380ab37b3f903261 as kcty4fbxag6hngkemyhs5dn82: open /var/lib/docker/overlay2/q32w4cbarh6m1wpr169nbtyib/.tmp-committed2557151495: no such file or directory.

Example with Docker 27.1.1 on Ubuntu 24.04:

$ docker build --tag registry.com/current/image:prod-1_8799-9894dfca4 --target deploy --pull --no-cache --build-context node:16=docker-image://registry.com/upstream/node:20 --build-context node:16-slim=docker-image://registry.com/upstream/node:20-slim --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from registry.com/current/image:latest --tag registry.com/current/image:latest --file Dockerfile .
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile:
#1 transferring dockerfile: 1.03kB done
#1 DONE 0.2s
#2 [auth] upstream/node:pull token for registry.com
#2 DONE 0.0s
#3 [context node:16] load metadata for registry.com/upstream/node:20
#3 ...
#4 [context node:16-slim] load metadata for registry.com/upstream/node:20-slim
#4 DONE 1.3s
#3 [context node:16] load metadata for registry.com/upstream/node:20
#3 DONE 1.4s
#5 [internal] load .dockerignore
#5 transferring context:
#5 transferring context: 2B done
#5 DONE 0.4s
#6 importing cache manifest from registry.com/current/image:latest
#6 DONE 0.0s
#7 [internal] load build context
#7 DONE 0.0s
#8 [context node:16-slim] registry.com/upstream/node:20-slim
#8 resolve registry.com/upstream/node:20-slim 0.1s done
#8 DONE 0.1s
#9 [context node:16] registry.com/upstream/node:20
#9 resolve registry.com/upstream/node:20
#9 resolve registry.com/upstream/node:20 0.2s done
#9 DONE 0.2s
#8 [context node:16-slim] registry.com/upstream/node:20-slim
#8 DONE 0.1s
#9 [context node:16] registry.com/upstream/node:20
#9 CACHED
#10 [deploy 1/9] WORKDIR /app
#10 CACHED
#7 [internal] load build context
#7 transferring context: 53.65MB 0.2s done
#7 DONE 0.9s
#11 [build 1/5] COPY --chown=node:node . /app
#11 ERROR: failed to prepare sha256:687657594431858439208623df266bff79437bdbc3e44d922efa979a4582211a as iuvqndpvxpc533hacpitcxv0n: open /var/lib/docker/overlay2/58kyxx3nsrmorhmav02ik037c/.tmp-committed3822505730: no such file or directory
------
> [build 1/5] COPY --chown=node:node . /app:
------
Dockerfile:8
--------------------
   6 |     USER node
   7 |
   8 | >>> COPY --chown=node:node . /app
   9 |
  10 |     WORKDIR /app
--------------------
ERROR: failed to solve: failed to prepare sha256:687657594431858439208623df266bff79437bdbc3e44d922efa979a4582211a as iuvqndpvxpc533hacpitcxv0n: open /var/lib/docker/overlay2/58kyxx3nsrmorhmav02ik037c/.tmp-committed3822505730: no such file or directory

and an other example with Docker 27.1.2 on Ubuntu 24.04 on a different machine building a different image:

$ docker build  --tag registry.com/current/otherimage:prod-1_9333-5e1992a2  --pull --no-cache  --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from registry.com/current/otherimage:latest --tag registry.com/current/otherimage:latest --file Dockerfile .
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile:
#1 transferring dockerfile: 382B done
#1 DONE 0.2s
#2 [internal] load metadata for registry.com/upstream/baseimage:version
#2 ...
#3 [auth] upstream/baseimage:pull token for registry.com
#3 DONE 0.0s
#2 [internal] load metadata for registry.com/upstream/baseimage:version
#2 DONE 0.3s
#4 [internal] load .dockerignore
#4 transferring context:
#4 transferring context: 2B done
#4 DONE 0.2s
#5 [internal] load build context
#5 DONE 0.0s
#6 importing cache manifest from registry.com/current/otherimage:latest
#6 ...
#7 [1/2] FROM registry.com/upstream/baseimage:version@sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75
#7 DONE 0.0s
#8 [auth] current/otherimage:pull token for registry.com
#8 DONE 0.0s
#6 importing cache manifest from registry.com/current/otherimage:latest
#6 inferred cache manifest type: application/vnd.docker.distribution.manifest.v2+json done
#6 DONE 0.4s
#7 [1/2] FROM registry.com/upstream/baseimage:version@sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75
#7 resolve registry.com/upstream/baseimage:version@sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75
#7 ...
#5 [internal] load build context
#5 transferring context: 107.17kB 0.0s done
#5 DONE 0.2s
#7 [1/2] FROM registry.com/upstream/baseimage:version@sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75
#7 resolve registry.com/upstream/baseimage:version@sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75 0.2s done
#7 sha256:84810a8255ef19a61f11859bd06860144478214d09d8d6c2f3fdcf949ec1aea9 12.48kB / 12.48kB done
#7 sha256:49fd4d9d3e0237006d803ea88ec672f91ca150060290f3f9695b811b69de9e75 7.44kB / 7.44kB done
#7 sha256:618c79494d3b65fac75fee08b1c8d945e96242c63235f0a292516c20884f5d92 0B / 4.28kB 0.4s
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 0B / 22.03MB 0.4s
#7 sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0B / 32B 0.4s
#7 sha256:618c79494d3b65fac75fee08b1c8d945e96242c63235f0a292516c20884f5d92 4.28kB / 4.28kB 0.5s done
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 5.24MB / 22.03MB 0.7s
#7 sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 0.5s done
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 9.44MB / 22.03MB 0.9s
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 17.83MB / 22.03MB 1.2s
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 20.97MB / 22.03MB 1.3s
#7 extracting sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435
#7 sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 22.03MB / 22.03MB 1.5s done
#7 extracting sha256:5ff99fb216157350c7ab6fe2b60a021f7f3effa104e0edf77471371e84bc1435 0.3s done
#7 extracting sha256:618c79494d3b65fac75fee08b1c8d945e96242c63235f0a292516c20884f5d92 0.0s done
#7 extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
#7 extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s done
#7 DONE 4.9s
#9 [2/2] COPY --link /local-config/ /config/
#9 ERROR: failed to prepare sha256:b658d8ac8aea7292c464218dbf62c5bcc46468f8b7e017c0380ab37b3f903261 as kcty4fbxag6hngkemyhs5dn82: open /var/lib/docker/overlay2/q32w4cbarh6m1wpr169nbtyib/.tmp-committed2557151495: no such file or directory
------
> [2/2] COPY --link /local-config/ /config/:
------
Dockerfile:4
--------------------
   2 |     FROM registry.com/upstream/otherimage:version
   3 |
   4 | >>> COPY --link /local-config/ /config/
   5 |
   6 |     # Comment
--------------------
ERROR: failed to solve: failed to prepare sha256:b658d8ac8aea7292c464218dbf62c5bcc46468f8b7e017c0380ab37b3f903261 as kcty4fbxag6hngkemyhs5dn82: open /var/lib/docker/overlay2/q32w4cbarh6m1wpr169nbtyib/.tmp-committed2557151495: no such file or directory

The builder is the default builder, with this part in daemon.json:

  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "40GB"
    }
  }

I don't know if it is what is happening, but since this is hard to reproduce, it is possible that the other "docker build" command starts cleaning the cache and that is why the layer is suddenly missing.