[QUESTION] How to debug image patching issues

[VivekSubr]

What is your question?

We are encountering buildx failures often when trying to patch images with copa, like this,

2024-12-06T05:10:35.9707882Z + /mnt/vss/_work/1/ws/cna-istio/.build/copa patch --debug --timeout 60m -i a4oprodaf.a4opacketcore.microsoft.com/rel_build_docker/install-cni:temp-patch -r /mnt/vss/_work/1/ws/cna-istio/interface/scripts/copa/cni.json -t 1.19.0.1 -a buildx://copademo
2024-12-06T05:10:45.2230553Z time="2024-12-06T05:10:45Z" level=debug msg="updates to apply: &{{{ubuntu 22.04} {amd64}} [{bash 5.1-6ubuntu1 5.1-6ubuntu1.1 CVE-2022-3715} {bsdextrautils 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {bsdutils 1:2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {curl 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.14 CVE-2023-38545} {curl 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.15 CVE-2023-46218} {curl 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.16 CVE-2024-2398} {curl 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.17 CVE-2024-7264} {curl 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.14 CVE-2023-38546} {libblkid1 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.4 CVE-2023-4911} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-5156} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.7 CVE-2024-2961} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33599} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33600} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33601} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33602} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-4806} {libc-bin 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-4813} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.4 CVE-2023-4911} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-5156} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.7 CVE-2024-2961} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33599} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33600} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33601} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.8 CVE-2024-33602} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-4806} {libc6 2.35-0ubuntu3.1 2.35-0ubuntu3.5 CVE-2023-4813} {libcurl4 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.14 CVE-2023-38545} {libcurl4 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.15 CVE-2023-46218} {libcurl4 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.16 CVE-2024-2398} {libcurl4 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.17 CVE-2024-7264} {libcurl4 7.81.0-1ubuntu1.13 7.81.0-1ubuntu1.14 CVE-2023-38546} {libgnutls30 3.7.3-4ubuntu1.2 3.7.3-4ubuntu1.3 CVE-2023-5981} {libgnutls30 3.7.3-4ubuntu1.2 3.7.3-4ubuntu1.4 CVE-2024-0553} {libgnutls30 3.7.3-4ubuntu1.2 3.7.3-4ubuntu1.4 CVE-2024-0567} {libgnutls30 3.7.3-4ubuntu1.2 3.7.3-4ubuntu1.5 CVE-2024-28834} {libgnutls30 3.7.3-4ubuntu1.2 3.7.3-4ubuntu1.5 CVE-2024-28835} {libgssapi-krb5-2 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.3 CVE-2023-36054} {libgssapi-krb5-2 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37370} {libgssapi-krb5-2 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37371} {libk5crypto3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.3 CVE-2023-36054} {libk5crypto3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37370} {libk5crypto3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37371} {libkrb5-3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.3 CVE-2023-36054} {libkrb5-3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37370} {libkrb5-3 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37371} {libkrb5support0 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.3 CVE-2023-36054} {libkrb5support0 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37370} {libkrb5support0 1.19.2-2ubuntu0.2 1.19.2-2ubuntu0.4 CVE-2024-37371} {libldap-2.5-0 2.5.14+dfsg-0ubuntu0.22.04.2 2.5.16+dfsg-0ubuntu0.22.04.2 CVE-2023-2953} {libmount1 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {libnghttp2-14 1.43.0-1build3 1.43.0-1ubuntu0.1 CVE-2023-44487} {libnghttp2-14 1.43.0-1build3 1.43.0-1ubuntu0.2 CVE-2024-28182} {libpam-modules 1.4.0-11ubuntu2.3 1.4.0-11ubuntu2.4 CVE-2024-22365} {libpam-modules-bin 1.4.0-11ubuntu2.3 1.4.0-11ubuntu2.4 CVE-2024-22365} {libpam-runtime 1.4.0-11ubuntu2.3 1.4.0-11ubuntu2.4 CVE-2024-22365} {libpam0g 1.4.0-11ubuntu2.3 1.4.0-11ubuntu2.4 CVE-2024-22365} {libprocps8 2:3.3.17-6ubuntu2 2:3.3.17-6ubuntu2.1 CVE-2023-4016} {libsmartcols1 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {libssh-4 0.9.6-2ubuntu0.22.04.1 0.9.6-2ubuntu0.22.04.2 CVE-2023-48795} {libssh-4 0.9.6-2ubuntu0.22.04.1 0.9.6-2ubuntu0.22.04.3 CVE-2023-6004} {libssh-4 0.9.6-2ubuntu0.22.04.1 0.9.6-2ubuntu0.22.04.3 CVE-2023-6918} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.16 CVE-2022-40735} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-5363} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.18 CVE-2024-6119} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-2975} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-3446} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-3817} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-5678} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-6129} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-6237} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2024-0727} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-2511} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-4603} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-4741} {libssl3 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-5535} {libuuid1 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {login 1:4.8.1-2ubuntu2.1 1:4.8.1-2ubuntu2.2 CVE-2023-4641} {mount 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.16 CVE-2022-40735} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-5363} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.18 CVE-2024-6119} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-2975} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-3446} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.12 CVE-2023-3817} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-5678} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-6129} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2023-6237} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.14 CVE-2024-0727} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-2511} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-4603} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-4741} {openssl 3.0.2-0ubuntu1.10 3.0.2-0ubuntu1.17 CVE-2024-5535} {passwd 1:4.8.1-2ubuntu2.1 1:4.8.1-2ubuntu2.2 CVE-2023-4641} {perl-base 5.34.0-3ubuntu1.2 5.34.0-3ubuntu1.3 CVE-2023-47038} {perl-base 5.34.0-3ubuntu1.2 5.34.0-3ubuntu1.3 CVE-2022-48522} {procps 2:3.3.17-6ubuntu2 2:3.3.17-6ubuntu2.1 CVE-2023-4016} {tar 1.34+dfsg-1ubuntu0.1.22.04.1 1.34+dfsg-1ubuntu0.1.22.04.2 CVE-2023-39804} {util-linux 2.37.2-4ubuntu3 2.37.2-4ubuntu3.3 CVE-2024-28085}]}"
2024-12-06T05:38:22.0700382Z time="2024-12-06T05:38:22Z" level=debug msg="Connect to buildx instance" driver=docker-container endpoint="unix:///var/run/docker.sock" name=copademo0
2024-12-06T06:10:46.2208907Z time="2024-12-06T06:10:46Z" level=error msg="patch exceeded timeout 1h0m0s"
2024-12-06T06:10:46.2209589Z Error: patch exceeded timeout 1h0m0s

It looks like there is an issue with docker buildx actions taken by copa, is there any way to debug more just from copa? Or any way to circumvent this?

[ashnamehrotra]

@VivekSubr are you able to connect to the buildx instance outside of copa? would you also be able to share the image to try to reproduce this issue?