-
Notifications
You must be signed in to change notification settings - Fork 633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GHA cache image build dependencies #3583
Conversation
4fd4147
to
ed7df0b
Compare
@@ -69,6 +69,7 @@ RUN xx-apt-get update -qq && xx-apt-get install -qq --no-install-recommends \ | |||
libbtrfs-dev \ | |||
libseccomp-dev \ | |||
pkg-config | |||
RUN git config --global advice.detachedHead false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Silence useless / noisy git messages.
Dockerfile
Outdated
|
||
FROM build-base AS build-minimal | ||
RUN BINDIR=/out/bin make binaries install | ||
# We do not set CMD to `go test` here, because it requires systemd | ||
|
||
FROM build-base AS build-full | ||
FROM build-base AS build-deps |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the part that splits out third-party dependencies from nerdctl.
ed7df0b
to
3215ecc
Compare
@AkihiroSuda @djdongjin this should not go in before #3535, but this looks really promising in term of minimizing transactions with Hub / third party services. LMK your thoughts overall and I will polish it and rebase on top of ^ after it merges |
3215ecc
to
d38e4b5
Compare
ARG TARGETARCH | ||
ENV GOARCH=${TARGETARCH} | ||
RUN BINDIR=/out/bin make binaries install |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move down nerdctl, which is the part that changes (almost) all the time.
@@ -181,13 +176,6 @@ RUN git clone https://github.com/containerd/imgcrypt.git /go/src/github.com/cont | |||
git checkout "${IMGCRYPT_VERSION}" && \ | |||
CGO_ENABLED=0 make && DESTDIR=/out make install && \ | |||
echo "- imgcrypt: ${IMGCRYPT_VERSION}" >> /out/share/doc/nerdctl-full/README.md | |||
ARG ROOTLESSKIT_VERSION |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move down rootless kit, as we support two versions.
@@ -237,6 +232,14 @@ RUN echo "" >> /out/share/doc/nerdctl-full/README.md && \ | |||
mv /tmp/SHA256SUMS /out/share/doc/nerdctl-full/SHA256SUMS && \ | |||
chown -R 0:0 /out | |||
|
|||
FROM build-dependencies AS build-full |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Restore nerdctl from above.
@@ -56,7 +99,7 @@ jobs: | |||
run: make test-unit | |||
|
|||
test-integration: | |||
timeout-minutes: 60 | |||
timeout-minutes: 30 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lower timeouts.
@@ -262,11 +344,6 @@ jobs: | |||
go-version: ${{ env.GO_VERSION }} | |||
cache: true | |||
check-latest: true | |||
- name: "Print docker info" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debugging remnants.
steps: | ||
- uses: actions/[email protected] | ||
with: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Harmonize
d38e4b5
to
b187986
Compare
@AkihiroSuda at your convenience. Would like to get this is in for a few PRs to better evaluate how the cache behaves. |
.github/workflows/test.yml
Outdated
run: | | ||
sudo mkdir -p /etc/docker | ||
echo '{"ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "experimental": true, "ip6tables": true}' | sudo tee /etc/docker/daemon.json | ||
echo '{"features": {"containerd-snapshotter": true}, "ipv6": true, "fixed-cidr-v6": "2001:db8:1::/64", "experimental": true, "ip6tables": true}' | sudo tee /etc/docker/daemon.json |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
containerd-snapshotter
isn't necessary if you use docker buildx
? (probably with docker buildx create
to create a standalone non-moby buildkitd instance)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting.
That would certainly be better.
Let me look into that later today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks for the tips.
aa5862c
to
5d836d6
Compare
5d836d6
to
8dc03b7
Compare
.github/workflows/test.yml
Outdated
fetch-depth: 1 | ||
- name: "Expose GitHub Runtime variables for gha" | ||
uses: crazy-max/ghaction-github-runtime@v3 | ||
- name: "Enable containerd to be able to use gha cache" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not containerd
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. Thanks.
Removed the separate step altogether with the latest push.
8dc03b7
to
3b1d250
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
@AkihiroSuda failure is likely #3556 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, sorry for slow review.
Signed-off-by: apostasie <[email protected]>
3b1d250
to
de193b3
Compare
No problem @ktock. |
While working on something else, I am quickly getting 429s from gha cache, and jobs failing / timeouting because of throttled requests. I am not optimistic about this overall. Suggesting we give it a couple of weeks and consider reverting this / tweak it / going back to other ideas (eg: proxy caching). |
Folllow-on to discussion in #3580
What this does:
The result is:
The key benefit here is about reducing the network traffic required to produce out test images (hence reducing the opportunity for failure due to third-party server hiccups). It is similar in intent to #3580 .
Incidentally, we will also get a small speed boost for the overall run.
Obviously, GHA cache is not "free": it takes time to retrieve and time to store - so, part of the speed gains from not-building are negated by the cache r/w.
Nevertheless, this looks promising for increased reliability (and reduced transactions with docker hub / debian / ubuntu).
This PR also has a couple of minor changes to the workflow file (reduced timeouts, cosmetic comments, along with bumping the size of the arm64 instance as previously discussed). If preferable, I can split these out.
Further refactoring / changes to the Dockerfile could bring more stuff in the dependencies stage.
This PR has been conservative on that front and staid with the minimal possible changes to the Dockerfile, so that we can decide separately if we want a more in-depth restructuring of it or not.
Finally note that GHA cache is rather limited (10G), and going over the limit will prune prior entries indiscriminately - we might want to keep an eye on that and check that this proposed implementation here stays under the limit to fully benefit from it.