Switch to repos to maintaned pkgs.k8s.io #657
Conversation
deric
force-pushed
the
pkgs
branch
2 times, most recently
from
db17097
to
53b786c
Compare
deric
force-pushed
the
pkgs
branch
2 times, most recently
from
fcd52f8
to
eb3f67c
Compare
|
The apt source must have following definition: Because otherwise we'd get following errors (as CDN is being used): The absence of The param apt::source { 'kubernetes':
...
key => {}
}and apt::source { 'kubernetes':
...
keyring => '',
}I'm not sure if we can write this in a better, backward compatible way 🤔 |
The |
|
Thanks for working on this, I am also interested in having it merged as soon as possible. For RedHat based OS, would it be possible to also add Like in the official example provided here: https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/#how-to-migrate-rpm Thank you |
As far as I can find and understand it, this excludes updates or installs of these packages from this repository. It does not really make sense to me, however, as this would be the main packages you want to install from this repo or what am I missing? |
|
Yeah disregard my previous comment. My idea was to exclude them from the repo definition so they won't be accidentally upgraded during a regular OS update. But it seems that we exclude them there, puppet cannot installed them either. Not sure why it is part of the configuration example... |
|
@danifr You might be able to use something like is that the regular update could upgrade only to next patch version. Which also not ideal because the upgrade should be applied to control-plane nodes first. That's probably the reason why k8s packages are excluded, and when performing manual upgrade you should do this: |
|
I think the recommendation by Kubernetes to exclude the Kubernetes packages is counterintuitive when configuration management is involved. To use the excludes the only real option with Puppet is |
Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream.
|
It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up. |
As announced back in August, the legacy repos have been frozen for month now and all Kubernetes version that where available through the old repos are long end of life now. As the end of February has passed the old repos are no longer available and if you are still running any version that was available in the old repos you should be looking into upgrading to a supported version as soon as possible. [1] https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/ |
| kubernetes_version => '1.28.2', | ||
| kubernetes_package_version => '1.28.2-1.1', |
There was a problem hiding this comment.
As 1.29.2 is already out by now this probably should get updated to that new version.
There was a problem hiding this comment.
Sure, this is just version for tests. The point was to update the version, that is available in the new repos. We should definitely update the default version, because the current value is 1.10.2.
| # For pkgs.k8s.io use GPG siging key | ||
| $_keyring = '/usr/share/keyrings/kubernetes-apt-keyring.gpg' | ||
| # TODO: Switch to apt::keyring once supported by puppetlabs-apt | ||
| # see: https://github.com/puppetlabs/puppetlabs-apt/pull/1128 | ||
| archive { '/tmp/kubernetes-apt-keyring.gpg': | ||
| source => "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key", | ||
| extract => true, | ||
| extract_path => '/usr/share/keyrings', | ||
| extract_command => 'gpg --dearmor < %s > kubernetes-apt-keyring.gpg', | ||
| creates => $_keyring, | ||
| } | ||
|
|
||
| Apt::Source<| title == 'kubernetes' |> { | ||
| keyring => $_keyring, | ||
| require => Archive['/tmp/kubernetes-apt-keyring.gpg'], | ||
| } |
There was a problem hiding this comment.
The support for modern keyrings has been released in puppetlabs-apt since three month now in version 9.2. Version 9.4 is also already out. To me, it would make sense to switch to apt::keyring, but if you would like to keep it like this for compatibility for now this would also be fine for me.
The most important for me would be that this PR finally get's merged and there is a release of puppetlabs-kubernetes that has useful, modern defaults and works with the newest version of Kubernetes out of the box.
There was a problem hiding this comment.
As I wrote earlier, it would be very nice to switch to modern apt::keyring. But we should make incremental changes. Adding dependency on puppetlabs-apt >= 9.2.0 means effectively bumping dependency on puppetlabs-stdlib from >= 4.25.0 to >= 9.0.0. This is rather huge step for the mankind, especially done in a single innocent PR called "Swich repos...".
Depending on puppet code base setup this might mean upgrading many (10 or 40) modules just because of this minor change. I would prefer to release this "old but working version" as a 8.1.0 version. And then change module dependencies in separate PR, while bumping the module version e.g. to 9.0.0.
There was a problem hiding this comment.
As mentioned in #681, the actual puppetlabs/stdlib requirement is currently >= 8.4.0. So the change wouldn't be that huge.
spec/classes/repos_spec.rb
Outdated
| full: '16.04' | ||
| }, | ||
| distro: { | ||
| codename: 'xenial' |
There was a problem hiding this comment.
This should perhaps also get updated to a newer version like 22.04, as 24.04 will soon be released, 16.04 is long EOL now and it doesn't help me at all, if it works on some ancient release, but not a current one.
There was a problem hiding this comment.
This has nothing to do with newer distributions. Google was releasing all packages under xenial codename because the binary packages weren't dependent any specific distribution.
There was a problem hiding this comment.
Yes, I know that everything was released under the kubernetes-xenial distribution (release in puppet), but I am not completely mistaken this is what got removed as default in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-b27acae942df5d6ec2b58798add49fff0f625bec33093d74ea3cb1ac39fd0c6aL69, https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L26 and tested against in https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L50
but the standalone xenial should refer to the distro, not the kubernetes-repo.
spec/classes/repos_spec.rb
Outdated
| 'kubernetes_yum_baseurl' => 'https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64', | ||
| 'kubernetes_yum_gpgkey' => 'https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg', | ||
| 'docker_apt_location' => 'https://download.docker.com/linux/ubuntu', | ||
| 'docker_apt_release' => 'xenial', | ||
| 'docker_apt_repos' => 'main', | ||
| 'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable', | ||
| 'docker_yum_gpgkey' => 'https://download.docker.com/linux/centos/gpg', |
There was a problem hiding this comment.
Same here: CentOS 7 / RHEL 7 will soon be EOL - CentOS Stream 9 / RHEL 9 have been out for almost 2 years now. For sure it would be good, if it still works on older releases, but people should be moving away from them anyway and ensuring that it works on newer versions is crucial to not hold people back and force them to stick with increasingly insecure OS versions.
There was a problem hiding this comment.
Sorry, I've copy pasted those tests. But these test are garbage in & garbage out tests. It doesn't actually test these distributions. It only tests string getting passed to the class, that it gets propagated. Actual tests should be written in acceptance tests.
There was a problem hiding this comment.
Yeah sure, you take some (more or less arbitrary base os) and then test, if 1. If you don't specify a specific repository, it now correctly defaults to the new repo and 2. If you override it with something (e.g. the old repo), it still includes the specified repo as desired, to e.g. still use the old repo for whatever reason.
As this could in theory be OS version dependent, this should in my opinion be tested on one recent release (Ubuntu 22.04 (as you changed it now - thank you!) or Debian 12 and CentOS Stream 9 / RHEL 9) and optionally additionally on some older version as well.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R5-R70 does 1. as far as I understand it: when testing on (now) Ubuntu 22.04 with Kubernetes version 1.28, no additional kubernetes_apt_* and for whatever reason questionable kubernetes_yum_* config, it correctly set's the apt source to the new 1.28 repo.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R71-R132 does 2., testing that on a Debian-based distro when overriding the repo, it set's it to the override.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114R195-R246 tests 1., that on CentOS 7 the new default works.
https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 tests 2., that the override works for CentOS 7, while still configuring some questionable apt stuff, that should be absolutely irrelevant on dnf-based distros.
And finally https://github.com/puppetlabs/puppetlabs-kubernetes/pull/657/files#diff-6ff1ce06229c83037f537d2d0cb33aeba339e8fe801238cfb51c2f538c01c114L127-R297 does 2. again on CentOS 7, but now with packaged containerd instead of docker archives.
| operatingsystemrelease: '7.0', | ||
| kernel: 'Linux', | ||
| os: { | ||
| family: 'RedHat', | ||
| name: 'RedHat', | ||
| release: { | ||
| full: '7.0' | ||
| } | ||
| } | ||
| } | ||
| end | ||
|
|
||
| let(:params) do | ||
| { | ||
| 'container_runtime' => 'docker', | ||
| 'kubernetes_version' => '1.28.1', | ||
| 'kubernetes_apt_location' => '', | ||
| 'kubernetes_apt_release' => '', | ||
| 'kubernetes_apt_repos' => '', | ||
| 'kubernetes_key_id' => '', | ||
| 'kubernetes_key_source' => '', | ||
| 'kubernetes_yum_baseurl' => '', | ||
| 'kubernetes_yum_gpgkey' => '', | ||
| 'docker_apt_location' => 'https://download.docker.com/linux/ubuntu', | ||
| 'docker_apt_release' => 'xenial', | ||
| 'docker_apt_repos' => 'main', | ||
| 'docker_yum_baseurl' => 'https://download.docker.com/linux/centos/7/x86_64/stable', |
|
Just wanted to add a couple of notes --
I could not track down specifically what I might have been doing to cause this, if anything, but I just wanted to mention it. It also did not occur on ALL of our hosts using this module -- so there's a good chance it's something we caused. None-the-less, it does not occur in 8.0.0 so I wanted to bring it up not as a problem with this PR, but as a note. I believe your fork only has work-since-8.0.0 plus pkgs changes. I can share the hiera method of working around this issue with 8.0.0 if anyone needs it. |
@jadestorm from looking at the code something like setting in the hiera should work (given that your containerd, systemd, etc. config is updated already) , but haven't tested it directly. |
Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream. (cherry picked from commit 07acbc1)
Node this is a cherry pick of certain parts of puppetlabs#657 This is stalled due to yum issues and so doing this quick fix for now to get us going. Once above is merged we can switch back to upstream. (cherry picked from commit 07acbc1) Change-Id: I452a7e087cda5bb0089b5faa7b14c6c633ed1497
|
FYI for folks blocked, my workaround on Ubuntu 24.04 was: |
Summary
New Kubernetes releases will be published to community owned repost
pkgs.k8s.ioinstead of Google hosted repos.v1.23available only inpackages.cloud.google.comv1.24tov1.27can be installed from both reposv1.28onward should be installed frompkgs.k8s.ioAdded tests to ensure backwards compatibility.
Related Issues (if any)
Checklist
puppet apply)