Install Kubernetes artifacts from pkgs.k8s.io

[mboersma]

What this PR does / why we need it:

Updates image-builder to use pkgs.k8s.io for Kubernetes artifacts, according to this guidance: https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/

Which issue(s) this PR fixes:

Fixes #1278

Additional context

[mboersma]

/retitle [WIP] Install Kubernetes artifacts from pkgs.k8s.io

[mboersma]

/test pull-azure-sigs

[mboersma]

Photon is failing, but I'm not sure what to make of the error.

�[0;32m    vsphere-clone:�[0m
�[0;32m    vsphere-clone: TASK [kubernetes : Install Kubernetes] *****************************************�[0m
�[0;32m    vsphere-clone: fatal: [default]: FAILED! => {"changed": true, "cmd": ["tdnf", "install", "kubelet-1.26.7", "kubeadm-1.26.7", "kubectl-1.26.7", "kubernetes-cni-1.2.0", "--nogpgcheck", "-y"], "delta": "0:00:00.265955", "end": "2023-09-01 23:18:01.328092", "msg": "non-zero return code", "rc": 86, "start": "2023-09-01 23:18:01.062137", "stderr": "Error: 403 when downloading https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml\n. Please check repo url or refresh metadata with 'tdnf makecache'.\nError(1622) : Invalid argument\nError: Failed to synchronize cache for repo 'kubernetes'\nError(1622) : Invalid argument", "stderr_lines": ["Error: 403 when downloading https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml", ". Please check repo url or refresh metadata with 'tdnf makecache'.", "Error(1622) : Invalid argument", "Error: Failed to synchronize cache for repo 'kubernetes'", "Error(1622) : Invalid argument"], "stdout": "Refreshing metadata for: 'kubernetes'", "stdout_lines": ["Refreshing metadata for: 'kubernetes'"]}�[0m

[AverageMarcus]

Could it be a flake? Not sure why it'd be getting a 403 😕

/retest

[AverageMarcus]

I guess not 🤔

I did find this issue but no solution: kubernetes/release#3219

[AverageMarcus]

I'm able to get the same error when running photon:5.0 in a container and following the instructions in the blog post. I suspect tdnf is sending some extra headers or payload that the repo doesn't like but I can't figure out how to see more info about what it's doing.

[mboersma]

Thanks @AverageMarcus–I agree that #3219 looks like the same thing similar. Unfortunately it doesn't seem there's been any progress on that issue.

@kubernetes-sigs/cluster-api-provider-vsphere-maintainers this PR is currently stuck on the VSphere Photon builds. Just FYI, and in case any of you have ideas for a workaround.

[sbueringer]

@mboersma Sorry about that, thx for the mention.

@chrischdi Do you have some time to investigate? Could it potentially be related to your recent PR? #1288

[mboersma]

As another data point, the Mariner Linux build for Azure seems to work fine with the new pkgs.k8s.io repository, and Mariner is also a tdnf-based distro.

The fact that the error is 403 makes me think similarly to @AverageMarcus:

I suspect tdnf is sending some extra headers or payload that the repo doesn't like

[sbueringer]

/assign @kkeshavamurthy

[chrischdi]

I tried to reproduce it and I am able to reproduce it locally:

docker run -ti --rm photon:latest

Inside the container

cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=kubernetes
description=the kubernetes yum repo
baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
gpgcheck=True
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
enabled=1

EOF

tdnf install kubelet-1.26.7 --nogpgcheck -y

Output:

Refreshing metadata for: 'kubernetes'
kubernetes                                 919 100%
Error: 403 when downloading https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml
. Please check repo url or refresh metadata with 'tdnf makecache'.
Error(1622) : Invalid argument
Error: Failed to synchronize cache for repo 'kubernetes'
Error(1622) : Invalid argument

[chrischdi]

I debugged it using mitmdump. The reason for getting 403 is that tdnf does not set a user agent.

It is reproducible using curl and setting the user agent to empty:

curl -L -H 'User-Agent:' https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: UbCX8uy59GTtjMeZj_HN-cL6o9ng6qnBPLByOF3OeczliuSY-TiGQA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

[AverageMarcus]

Oh nice work @chrischdi!

Are we able to configure tdnf to provide a user-agent? EDIT: Just seen your issue (vmware/tdnf#451) where you state no flag to configure it.

Should we also raise this as an issue upstream? Would be good if others didn't have to work around this too.

[AverageMarcus]

I've opened an issue upstream - kubernetes/release#3261

I've included the reproducible steps from this issue, thanks Christian!

[chrischdi]

Awesome, thank you @AverageMarcus !

[AverageMarcus]

Hopefully it's just a checkbox or something that can be toggled in CloudFront to fix it 🤞 😅

[chrischdi]

Seems to be configured on puprose/default at AWS WAF. (via https://kubernetes.slack.com/archives/C03U7N0VCGK/p1694194051373989 )

[chrischdi]

/test pull-ova-all

pkgs.k8s.io should now allow empty user agents.

[AverageMarcus]

Upstream fix for reference: kubernetes/k8s.io#5823

[AverageMarcus]

🎉 Proton now passing! 😁

@mboersma Anything outstanding on this PR? Can the [WIP] now be removed?

[sbueringer]

Nice, thx folks for working on this!

[mboersma]

/retitle Install Kubernetes artifacts from pkgs.k8s.io

Thanks so much @chrischdi and @sbueringer and @AverageMarcus! I think this is ready for review then.

[AverageMarcus]

/approve

🎉 Nice work!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AverageMarcus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [AverageMarcus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AverageMarcus]

Oh, needs LGTM too

/lgtm

[AverageMarcus]

🤦 Accidentally clicked the wrong button. Sorry about that, need to wait for the tests again 😞