tdnf does not set a user agent when doing http requests

[chrischdi]

Describe the bug

The kubernetes-sigs project image-builder tries to switch the used repositories to the new upstream repository at pkgs.k8s.io (via this PR: kubernetes-sigs/image-builder#1280 ).

However, the installation or to be more precise tdnf makecache fails when using that repository with a 403 http error.

It turns out that tdnf does not set any user agent in its requests.

Note: I was also not able to find a possibility to configure the user agent for tdnf.

Reproduction steps

1. Run photon in a container docker run -ti --rm photon:latest

2. Create the repo entry:

   cat << EOF > /etc/yum.repos.d/kubernetes.repo
   [kubernetes]
   name=kubernetes
   description=the kubernetes yum repo
   baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
   gpgcheck=True
   gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
   enabled=1
   
   EOF
   

3. Try to update the caches: tdnf install kubelet-1.26.7 --nogpgcheck -y

Output:

Refreshing metadata for: 'kubernetes'
kubernetes                                 919 100%
Error: 403 when downloading https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml
. Please check repo url or refresh metadata with 'tdnf makecache'.
Error(1622) : Invalid argument
Error: Failed to synchronize cache for repo 'kubernetes'
Error(1622) : Invalid argument

Expected behavior

tdnf makecache to succeed.

Additional context

According to slack, this happens at the repository because they use AWS AWF, which blocks requests which have an empty/not set user agent.

After analysing the request using mitmdump I was able to see that tdnf does not set any request, when requesting: https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml

It is reproducible using curl:

curl -L -H 'User-Agent:' https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml

xref: kubernetes-sigs/image-builder#1280 (comment)

Slack thread with the maintainers of the repository: https://kubernetes.slack.com/archives/C03U7N0VCGK/p1694194051373989

[chrischdi]

Note: they changed the pkgs.k8s.io infrastructure to now allow empty user agents.

However, I think this would be still a valuable feature to implement.

[oliverkurth]

Interesting. If just setting an empty User-Agent header is enough to make kubernetes-sigs happy then that sounds like an issue with the latter - it doesn't get any information from it. I am glad they fixed it.

But I don't see any issue with adding that header to tdnf.

[chrischdi]

Yeah, it is a default settings to get blocked if you are hosting behind AWS WAF https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-baseline.html

And I think at least having the option to configure it is good.

[oliverkurth]

The header can be anything, right? So I think if we add it, then with something useful - like mentioning the version of tdnf, and maybe the OS and version. I don't think it needs to be configurable.

[oliverkurth]

This has been fixed a while ago with #453 . I forgot to close this.