Blocked from APT repository by CloudFront

[remram44]

On multiple occasions over the past week, for multiple IP addresses, I am getting an error message from CloudFront when I try to use the APT repository.

Example Sunday:

Again today:

text for searchability purposes

403 ERROR

The request could not be satisfied.

Request blocked. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Generated by cloudfront (CloudFront)

The specific URL: https://pkgs.k8s.io/core:/stable:/v1.27/deb/InRelease (302 redirect to https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.27/deb/InRelease which serves the page above with status 403)

This prevents me from deploying Kubernetes on nodes via Ansible and will probably impact cluster upgrades. This is a 40-node cluster at New York University that is behind NAT. The NAT uses multiple IP addresses but they are all getting blocked really quickly.

Example IPs: 216.165.12.8, 216.165.12.11

We never had this problem with the Google repository, only the so-called "community" repository.

Related:

• Enable WAF for prod-cdn.packages.k8s.io CloudFront distribution #5617
• Community-Owned package repositories fail with 403 if no user-agent set release#3261
• Error installing from official repos onto Ubuntu release#3219

[ameukam]

cc @kubernetes/release-engineering

[ameukam]

/area infra/aws
/area infra
/sig release
/area release/rel-eng

[k8s-ci-robot]

@ameukam: The label(s) area/release/rel-eng cannot be applied, because the repository doesn't have them.

In response to this:

/area infra/aws
/area infra
/sig release
/area release/rel-eng

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[saschagrunert]

@kubernetes/sig-k8s-infra can we tweak the cloudfront config somehow to avoid issues like that?

[xmudrii]

The infra for pkgs.k8s.io is maintained by SIG Release. I searched WAF logs for provided IP addresses and I was able to find that your requests are being blocked by the AWS-AWSManagedRulesAmazonIpReputationList (AWS#AWSManagedRulesAmazonIpReputationList#AWSManagedIPReputationList) rule when trying to access /repositories/isv:/kubernetes:/core:/stable:/v1.27/deb/InRelease

The AWS doc on WAF rules says that this is because your IP address has been identified as a bot and is blocked because of that. Do you have some other workload that might, for example, scrape resources hosted on AWS? Just trying to download packages, even if it's multiple nodes behind a NAT, shouldn't really trigger this. However, we should check with the AWS folks how to prevent this and what are recommended next steps (this is mostly like going to wait the next week because a lot of folks are on holidays).

I'll take a look into this.
/assign

[remram44]

NYU is a research university. Our students and faculty conduct all sort of research activities out of our compute infrastructure. If that can lead to our university being blocked from accessing your package repository, maybe you should consider whether this reputation/firewall service is really appropriate for a "community" repository? For an opensource project?

Is there any way to know what/when/who would have angered the AWS algorithms that apparently gate-keep the access to Kubernetes releases? Is there any recourse for the people affected short of "abide to the unwritten rules of AWS"?

I am sure you understand how much more community-friendly the Google-operated repository now seems to be, compared to this commercial repository that you call "community" but is apparently entirely gated by the goodwill of the Amazon corporation.

[remram44]

Is the WAF needed in practice? Are there spam activities that require the WAF to reduce the cost on the repository? I am sure it is enabled for a reason, I am trying to understand. Would it help if we set up our own mirror? Is there a way to do that to remove CloudFront from the path?

[puerco]

@remram44 regarding this:

this commercial repository that you call "community" but is apparently entirely gated by the goodwill of the Amazon corporation

and this comment on #5617:

This seems very community-hostile to be honest. Very disappointing to see this move right after launching this supposedly "community" repository.

This is not a "supposed" community repository, this is infrastructure funded by donations, managed and budgeted by community volunteers. The project has limited resources and we try to shield them from any potential abuse.

We want to accommodate any fair use of our artifacts and some of the traffic patterns of the repos are just becoming apparent as the traffic has now begun pounding the new repositories. This means that now that we are uncovering edge cases like this one, we would appreciate your help understanding the traffic coming from your IP ranges to help us decide how to serve you better. The WAF configuration is public if you want to help us debug and/or suggest improvements.

[upodroid]

https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html

In addition to what @puerco mentioned:

1. If you are an AWS customer at NYU, please reach out to Amazon and ask them to remove the NYU IP space from the filters described on this website. https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html
2. Is the WAF needed in practice? Yes. This is to block malicious users from abusing our services. Unfortunately, a small number of IPs do get incorrectly classified by our cloud vendors and they have a process for fixing that.
3. Would it help if we set up our own mirror? This is not possible, the origin is a private S3 bucket that can only be reached via CloudFront
4. Is there a way to do that to remove CloudFront from the path? No. The origin is an S3 bucket and it is very expensive to serve the traffic directly from the bucket.
5. https://www.abuseipdb.com/check-block/216.165.12.0/24 both IPs have been reported recently for abusive behavior

[xmudrii]

Would it help if we set up our own mirror? This is not possible, the origin is a private S3 bucket that can only be reached via CloudFront.

If they have a machine that's outside that subnet and not blocked by AWS/CloudFront, setting up their own mirror would be the best solution at this time for this particular problem. They can use apt-mirror to mirror pkgs.k8s.io to their own infrastructure and then point nodes to use that mirror.

[thoms27]

Hello,

I have the same issue since fews days or weeks from all ips in 5.149.176.0/20 range.
We don't know why ... How can I contact the AWS support in charge of WAF ?

Thank you in advance.

Regards

[xmudrii]

/unassign

[BenTheElder]

In addition to the comments above #6106 (comment) #6106 (comment) #6106 (comment)

NYU is a research university. Our students and faculty conduct all sort of research activities out of our compute infrastructure. If that can lead to our university being blocked from accessing your package repository, maybe you should consider whether this reputation/firewall service is really appropriate for a "community" repository? For an opensource project?

It is very common for universities to host package mirrors for OSS packages, I would recommend considering this here.

We don't know why ... How can I contact the AWS support in charge of WAF ?

I'm sorry, we can't answer this.