[get.jenkins.io] provide a mirror for Jenkins Russian users

[dduportal]

Service(s)

get.jenkins.io

Summary

As per #4128 (comment), the Jenkins Russian users recently went trough troubles when it comes to downloading Jenkins plugins/installers through our mirror system get.jenkins.io:

• The closest mirror, for Russian users, used to be [Aachen University}(https://ftp.halifax.rwth-aachen.de/jenkins/). It recently has been blocked by Roskomnadzor for reasons outside the Jenkins project scope (and ability to act).
• Hopefully, we recently benefited from a new mirror in Romania thanks to Hostico (see our announcement)
• Alas, our friends at Hostico are now receiving a LOT of traffic due to their proximity which is not sustainable at all

As such, this issue is to find a solution (and track its resolution up to completion) from the Jenkins project infrastructure.
The goals are:

• Ensure Hostico can keep sponsoring us sustainably by keeping their traffic under an acceptable (for them!) threshold
• Ensure that Jenkins Russian users have a download mirror, not blocked with acceptable performances
• Ensure that the Jenkins Infrastructure can sustains the cost of any solution here (sponsored, credits, billin, any kind)

Reproduction steps

No response

[dduportal]

First set of options after initial discussions:

• It looks like there is an unofficial Yandex hosted mirror at https://mirror.yandex.ru/mirrors/jenkins/plugins/ => let's contact them to see if we can "simply" (modulo having an rsync or FTP service for scanning). Could be the easiest solution!
• The Jenkins project currently has access to different clouds with sponsored credits (or billing) where a mirror could be managed by the Jenkins infra team with a location close to Russia (at least West Russia) for QoS, and we could set up fake coordinates in the mirror setup inside mirrorbits to force this mirror to be picked when requests comes from West Russia (instead of fall backing to Hostico in Romania)
  • we have an AWS account. Since AWS has DC in Sweden (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html), it could be close enough
  • Azure has a Poland region: https://azure.microsoft.com/en-us/updates/general-availability-microsoft-azure-available-from-new-cloud-region-in-poland/
  • DigitalOcean has a Frankfurt region from where we operate
  • We could also try to start new sponsor accounts: OVH has Poland DCs as well (see https://www.ovhcloud.com/en/vps/vps-poland/)

[mikhirev]

I tried to contact the mirror.yandex team via the official email address, but got no reply. Maybe @w-e-g could directly contact the engineer he mentioned.

[w-e-g]

My friend pinged a man who worked on mirror.yandex.ru, but no answer at all.

[timja]

We've had a response from yandex to the jenkins infra team FYI, they would like to make their mirror an official one.

[dduportal]

We're going to add this mirror to get.jenkins.io today (Monday 24 June 2024) between 01:35pm UTC and 02:00pm UTC (along with #4145)

[lemeurherve]

Yandex mirror has been added with success 🥳

https://get.jenkins.io/debian/jenkins_2.463_all.deb?mirrorstats

Note: we had the same issue as for Hostico (mirror marked as "down", cf #3976 (comment)), resolved after some times.

Not sure what worked, mirrorbits edit wasn't at first, we then deleted and readded the mirror but it was only after seeing files being individually scanned in mirrorbits logs that the mirror appeared as "up".

[Unknown-Guy]

Is this the cause now for:

java.io.IOException: Unable to tunnel through proxy. Proxy returns "HTTP/1.1 403 Forbidden"
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling0(HttpURLConnection.java:2271)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2143)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect0(HttpURLConnection.java:2909)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.followRedirect(HttpURLConnection.java:2818)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1929)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1323)
Caused: java.io.IOException: Failed to load https://updates.jenkins.io/download/plugins/build-timeout/1.33/build-timeout.hpi to /var/lib/jenkins/plugins/build-timeout.jpi.tmp
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1334)
Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/build-timeout/1.33/build-timeout.hpi (redirected to: https://mirror.yandex.ru/mirrors/jenkins/plugins/build-timeout/1.33/build-timeout.hpi)
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1368)
	at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1925)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2237)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1899)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.base/java.lang.Thread.run(Thread.java:1583)

yandex is blocked in a lot of corporate environments.

[dduportal]

yandex is blocked in a lot of corporate environments.

Hi @Unknown-Guy , there isn't a lot we can do on the infra, unless one of these corporations helps by providing a (public) download mirror in the Russia area. As a reminder, the Jenkins project is living on sponsorships: we do not have the financial resource to cover all cases and quality of service unless users helps by sponsoring.

Other alternative is to contact companies such as CloudBees (and pay their product) to ensure this kind of constraint is solved for your company.

[Unknown-Guy]

I don't need the yandex mirror at all, we are in EU, it seems, after this change, closest mirror is yandex one which is blocked, some countries even block yandex (Finland and Norway come to mind). So this is breaking change.

[Stikus]

Looks like this is vice versa situation.

@dduportal It is time to add mirror selection preferences on Jenkins end, what do you think?

[QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4]

Maybe Jenkins team just could use CountryOnly option on mirrorbits, to limit Yandex to RU?

That would be (IMO) the easiest solution.

Ref:

• https://github.com/etix/mirrorbits/blob/53725c9d3ba2e676ed40a9d35dd7b0dc9d858817/cli/commands.go#L264
• https://github.com/etix/mirrorbits/blob/53725c9d3ba2e676ed40a9d35dd7b0dc9d858817/mirrors/mirrors.go#L36
• https://github.com/etix/mirrorbits/blob/53725c9d3ba2e676ed40a9d35dd7b0dc9d858817/http/selection.go#L96

[QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4]

@lemeurherve I've seen you comment come and go, but in case the CountryOnly code *is set*, yet EU users are routed to Yandex, maybe the CountryCodes were incorrectly detected during "add" operation?

Something along the following code-path - https://github.com/etix/mirrorbits/blob/master/rpc/rpc.go#L329-L358

[Stikus]

@QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4 This addition will fix current problem (for some EU countries), but will lead for new problems:

• yandex mirror will still be best for some countries, but they cannot reach it due to CountryCodes blocking
• hostico traffic will increase and this was a problem (as stated in the beginning of the topic)
• some other mirrors can be blocked for various reasons - it has to be setting to add preferable mirror for Jenkins server admin

I think we need another issue for this discussion, what do you think?

[lemeurherve]

I've seen you comment come and go

Yes sorry I deleted my comment which was wrong, this CountryOnly setting is currently not set.
I should have edited my comment instead of deleting it.

[dduportal]

Hi folks, thanks for the pointers and details.

• @Unknown-Guy thanks for commenting and detailing after my comment that you are in Europe (but close to the Yandex mirror). This is not a breaking change because this is an open source project living by sponsoring and there are no rules of engagement as every contributor is doing their best. Don't get me wrong: we understand and acknowledge the issue now you've pointed that you are in EU (I did not know you were with your first message): we are going to do something but ETA is next week. We are sorry for the inconvenience.
• @QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4 thanks for the reminder. As @lemeurherve proposed, we should be able to act on this next week, but we need to plan it carefully to avoid problems such as described in [get.jenkins.io] provide a mirror for Jenkins Russian users #4147 (comment)

is time to add mirror selection preferences on Jenkins end, what do you think?

If this is something which look interesting to you, you can raise a feature request on mirrorbits open source software we use: if they implement it then we could benefit from it. However, I don't think it is a wise choice as it would allow anyone to bypass the redirections, which defeat the whole purpose (ref. https://www.sonatype.com/blog/maven-central-and-the-tragedy-of-the-commons). If the goal is to solve local optimums and break the other, then it start to be a problem.
If you have a solution which does not involve weeks of works for the infra contributors, please share it with us we are all hears!

[dduportal]

Reopening as this is a problem to be solved

[dduportal]

Proposal based on the above comments from different users + internal discussions:

• Set Yandex mirror to Russian country only => this mirror would only be selected for Russian users
• All East-Europe traffic will get to Hostico due to this one => if is not sustainable for them, then we'll also restrict them to Romania only
• It means, in the worst case scenario, we'll back to the situation one month ago with traffic going to Germany and/or Belgium (except for Russian and Romania) so still better than before

Any objection of thing to consider if we set up this change next week?

[Stikus]

@dduportal Can you describe what Jenkins will do if first mirror picked is not reachable for some reason? Will it fail or pick second mirror? If it can pick second mirror - it will fix all these problems, don't you think so? Or it is hard to implement? (I'm not familiar with mirrorbits, sorry)

[dduportal]

@dduportal Can you describe what Jenkins will do if first mirror picked is not reachable for some reason? Will it fail or pick second mirror? If it can pick second mirror - it will fix all these problems, don't you think so? Or it is hard to implement? (I'm not familiar with mirrorbits, sorry)

No problem, this is a good question. Mirrorbits only emits an HTTP 3xx redirection, with the Location header set to the mirror URL closest to you. There is no dynamic reverse proxying so we cannot do such feature.

the goal for it is to be simple to defer traffic to mirrors (handling only http/3xx request does not cost much). So it can handle a lot of traffic.

[dduportal]

Proposal based on the above comments from different users + internal discussions:

* Set Yandex mirror to Russian country only => this mirror would only be selected for Russian users

* All East-Europe traffic will get to Hostico due to this one => if is not sustainable for them, then we'll also restrict them to Romania only

* It means, in the worst case scenario, we'll back to the situation one month ago with traffic going to Germany and/or Belgium (except for Russian and Romania) so still better than before

Any objection of thing to consider if we set up this change next week?

We are starting the operation

[dduportal]

Proposal based on the above comments from different users + internal discussions:

* Set Yandex mirror to Russian country only => this mirror would only be selected for Russian users

* All East-Europe traffic will get to Hostico due to this one => if is not sustainable for them, then we'll also restrict them to Romania only

* It means, in the worst case scenario, we'll back to the situation one month ago with traffic going to Germany and/or Belgium (except for Russian and Romania) so still better than before

Any objection of thing to consider if we set up this change next week?

We are starting the operation

Operation finished: both Yandex and Hostico mirrors are now set up as "Country only".
It means than these mirrors will only be selected to serve traffic originating from the country on which they are marked from (Russia and Romania).

You can see this result in https://get.jenkins.io/war/2.464/jenkins.war?mirrorlist as example:

[dduportal]

Ping @anonimus-asuseeepc (ref. #4160) @QMiqTx6DHn1bA9yaNaAbsD3CLG8gTmd4 @Stikus @Unknown-Guy could you confirm if it fixes your problems?

[Unknown-Guy]

It seems all is good now, our Jenkins updated all plugins without issues.

[dduportal]

It seems all is good now, our Jenkins updated all plugins without issues.

Many thanks!