Timeout to ftp.halifax.rwth-aachen.de

[KweezyCode]

Service(s)

get.jenkins.io

Summary

Caused: java.io.IOException: Failed to download from https://updates.jenkins.io/download/plugins/workflow-api/1316.v33eb_726c50b_a_/workflow-api.hpi (redirected to: https://ftp.halifax.rwth-aachen.de/jenkins/plugins/workflow-api/1316.v33eb_726c50b_a_/workflow-api.hpi)

Reproduction steps

location: Russia (Vimpelcom)

[lemeurherve]

Hello @KweezyCode,

You're probably in the same situation as #4096 & #4105, I suggest you to send a mail to [email protected] explaining your issue.

[KweezyCode]

Hello @KweezyCode,

You're probably in the same situation as #4096 & #4105, I suggest you to send a mail to [email protected] explaining your issue.

how could i change mirror? it would be faster anyway

[C-Otto]

I doubt it. I would have responded by now.

[dduportal]

Hello @KweezyCode,
You're probably in the same situation as #4096 & #4105, I suggest you to send a mail to [email protected] explaining your issue.

how could i change mirror? it would be faster anyway

Hi!

• The mirror redirector system does not allow to specify a custom mirror as far as we know. It's using GeoIP location.
• As soon as New Jenkins mirror in Romania by Hostico #3976 is finished, there should be a closer mirror hosted in Romania. Of course, iy you know a company or organization close to your location who would be able to host a mirror (need webserver, a rsync or FTP system, and ~750 Gb of disk), we would be happy to add it!

I'll close the issue as there are no actionnable for the Jenkins infra team:

• You have to contact [email protected] to solve your problem

[Stikus]

@dduportal Correct me if I'm wrong, but if its our ISP problem and we cannot switch ISP we should use any method to change our IP so GeoIP would pick another mirror for us?

I've contacted @C-Otto (thx for fast answer again) - our IP is not blocked by [email protected]

[C-Otto]

If it's a problem caused by your ISP, you should ask them to fix it (and postpone payments until they do).

[mikhirev]

Hi!

This problem is not caused by ISP. The ftp.halifax.rwth-aachen.de server does not respond to any connect attempts from Russian IP addresses. Can you change your GeoIP settings to exclude this server for requests originating from Russia?

[Stikus]

We are trying to connect our ISP now, but our network engineers confirming, that all Russian ISP have this problem and they didn't find legal reason for that (any government act or so).

[KweezyCode]

just tried with different providers. Yea, looks like all russian ips banned

[C-Otto]

just tried with different providers. Yea, looks like all russian ips banned

It's the reverse. We don't ban Russian IPs. Instead, Russia (the state/government?) blocks our IP address.

[KweezyCode]

just tried with different providers. Yea, looks like all russian ips banned

It's the reverse. We don't ban Russian IPs. Instead, Russia (the state/government?) blocks our IP address.

it is not, ip or/and domain are not in russian blocklists

[C-Otto]

Well, then it's just a very weird coincidence that the reports from Russian users started to come in shortly after we added a Tor relay (which can be used for free speech, anti-censorship, ... - concepts I don't think are at home in Russia).

[Stikus]

@KweezyCode Carsten can say same thing on his end. We need more debugging. Can anyone user traceroute or something like this (its not my strong side)?

@C-Otto What about @mikhirev suggestion:

Can you change your GeoIP settings to exclude this server for requests originating from Russia?

Is it possible?

[C-Otto]

Here's how a tcptraceroute 137.226.34.46 443 should look like:

[...]
 8  cr-fra2-be5.x-win.dfn.de (188.1.242.9)  4.198 ms  4.098 ms  4.119 ms
 9  kr-aah15-0.x-win.dfn.de (188.1.242.110)  7.579 ms  7.314 ms  7.277 ms
10  fw-xwin-1-vl106.noc.rwth-aachen.de (134.130.3.236)  7.287 ms  7.012 ms  7.026 ms
11  n7k-ww10-1-vl158.noc.rwth-aachen.de (134.130.3.243)  7.952 ms * *
12  * n7k-ww10-2-et1-1.noc.rwth-aachen.de (137.226.139.42)  8.083 ms *
13  * ftp.halifax.rwth-aachen.de (137.226.34.46) <syn,ack>  7.390 ms *

[C-Otto]

@Stikus I'm not involved with Jenkins, you might want to ask them to exclude our server (create a new issue in this repository).

[KweezyCode]

[KweezyCode]

connection refused means that connection to server is ok :/

[C-Otto]

traceroute does not accept the port (443) as an argument, either use tcptraceroute or drop the port. It looks like it is sending 443 byte packets instead, which is weird.

And: please don't post pictures of text.

[C-Otto]

Connection refused does not mean anything. It can be a firewall that cuts the connection - and I believe that's exactly what's happening here.

[C-Otto]

Oh, and one more thing - sorry for the comment spam. If traceroute succeeds, that just means that ICMP packages are routed correctly. That doesn't mean that TCP connections work. You might want to use tcptraceroute instead, for that reason.

[C-Otto]

If any of you is interested, we can do a live screensharing session where you can see the output of tcpdump for specific IPs. If you have access to both a Russian IP and some non-blocked non-Russian IP, we might be able to see the difference between ICMP packets (working fine) and TCP packets (blocked somewhere in Russia).

[KweezyCode]

[C-Otto]

@KweezyCode is there a specific reason why you post pictures of text? That's just weird.

[KweezyCode]

@KweezyCode is there a specific reason why you post pictures of text? That's just weird.

I'm just too lazy to copy the text, so I take a screenshot of the area and just paste. I can copy the text if it bothers you

[C-Otto]

Na, it's fine. I'll try to be lazy, too.

[dduportal]

Hi @KweezyCode @Stikus @ihatethecloud, we've finally enabled (a few minutes ago) the Hostico mirror in Romania.

Could you share with us the result of https://get.jenkins.io/windows-stable/2.452.1/jenkins.msi?mirrorlist to see how is the mirror redirector behaving in your areas?

It should shows Hostico with a higher priority than Aaechen University's mirror: could you confirm?

[w-e-g]

Hi all!

I experienced the same issue but from two providers. Cloud.ru and from MTS. But the second case is quite strange. Sometimes it works, sometimes not.

Traceroute path when it works from MTS:

traceroute 137.226.34.46
traceroute to 137.226.34.46 (137.226.34.46), 64 hops max, 40 byte packets
 1  router.lan (X.X.X.X)  8.191 ms  1.362 ms  1.232 ms
 2  46.138.240.1 (46.138.240.1)  5.324 ms  4.350 ms  4.202 ms
 3  mpts-ss-51.msk.mts-internet.net (212.188.1.6)  4.565 ms  4.685 ms  4.987 ms
 4  mag9-cr03-be12.51.msk.mts-internet.net (212.188.1.5)  3.580 ms * *
 5  mag9-cr02-be13.77.msk.mts-internet.net (195.34.53.206)  11.063 ms * *
 6  a433-cr02-be15.77.msk.mts-internet.net (212.188.28.102)  14.068 ms  5.057 ms  4.635 ms
 7  mmon-cr01-be1.78.spb.mts-internet.net (212.188.2.53)  18.767 ms  18.050 ms  27.175 ms
 8  radio-cr01-ae3.0.hel.mts-internet.net (212.188.29.109)  28.552 ms
    radio-cr01-ae9.0.hel.mts-internet.net (212.188.29.23)  25.208 ms
    radio-cr01-ae3.0.hel.mts-internet.net (212.188.29.109)  24.062 ms
 9  cr-fra2-be1.x-win.dfn.de (80.81.192.222)  44.508 ms  42.656 ms  43.699 ms
10  kr-aah15-0.x-win.dfn.de (188.1.242.110)  52.869 ms  52.720 ms  54.552 ms
11  fw-xwin-1-vl106.noc.rwth-aachen.de (134.130.3.236)  53.404 ms  51.378 ms  52.071 ms
12  n7k-ww10-1-vl158.noc.rwth-aachen.de (134.130.3.243)  53.843 ms  54.543 ms  53.805 ms
13  n7k-sw23-2-et1-1.noc.rwth-aachen.de (137.226.38.58)  53.804 ms  53.633 ms  52.962 ms
14  ftp.halifax.rwth-aachen.de (137.226.34.46)  49.589 ms  48.705 ms  48.299 ms

From the cloud.ru it looks banned from inside(?). Does anyone test cloud.ru connection to ftp.halifax.rwth-aachen.de?

[dduportal]

Hi @w-e-g we could you check my message above please and share the result?

[KweezyCode]

works fine after changing mirror

UPD: still fails with some plugins, but downloads them much faster

java.io.IOException: Failed to load: Pipeline: GitHub Groovy Libraries (pipeline-github-lib 61.v629f2cc41d83)
 - Failed to load: Git plugin (git 5.2.2)
	at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:992)
	at hudson.PluginManager.dynamicLoad(PluginManager.java:949)
Caused: java.io.IOException: Failed to install pipeline-github-lib plugin
	at hudson.PluginManager.dynamicLoad(PluginManager.java:963)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2249)
Caused: java.io.IOException: Failed to dynamically deploy this plugin
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2253)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1899)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.base/java.lang.Thread.run(Unknown Source)
	```

[dduportal]

works fine after changing mirror

UPD: still fails with some plugins, but downloads them much faster

java.io.IOException: Failed to load: Pipeline: GitHub Groovy Libraries (pipeline-github-lib 61.v629f2cc41d83)
 - Failed to load: Git plugin (git 5.2.2)
	at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:992)
	at hudson.PluginManager.dynamicLoad(PluginManager.java:949)
Caused: java.io.IOException: Failed to install pipeline-github-lib plugin
	at hudson.PluginManager.dynamicLoad(PluginManager.java:963)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2249)
Caused: java.io.IOException: Failed to dynamically deploy this plugin
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2253)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1899)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:121)
	at java.base/java.lang.Thread.run(Unknown Source)

Do you have the URL effectively used when failing? The snippet you show is only a partial of the whole logs so it's hard to diagnose.

[w-e-g]

@dduportal, just tested my script for mirroring plugins directory. Looks like connections are redirected to new mirror.

Btw, is new mirror have any limitations? Connections, address pool, etc. And which is the best way to sync Jenkins plugins? Rsync, http?

[dduportal]

@dduportal, just tested my script for mirroring plugins directory. Looks like connections are redirected to new mirror.

Hi @w-e-g , thanks for the confirmation. We'll close the issue again.

Many thanks @C-Otto for the support. Note that with the effective enablement of the new mirror in Romania, you should see a load decrease on your mirror.

Btw, is new mirror have any limitations? Connections, address pool, etc.

@w-e-g We don't know: we are a sponsored project so we take what is given. You may want to contact Hostico for details on this.

And which is the best way to sync Jenkins plugins? Rsync, http?

What do you mean by "sync Jenkins plugins" exactly? It looks like you are trying to build your own local mirror? If you are able to provide one in Russia, that would help you and other users as well: we need an HTTPS public server with around 500 to 750 Gb disk, and we also need an rsync or FTP server so we can scan the content of your mirror (to make the mirror redirector system working). Happy to give you details if you are interested to help the community

[C-Otto]

Note that with the effective enablement of the new mirror in Romania, you should see a load decrease on your mirror.

Jenkins currently is served with around 66 MBit/sec on average, which is 1.4% of the total traffic. I don't think we'll notice any change...

[mikhirev]

TCP tracerute breaks immediately after my ISP network

% sudo traceroute --tcp --port=443 ftp.halifax.rwth-aachen.de
traceroute to ftp.halifax.rwth-aachen.de (137.226.34.46), 30 hops max, 60 byte packets
 1  OpenWrt.home (192.168.1.1)  0.310 ms  0.480 ms *
 2  172.30.174.1 (172.30.174.1)  2.951 ms  2.976 ms  2.982 ms
 3  p2sr-e38sr.line-r.ru (213.108.208.98)  2.992 ms  3.003 ms  3.014 ms
 4  e38sr-g1.line-r.ru (213.108.208.169)  3.025 ms  3.045 ms  3.050 ms
 5  g1-e38sri.line-r.ru (213.108.208.166)  3.056 ms  3.188 ms  3.201 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

This server is not present in the static blocking list. This means that it is blocked by TSPU that is a black box for ISPs and is directly controlled by Roskomnadzor. Usually when HTTPS sites are blocked, the TCP session is being reset, however in this case it seems that packets are simply dropped. So it is likely that the block is caused by the Tor node, not the website content.

I see only two solutions:

1. Do not redirect users from Russia to this mirror.
2. Serve the Jenkins mirror and the Tor node from different IP addresses.

[C-Otto]

Third and obvious option: have a nice chat with Roskomnadzor and kindly ask them to remove the block.

[dduportal]

• The problem is not related to the Jenkins infrastructure and neither to the Aachen University. As the root cause as been identified to be a Roskomnadzor block, please check with them
• As explained in Timeout to ftp.halifax.rwth-aachen.de #4128 (comment), we provided a mirror close to Russia and at least 3 Russian users did confirm it work for them

=> We've done everything we could. If you still have issues, please help us by sponsoring the project and finding an organization, within the Russian network, which could provide us a mirror (see comments below) for Russian users.

[KweezyCode]

Third and obvious option: have a nice chat with Roskomnadzor and kindly ask them to remove the block.

haha that's not how it works :D

[mikhirev]

If you still have issues, please help us by sponsoring the project and finding an organization, within the Russian network, which could provide us a mirror (see comments below) for Russian users.

There's a mirror at Yandex. Is it unofficial? I can try to contact them if you tell me what they need to do to make it used by default for Russian users.

[w-e-g]

@mikhirev maybe I can help you. I have a contact of mirror.yandex.ru engineer. But it has been a very long time ago and I'm not sure that he's work on this today.

[dduportal]

@mikhirev @w-e-g interesting! I confirm it is an unofficial mirror: but if they can provide us (Jenkins infra team - contact email at jenkins-infra-team googlegroups.com) an Rsync or FTP (so we can scan files and setup the redirector system) then we could absolutely add it as another mirror!

[lemeurherve]

@mikhirev @w-e-g that would be great!

For adding a mirror you can send us on the private Jenkins infra team mail address above:

• an URL where the mirror will be available
• a rsync address and/or a FTP address
• an administrative email address

Current size of all mirrored files: around 530Gio.

For the initial sync, we suggest using OSUOSL mirror:

https://ftp-nyc.osuosl.org/pub/jenkins
rsync://ftp-nyc.osuosl.org/jenkins
ftp://ftp-nyc.osuosl.org/pub/jenkins

[dduportal]

Hey @mikhirev @w-e-g and others: I've opened #4147 to work specifically on mirros for Jenkins Russian users if you are interested to help or at least being informed!

[timja]

FYI Russia mirror completed in #4147 (comment)