Stuck at 99.34%

[tylerweitzman]

Hi,
I get the following

got -c 128 https://datashare.ed.ac.uk/download/DS_10283_3443.zip
  99.34% ▕████████████████████████▉▏ 12 GB/12 GB @ 0 B/s

I'm using a VM with 8 vCPUs, so it's 16 threads per core. It downloaded @40-90MB/s at the beginning and then once it reached 99.45% it slows down to 0 B/s. What's going on?

[melbahja]

Maybe you're IP address get blocked, 128 is too many concurrent connections.

[melbahja]

I would recommend that u always keep the -c value below 10.

[tylerweitzman]

I actually tried even 1000 connections. It was interesting because I was able to get up to 90 MB/s on that whereas with wget I get 400Kb. When I use 10 connections with got its around 3-5 MB/s. It kept up on 90 up until 90% and only then I started seeing it go down to 0 Is there any way that we could recover from rate limits that are delayed? For instance, start with 1000 connections, and then decrease to 10 later

On Tue, Apr 26, 2022 at 9:13 PM Mohamed El Bahja ***@***.***> wrote: I would recommend that u always keep the -c value below 10. — Reply to this email directly, view it on GitHub <#47 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP3R7X3PVS57BIZVZZH2ETVHCBBRANCNFSM5UNHZICA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *---* *Tyler Weitzman* *LinkedIn <http://linkedin.com/in/tylerweitzman>* *BlackSMS <http://blacksms.info/>*

[zQueal]

I actually tried even 1000 connections. It was interesting because I was able to get up to 90 MB/s on that whereas with wget I get 400Kb.

Because you're entirely monopolizing the throughput from the server by using so many connections, which as melbahja pointed out, you've probably been IP banned as an anti-DDoS measure...or maybe the server admin noticed the uptick in simultaneous connections and rate limited the number of connections from 1 host.

Is there any way that we could recover from rate limits that are delayed?

Something like this has nothing to do with got. It's entirely up to the server you're pulling from what kind of bandwidth is acceptable, and what is not. You weren't very nice to the server, so being unbanned is probably not going to happen.

[melbahja]

Adding X-Rate-Limit-Limit support will help prevent issues like this.