Package API Rate Limiting

[dazlab]

Hi,

Apologies if this has been covered already; I can't find any reference to it.

Seem to have encountered the attached when adding a few packages. Not sure if it's a bug, or if the API is rate limited at present.

[Daeraxa]

It is indeed rate limited but I think we found that pulsar queries all its packages each time which was inflating the numbers. I think a pr was made to add a deeper cache to prevent this.

[confused-Techie]

I'll add here that you are right, I dont think we have updated the docs to say the API is rate limited. Although it should be high enough that any normal usage shouldn't hit it.

If you'd like if you could include some of the packages you clicked to install or search that failed and I can try to see what happened to cause the limit to get hit we can see about increasing this value.

But for reference, we enabled the rate limit out of need when our servers were getting hit hard. And since being enabled has already been bumped significantly to try and ensure users don't hit the limit in day to day activities.

Currently, our rate limit is 200 requests granted to authentictaed endpoints and 200 for unauthenticated endpoints per 15-minute window per IP Address

[confused-Techie]

@dazlab I did want to let you know, I did my best to review todays activity, and nothing seemed out of the ordinary.

The issue lies mainly in the fact that the old Atom code that hits these servers isn't very nice, meaning every time the editor is launched it checks if any packages need an update, which makes sense, but it does this for every installed package, all at the same time, so it can be a massive amount of hits each time you open the editor.

But to prevent this from affecting you again, I've raised the limit to both endpoints to 300, so ideally you shouldn't see this message again under normal usage

[dazlab]

@confused-Techie Cheers. I can see the rate limit code in the package-backend src in main.js. Yeah, not sure. I installed all the monocolor-* syntax themes, Editor-Background, Remote-FTP and Hide-Tabs. So maybe a dozen or so packages. I've also been unable to replicate, so maybe a transient issue?

[confused-Techie]

Or more likely that it's now bumped you shouldn't hit it. So if you're unable to replicate that means the bump is likely at a good level. But of course will keep an ear out if it happens again to you or anyone else

[dazlab]

OK, so I downloaded the latest Beta release last night and installed on my laptop. Opened it up, downloaded all the monocolor-* packages, hide-tabs, editor-background, seti-icons and went to search for the fonts package and got the same error message.

Are you able to re-create by following these steps? It's probably due to the request spamming that takes place when opening using up a bunch of requests, and then immediately downloading lots of packages within that first 15 min window.

I'll maybe do a Wireshark trace to confirm this.

[confused-Techie]

If you'd like to do a trace it would be helpful to determine exactly what requests are going out. I wonder if the built in Electron Chromium DevTools would show the requests even.

But yeah can confirm from the server logs that there were a lot of requests out to a lot of packages. Looked like normal usage but didnt investigate any individually