Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to sleep between max quota requests #76

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

viniciusbarros
Copy link

Background:

  • Existing observed issue: Rate exceeded #29
  • When running the auto subscribe in an AWS account with large amount of LogGroups (almost 1000). It was observed that throttling happened (more than 5 times) when getting data from LogGroups and it was not possible to auto subscribe to existing groups. The timeout was increased, but still we were not able to process all log groups in the account.
  • We tried an alternative method, which includes a sleep between the AWS Default Max Requests Per Sec Quota requests and not only we were able to process all log groups, but it was actually a bit faster than the existing exponential retry method.

Existing:

Took  190.973  seconds with 0 ms sleep
{ subs: 33, skips: 909, error: 5, throttled: 15 }

Optimized approach:

Took  189.895  seconds with 500 ms sleep
{ subs: 33, skips: 909, error: 5, throttled: 0 }

Changes:
Adding 2 optional parameters that allow users to specify the max quota (x) they have setup on their AWS accounts, and also to set a sleep time which happens right after (x) requests have been made.

…of requests per second

fix: format and making sure to parse env vars as int

fix: returning promise on sleep function

fix: formatting

Co-authored-by: Luciano Mammino <[email protected]>
Co-authored-by: Adminy <[email protected]>
@viniciusbarros viniciusbarros force-pushed the sleep-between-max-quota-requests branch from 32a5260 to 683578c Compare November 29, 2022 15:54
@lmammino
Copy link

👍🏽 for this feature!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants