Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github-org Action Needs a Throttle #49

Open
benr opened this issue Dec 12, 2022 · 1 comment
Open

github-org Action Needs a Throttle #49

benr opened this issue Dec 12, 2022 · 1 comment

Comments

@benr
Copy link
Member

benr commented Dec 12, 2022

Scans of orgs of any reasonable size will fail due to the Github API Rate Limit. There should be a parameter to reduce the scan rate.

Screenshot 2022-12-11 at 7 13 43 PM

A consequence of slowing the scan rate will be excessive number of minutes spent in the action, which might mean the only solutions are:

  1. Reduce the number of scans the cnspec/mondoo client is making
  2. Update docs to recommend using repo scanning with offsets to stretch out the scans over a longer period of time
@czunker
Copy link
Contributor

czunker commented Jan 31, 2023

Failed Job with rate limit hit: https://github.com/mondoohq/.github/actions/runs/4051544141/jobs/6969976920

Only rate limit of 1000, I could find, is the action limit:
https://docs.github.com/en/rest/overview/resources-in-the-rest-api?apiVersion=2022-11-28#rate-limits-for-requests-from-github-actions

When using GITHUB_TOKEN, the rate limit is 1,000 requests per hour per repository.

As this is applied by the hour, I don't think slowing the action down makes sense.

It wonders me, that we need more than 1000 API calls for an org scan. Perhaps the rate limit endpoint helps debugging this: https://docs.github.com/en/rest/rate-limit?apiVersion=2022-11-28

Having another look at the failed job, this isn't only an org scan, but it also scans all the repos in this org. So, my current assumption is, the rate limit is applied to the repo, the action is running in, not the repo that is queried. So, splitting up the repo scans across the repos would also solve the rate limit, but makes it also unmanageable in large orgs.

Stretching a scan of an org and all its repos in a single job, might create other problems:
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

How could we separate the org scan and the repo scans without losing the ability to automatically detect all the repos?

Perhaps @benr had this on his mind, when suggesting solution 2:
Perhaps something like pagination would work? Running a workflow for the org + 50 first repos, another workflow for the 2nd 50 repos, ...
But that would be another burden on the user, and might also miss repos.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants