Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rate limiting issues with Power BI admin APIs #1045

Merged

Conversation

mars-lan
Copy link
Contributor

@mars-lan mars-lan commented Nov 19, 2024

🤔 Why?

Most of the Power BI Admin APIs have a rate limit of 200 requests/hour. We reach the limit quickly when crawling a large number of workspaces.

🤓 What?

Replace the following per-workspace API calls with global API calls. These global APIs appear to be equivalent and are able to scale to thousands of items without any issues.

Unfortunately, there is no global version of GetTilesAsAdmin, so we need to fallback to the non-admin version Get Tiles In Group. This would require the service principal to be added to each workspace, which is already a requirement for getting refresh schedule, data sources, etc.

🧪 Tested?

Verified the before/after MCE against production instances.

☑️ Checks

  • My PR contains actual code changes, and I have updated the version number in pyproject.toml.

@mars-lan mars-lan enabled auto-merge (squash) November 19, 2024 22:18
Copy link

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
13448 12038 90% 85% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
metaphor/power_bi/extractor.py 89% 🟢
metaphor/power_bi/power_bi_client.py 68% 🟢
TOTAL 78% 🟢

updated for commit: b4cef52 by action🐍

Copy link

codecov bot commented Nov 19, 2024

Codecov Report

Attention: Patch coverage is 60.86957% with 9 lines in your changes missing coverage. Please review.

Project coverage is 89.51%. Comparing base (db596ce) to head (b4cef52).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
metaphor/power_bi/power_bi_client.py 30.76% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1045      +/-   ##
==========================================
- Coverage   89.51%   89.51%   -0.01%     
==========================================
  Files         211      211              
  Lines       13449    13448       -1     
==========================================
- Hits        12039    12038       -1     
  Misses       1410     1410              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

Copy link
Contributor

@alyiwang alyiwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mars-lan mars-lan merged commit 81572b4 into main Nov 20, 2024
4 of 6 checks passed
@mars-lan mars-lan deleted the marslan/sc-28446/use-pbi-non-admin-api-to-avoid-rate-limiting branch November 20, 2024 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants