-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[sc-27604] Parse TLL for UnityCatalog query logs #919
[sc-27604] Parse TLL for UnityCatalog query logs #919
Conversation
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #919 +/- ##
==========================================
+ Coverage 93.55% 93.74% +0.19%
==========================================
Files 216 177 -39
Lines 19498 19559 +61
==========================================
+ Hits 18241 18336 +95
+ Misses 1257 1223 -34 ☔ View full report in Codecov by Sentry. |
Pls also measure the time it takes for max_workers=2 to simulate the typical crawler environment. |
🤔 Why?
UnityCatalog does not compute the sources and targets of each query log, so we need to do it ourselves.
The TLL parser is largely based on our in-house lineage parser, with some incompatible methods stripped off.
🤓 What?
poetry export
step in CD'sdocker
job. This step creates therequirements.txt
file, which is populated during CD and is used to build the image.🧪 Tested?
☑️ Checks
pyproject.toml
.