Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[sc-2887] GX Crawler #1024

Closed
wants to merge 4 commits into from
Closed

[sc-2887] GX Crawler #1024

wants to merge 4 commits into from

Conversation

usefulalgorithm
Copy link
Contributor

@usefulalgorithm usefulalgorithm commented Oct 28, 2024 •
edited
Loading

🤔 Why?

Great Expectations is a well know data quality assurance tool, we want to get data monitor information from it.

🤓 What?

  • Implemented basic GX crawler. Currently only Snowflake and Postgres connections have been tested. Pandas datasources and simple csv files are not supported, as we can't link them to actual datasets in an existing data platform.

🧪 Tested?

Added unit tests.

Each unit test does the following:

  • Runs the GX script to generate artifacts that include validation results
  • Runs the extractor to get the MCEs.

☑️ Checks

  • My PR contains actual code changes, and I have updated the version number in pyproject.toml.

@usefulalgorithm usefulalgorithm requested review from mars-lan, elic-eon and alyiwang and removed request for mars-lan and elic-eon October 30, 2024 14:29
@usefulalgorithm usefulalgorithm marked this pull request as ready for review October 30, 2024 14:29
@usefulalgorithm
# This is a combination of 2 commits.
2926362 
# This is the 1st commit message:

fix

add test

Refactor Unity Catalog to fetch catalog/schema/table metadata from System tables (#1022)

# This is the commit message #2:

fix stuff
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 30, 2024 •
edited
Loading

☂️ Python Coverage

current status: ❌

Overall Coverage

Lines Covered Coverage Threshold Status
13394 11957 89% 85% 🟢

New Files

File Coverage Status
metaphor/great_expectations/config.py 100% 🟢
metaphor/great_expectations/extractor.py 82% 🔴
TOTAL 91% 🔴

Modified Files

File Coverage Status
metaphor/glue/extractor.py 96% 🟢
TOTAL 96% 🟢

updated for commit: a552c40 by action🐍

@usefulalgorithm
finish test
3f8dcd6
@usefulalgorithm
finish test
43cbf05 
add docs

bump version

fix lock

fix ci

Delete tests/great_expectations/snowflake/config.yml

add git ignore
@usefulalgorithm usefulalgorithm force-pushed the tsung-julii/sc-2887/great-expectations-crawler branch from 8c4e909 to 43cbf05 Compare October 30, 2024 14:39
@usefulalgorithm usefulalgorithm deleted the tsung-julii/sc-2887/great-expectations-crawler branch October 30, 2024 14:41
@usefulalgorithm usefulalgorithm restored the tsung-julii/sc-2887/great-expectations-crawler branch October 30, 2024 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alyiwang alyiwang Awaiting requested review from alyiwang

@mars-lan mars-lan Awaiting requested review from mars-lan

@elic-eon elic-eon Awaiting requested review from elic-eon

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@usefulalgorithm