Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] puts 1.0 code under test for how to define Expectations #10229

Merged

Conversation

Rachel-Reverie
Copy link
Contributor

Description

  • Puts 1.0 code into scripts under test for:
    • How to create an Expectation with preset parameters
    • How to create an Expectation that uses Expectation Parameters
    • How to retrieve a Batch of sample data for testing Expectations using the pandas_default Data Source
    • How to retrieve a Batch of sample data for testing Expectations from a Batch Definition
    • How to test and modify an Expectation that uses preset parameters
    • How to test and modify an Expectation that uses an Expectation Parameters dictionary
    • How to organize Expectations into Expectation Suites

Definition of done:

  • PR title is prefixed with one of: [BUGFIX], [FEATURE], [DOCS], [MAINTENANCE], [CONTRIB]
  • Code is linted - run invoke lint (uses ruff format + ruff check)
  • Appropriate tests and docs have been updated

Copy link

netlify bot commented Aug 18, 2024

Deploy Preview for niobium-lead-7998 ready!

Name Link
🔨 Latest commit f8cb66c
🔍 Latest deploy log https://app.netlify.com/sites/niobium-lead-7998/deploys/66c3a9abdea0830008eb878b
😎 Deploy Preview https://deploy-preview-10229.docs.greatexpectations.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

codecov bot commented Aug 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.65%. Comparing base (0fd5529) to head (f8cb66c).
Report is 3 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop   #10229      +/-   ##
===========================================
- Coverage    79.67%   79.65%   -0.02%     
===========================================
  Files          454      454              
  Lines        39507    39507              
===========================================
- Hits         31476    31469       -7     
- Misses        8031     8038       +7     
Flag Coverage Δ
3.10 66.29% <ø> (ø)
3.10 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.10 aws_deps ?
3.10 big ?
3.10 databricks ?
3.10 filesystem ?
3.10 mssql ?
3.10 mysql ?
3.10 postgresql ?
3.10 snowflake ?
3.10 spark ?
3.10 trino ?
3.11 66.29% <ø> (ø)
3.11 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.11 aws_deps ?
3.11 big ?
3.11 databricks ?
3.11 filesystem ?
3.11 mssql ?
3.11 mysql ?
3.11 postgresql ?
3.11 snowflake ?
3.11 spark ?
3.11 trino ?
3.12 64.84% <ø> (-0.02%) ⬇️
3.12 aws_deps 45.81% <ø> (ø)
3.12 big 54.46% <ø> (ø)
3.12 filesystem 60.45% <ø> (ø)
3.12 mssql 49.91% <ø> (ø)
3.12 mysql 49.97% <ø> (ø)
3.12 postgresql 54.11% <ø> (ø)
3.12 spark 57.58% <ø> (ø)
3.12 trino 52.18% <ø> (ø)
3.8 66.33% <ø> (-0.02%) ⬇️
3.8 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds 54.67% <ø> (ø)
3.8 aws_deps 45.84% <ø> (ø)
3.8 big 54.48% <ø> (ø)
3.8 databricks 47.10% <ø> (ø)
3.8 filesystem 60.47% <ø> (ø)
3.8 mssql 49.90% <ø> (ø)
3.8 mysql ?
3.8 postgresql 54.10% <ø> (ø)
3.8 snowflake 48.02% <ø> (ø)
3.8 spark 57.54% <ø> (ø)
3.8 trino 52.17% <ø> (ø)
3.9 66.31% <ø> (-0.02%) ⬇️
3.9 athena or clickhouse or openpyxl or pyarrow or project or sqlite or aws_creds ?
3.9 aws_deps ?
3.9 big ?
3.9 databricks ?
3.9 filesystem ?
3.9 mssql ?
3.9 mysql ?
3.9 postgresql ?
3.9 snowflake ?
3.9 spark ?
3.9 trino ?
cloud 0.00% <ø> (ø)
docs-basic 49.83% <ø> (+<0.01%) ⬆️
docs-creds-needed 50.53% <ø> (+<0.01%) ⬆️
docs-spark 49.49% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ons that utilize an Expectation Parameters dictionary.
Copy link
Contributor

@tyler-hoffman tyler-hoffman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple comments. Nothing blocking, but figured I'd verify if the one part is commented out before approving

# <snippet name="docs/docusaurus/docs/core/define_expectations/_examples/create_an_expectation.py - dynamic expectations">
passenger_expectation = gx.expectations.ExpectColumnMaxToBeBetween(
column="passenger_count",
min_value={"$PARAMETER": "expect_passenger_max_to_be_above"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just floating an idea: Does this become easier if you make a constant for expect_passenger_max_to_be_above, etc, to show how they are linked to what is passed in below?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in:

EXPECT_PASSENGER_MAX_TO_BE_ABOVE="expect_passenger_max_to_be_above"

...

passenger_expectation = gx.expectations.ExpectColumnMaxToBeBetween(
    column="passenger_count",
    min_value={"$PARAMETER": EXPECT_PASSENGER_MAX_TO_BE_ABOVE},

...

expectation_parameters = {
    EXPECT_PASSENGER_MAX_TO_BE_ABOVE: 1
}

?

I'm not certain if that's easier to comprehend or not, but I'm not against making the change if you think it is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like MAX_PASSENGER_PARAMETER, but that's the idea. I don't have a strong opinion here, I guess. If this were production code, I'd do it, but maybe that doesn't translate perfectly to documentation. Totally your call.

)
fare_expectation = gx.expectations.ExpectColumnMaxToBeBetween(
column="fare",
min_value={"$PARAMETER": "expect_fare_max_to_be_above"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this again, I'm wondering if we should consider abstracting away the $PARAMETER part of this before launch? Like what if it just became min_value=gx.expectations.parameter("expect_fare_max_to_be_above"),? Not blocking for this PR at all.

…ctations

# Conflicts:
#	docs/docusaurus/docs/components/examples_under_test.py
#	docs/docusaurus/docs/core/define_expectations/_retrieve_a_batch_of_test_data/_from_a_batch_definition.py
@Rachel-Reverie Rachel-Reverie added this pull request to the merge queue Aug 20, 2024
Merged via the queue into develop with commit 480753f Aug 20, 2024
61 checks passed
@Rachel-Reverie Rachel-Reverie deleted the d/doc-804/1-0_code_under_test_create_expectations branch August 20, 2024 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants