Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Update docs for UnexpectedRowsExpectation #10391

Merged
merged 23 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6df5369
Update OSS docs
NathanFarmer Sep 13, 2024
6fa5eed
Add unexpected_rows_expectation to list of available Expectations
NathanFarmer Sep 13, 2024
7109784
Cleanup
NathanFarmer Sep 13, 2024
94a8e77
Revert "Add unexpected_rows_expectation to list of available Expectat…
NathanFarmer Sep 18, 2024
ce26a47
Add section on Custom SQL Expectations
NathanFarmer Sep 18, 2024
2e96942
Remove custom SQL blurb at the top of the page
NathanFarmer Sep 18, 2024
7499ede
Revert "Remove custom SQL blurb at the top of the page"
NathanFarmer Sep 18, 2024
69910d7
Bleh
NathanFarmer Sep 18, 2024
d48d971
Update Custom SQL Expectations section
NathanFarmer Sep 18, 2024
50e10c1
Put backticks around {batch}
NathanFarmer Sep 18, 2024
58a43ca
Escape curly braces
NathanFarmer Sep 18, 2024
2851cc7
Escape curly braces
NathanFarmer Sep 18, 2024
b38ce9c
Remove backticks
NathanFarmer Sep 18, 2024
4bb4ac8
Remove backticks
NathanFarmer Sep 18, 2024
4ec017f
Try something else
NathanFarmer Sep 18, 2024
61a7740
Put in backticks now
NathanFarmer Sep 18, 2024
b46a002
Moar escapes
NathanFarmer Sep 18, 2024
56df736
Try this
NathanFarmer Sep 18, 2024
b3472fc
Make Custom SQL Expectations H2
NathanFarmer Sep 19, 2024
10d1c02
Add section to sidebar
NathanFarmer Sep 19, 2024
fcb8ac1
Merge branch 'develop' into d/ph-1472/custom-sql-expectations-docs
NathanFarmer Sep 19, 2024
7d2eee1
Merge branch 'develop' into d/ph-1472/custom-sql-expectations-docs
NathanFarmer Sep 19, 2024
b7009df
Merge branch 'develop' into d/ph-1472/custom-sql-expectations-docs
NathanFarmer Sep 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions docs/docusaurus/docs/cloud/expectations/manage_expectations.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@ An Expectation is a verifiable assertion about your data. They make implicit ass

<!-- [//]: # (TODO: To learn more about Expectations, see Expectation.) -->

:::info Custom SQL Query Expectations

To create custom SQL query Expectations, you'll need to use the GX API. See [Customize Expectations](/core/customize_expectations/customize_expectations.md).

:::

## Prerequisites

- You have a [Data Asset](/cloud/data_assets/manage_data_assets.md#create-a-data-asset).
Expand All @@ -23,7 +17,7 @@ To create custom SQL query Expectations, you'll need to use the GX API. See [Cus
The following table lists the available GX Cloud Expectations.

| Data Quality Issue | Expectation | Description |
| ------------------ | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|--------------------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| Cardinality | `expect_column_values_to_be_unique` | Expect each column value to be unique. |
| Cardinality | `expect_compound_columns_to_be_unique` | Expect the compound columns to be unique. |
| Cardinality | `expect_select_column_values_to_be_unique_within_record` | Expect the values for each record to be unique across the columns listed. Note that records can be duplicated. |
Expand Down Expand Up @@ -69,6 +63,18 @@ The following table lists the available GX Cloud Expectations.
| Volume | `expect_table_row_count_to_equal` | Expect the number of rows to equal a value. |
| Volume | `expect_table_row_count_to_equal_other_table` | Expect the number of rows to equal the number in another table within the same database. |

## Custom SQL Expectations

GX Cloud also offers the ability to write a custom Expectation using SQL. It is designed to fail validation if the provided SQL query returns one or more rows.

The provided query should be written in the dialect of the Data Source in which a given Data Asset lives.

:::info Optional `{batch}` named query

The optional `{batch}` named query references the Batch of data under test. When the Expectation is evaluated, the `{batch}` named query will be replaced with the Batch of data that is validated.

:::

## Add an Expectation

1. In GX Cloud, click **Data Assets**.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import PrereqPreconfiguredDataSourceAndAsset from '../_core_components/prerequis

Among the available Expectations, the `UnexpectedRowsExpectation` is designed to facilitate the execution of SQL or Spark-SQL queries as the core logic for an Expectation. By default, `UnexpectedRowsExpectation` considers validation successful when no rows are returned by the provided SQL query.

You customize an `UnexpectedRowsExpectation` in essentially the same manner as you would [define a custom Expectation](/core/customize_expectations/define_a_custom_expectation_class.md), by subclassing `UnexpectedRowsExpectation` and providing customized default attributes and text for Data Docs. However, there are some caveats around the `UnexpectedRowsExpectation`'s `unexpected_rows_query` attribute that deserve further detail.
Like any other Expectation, you can instantiate the `UnexpectedRowsExpectation` directly. You can also customize an `UnexpectedRowsExpectation` in essentially the same manner as you would [define a custom Expectation](/core/customize_expectations/define_a_custom_expectation_class.md), by subclassing `UnexpectedRowsExpectation` and providing customized default attributes and text for Data Docs. However, there are some caveats around the `UnexpectedRowsExpectation`'s `unexpected_rows_query` attribute that deserve further detail.

<!-- TODO: Do we want to discuss custom `_validate(...)` logic here, or should that be held for a future topic on building custom Expectation classes from scratch? -->

Expand Down Expand Up @@ -48,7 +48,7 @@ You customize an `UnexpectedRowsExpectation` in essentially the same manner as y

The `unexpected_rows_query` attribute is a SQL or Spark-SQL query that returns a selection of rows from the Batch of data being validated. By default, rows that are returned have failed the validation check.

Although the `unexpected_rows_query` should be written in standard SQL or Spark-SQL syntax, it must also contain the special `{batch}` placeholder. When the Expectation is evaluated, the `{batch}` placeholder will be replaced with the Batch of data that is validated.
The `unexpected_rows_query` should be written in standard SQL or Spark-SQL syntax, except that it can also contain the special `{batch}` named query. When the Expectation is evaluated, the `{batch}` keyword will be replaced with the Batch of data that is configured for your Data Asset.

In this example, `unexpected_rows_query` will select any rows where the passenger count is greater than `6` or less than `0`. These rows will fail validation for this Expectation:

Expand Down
5 changes: 5 additions & 0 deletions docs/docusaurus/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,11 @@ module.exports = {
label: 'Available Expectations',
href: '/docs/cloud/expectations/manage_expectations#available-expectations',
},
{
type: 'link',
label: 'Custom SQL Expectations',
href: '/docs/cloud/expectations/manage_expectations#custom-sql-expectations',
},
{
type: 'link',
label: 'Add an Expectation',
Expand Down
Loading