Skip to content

Commit

Permalink
[DOCS] Update docs for UnexpectedRowsExpectation (#10391)
Browse files Browse the repository at this point in the history
  • Loading branch information
NathanFarmer authored Sep 19, 2024
1 parent f567c84 commit 215b6bf
Show file tree
Hide file tree
Showing 3 changed files with 20 additions and 9 deletions.
20 changes: 13 additions & 7 deletions docs/docusaurus/docs/cloud/expectations/manage_expectations.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@ An Expectation is a verifiable assertion about your data. They make implicit ass

<!-- [//]: # (TODO: To learn more about Expectations, see Expectation.) -->

:::info Custom SQL Query Expectations

To create custom SQL query Expectations, you'll need to use the GX API. See [Customize Expectations](/core/customize_expectations/customize_expectations.md).

:::

## Prerequisites

- You have a [Data Asset](/cloud/data_assets/manage_data_assets.md#create-a-data-asset).
Expand All @@ -23,7 +17,7 @@ To create custom SQL query Expectations, you'll need to use the GX API. See [Cus
The following table lists the available GX Cloud Expectations.

| Data Quality Issue | Expectation | Description |
| ------------------ | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|--------------------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|
| Cardinality | `expect_column_values_to_be_unique` | Expect each column value to be unique. |
| Cardinality | `expect_compound_columns_to_be_unique` | Expect the compound columns to be unique. |
| Cardinality | `expect_select_column_values_to_be_unique_within_record` | Expect the values for each record to be unique across the columns listed. Note that records can be duplicated. |
Expand Down Expand Up @@ -69,6 +63,18 @@ The following table lists the available GX Cloud Expectations.
| Volume | `expect_table_row_count_to_equal` | Expect the number of rows to equal a value. |
| Volume | `expect_table_row_count_to_equal_other_table` | Expect the number of rows to equal the number in another table within the same database. |

## Custom SQL Expectations

GX Cloud also offers the ability to write a custom Expectation using SQL. It is designed to fail validation if the provided SQL query returns one or more rows.

The provided query should be written in the dialect of the Data Source in which a given Data Asset lives.

:::info Optional `{batch}` named query

The optional `{batch}` named query references the Batch of data under test. When the Expectation is evaluated, the `{batch}` named query will be replaced with the Batch of data that is validated.

:::

## Add an Expectation

1. In GX Cloud, click **Data Assets**.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import PrereqPreconfiguredDataSourceAndAsset from '../_core_components/prerequis

Among the available Expectations, the `UnexpectedRowsExpectation` is designed to facilitate the execution of SQL or Spark-SQL queries as the core logic for an Expectation. By default, `UnexpectedRowsExpectation` considers validation successful when no rows are returned by the provided SQL query.

You customize an `UnexpectedRowsExpectation` in essentially the same manner as you would [define a custom Expectation](/core/customize_expectations/define_a_custom_expectation_class.md), by subclassing `UnexpectedRowsExpectation` and providing customized default attributes and text for Data Docs. However, there are some caveats around the `UnexpectedRowsExpectation`'s `unexpected_rows_query` attribute that deserve further detail.
Like any other Expectation, you can instantiate the `UnexpectedRowsExpectation` directly. You can also customize an `UnexpectedRowsExpectation` in essentially the same manner as you would [define a custom Expectation](/core/customize_expectations/define_a_custom_expectation_class.md), by subclassing `UnexpectedRowsExpectation` and providing customized default attributes and text for Data Docs. However, there are some caveats around the `UnexpectedRowsExpectation`'s `unexpected_rows_query` attribute that deserve further detail.

<!-- TODO: Do we want to discuss custom `_validate(...)` logic here, or should that be held for a future topic on building custom Expectation classes from scratch? -->

Expand Down Expand Up @@ -48,7 +48,7 @@ You customize an `UnexpectedRowsExpectation` in essentially the same manner as y

The `unexpected_rows_query` attribute is a SQL or Spark-SQL query that returns a selection of rows from the Batch of data being validated. By default, rows that are returned have failed the validation check.

Although the `unexpected_rows_query` should be written in standard SQL or Spark-SQL syntax, it must also contain the special `{batch}` placeholder. When the Expectation is evaluated, the `{batch}` placeholder will be replaced with the Batch of data that is validated.
The `unexpected_rows_query` should be written in standard SQL or Spark-SQL syntax, except that it can also contain the special `{batch}` named query. When the Expectation is evaluated, the `{batch}` keyword will be replaced with the Batch of data that is configured for your Data Asset.

In this example, `unexpected_rows_query` will select any rows where the passenger count is greater than `6` or less than `0`. These rows will fail validation for this Expectation:

Expand Down
5 changes: 5 additions & 0 deletions docs/docusaurus/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,11 @@ module.exports = {
label: 'Available Expectations',
href: '/docs/cloud/expectations/manage_expectations#available-expectations',
},
{
type: 'link',
label: 'Custom SQL Expectations',
href: '/docs/cloud/expectations/manage_expectations#custom-sql-expectations',
},
{
type: 'link',
label: 'Add an Expectation',
Expand Down

0 comments on commit 215b6bf

Please sign in to comment.