forked from data-integrations/google-cloud
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
4a086e7
commit a7d143e
Showing
16 changed files
with
832 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
79 changes: 79 additions & 0 deletions
79
src/e2e-test/features/bigquerymultitable/BiqQueryMultiTableSinkError.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # Copyright © 2023 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @BQMT_SINK | ||
| Feature: BigQueryMultiTable sink - Validate BigQueryMultiTable sink plugin error scenarios | ||
|
|
||
| @BQMT_Required | ||
| Scenario Outline: Verify BigQueryMultiTable Sink properties validation errors for mandatory fields | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQueryMultiTable" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| Then Click on the Validate button | ||
| Then Validate mandatory property error for "<property>" | ||
| Examples: | ||
| | property | | ||
| | dataset | | ||
|
|
||
| Scenario:Verify BQMT Sink properties validation errors for incorrect value of chunk size | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQueryMultiTable" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" | ||
| And Replace input plugin property: "project" with value: "projectId" | ||
| And Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Override Service account details if set in environment variables | ||
| Then Enter input plugin property: "gcsChunkSize" with value: "bqmtInvalidChunkSize" | ||
| Then Click on the Validate button | ||
| Then Verify that the Plugin Property: "gcsChunkSize" is displaying an in-line error message: "errorMessageIncorrectBQMTChunkSize" | ||
|
|
||
| @BQMT_Required | ||
| Scenario:Verify BQMT Sink properties validation errors for incorrect dataset | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQueryMultiTable" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" | ||
| And Replace input plugin property: "project" with value: "projectId" | ||
| Then Override Service account details if set in environment variables | ||
| Then Enter input plugin property: "dataset" with value: "bqmtInvalidSinkDataset" | ||
| Then Click on the Validate button | ||
| Then Verify that the Plugin Property: "dataset" is displaying an in-line error message: "errorMessageIncorrectBQMTDataset" | ||
|
|
||
| Scenario:Verify BQMT Sink properties validation errors for incorrect reference name | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQueryMultiTable" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Replace input plugin property: "project" with value: "projectId" | ||
| And Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Override Service account details if set in environment variables | ||
| Then Enter input plugin property: "referenceName" with value: "bqmtInvalidSinkReferenceName" | ||
| Then Click on the Validate button | ||
| Then Verify that the Plugin Property: "referenceName" is displaying an in-line error message: "errorMessageIncorrectBQMTReferenceName" | ||
|
|
||
| Scenario:Verify BQMT Sink properties validation errors for incorrect value of temporary bucket name | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQueryMultiTable" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" | ||
| And Replace input plugin property: "project" with value: "projectId" | ||
| And Enter input plugin property: "dataset" with value: "dataset" | ||
| Then Override Service account details if set in environment variables | ||
| Then Enter input plugin property: "bucket" with value: "bqmtInvalidTemporaryBucket" | ||
| Then Click on the Validate button | ||
| Then Verify that the Plugin Property: "bucket" is displaying an in-line error message: "errorMessageIncorrectBQMTBucketName" |
133 changes: 133 additions & 0 deletions
133
...atures/bigquerymultitable/MultipleDatabaseTableToBigQueryMultiTableSink_WithMacro.feature
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| # Copyright © 2023 Cask Data, Inc. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); you may not | ||
| # use this file except in compliance with the License. You may obtain a copy of | ||
| # the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
| # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
| # License for the specific language governing permissions and limitations under | ||
| # the License. | ||
|
|
||
| @BQMT_SINK | ||
| Feature: BigQueryMultiTable sink -Verification of Multiple Database Tables to BigQueryMultiTable successful data transfer using macros | ||
|
|
||
| @MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required | ||
| Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with all datatypes using macros | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Source" | ||
| When Select plugin: "Multiple Database Tables" from the plugins list as: "Source" | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "Multiple Database Tables" | ||
| Then Replace input plugin property: "referenceName" with value: "ref" | ||
| Then Enter input plugin property: "connectionString" with value: "connectionString" for Credentials and Authorization related fields | ||
| Then Replace input plugin property: "jdbcPluginName" with value: "mysql" | ||
| Then Replace input plugin property: "user" with value: "user" for Credentials and Authorization related fields | ||
| Then Replace input plugin property: "password" with value: "pass" for Credentials and Authorization related fields | ||
| And Select radio button plugin property: "dataSelectionMode" with value: "sql-statements" | ||
| Then Click on the Add Button of the property: "sqlStatements" with value: | ||
| | selectQuery| | ||
| Then Click on the Validate button | ||
| And Close the Plugin Properties page | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" | ||
| Then Click on the Macro button of Property: "projectId" and set the value to: "bqProjectId" | ||
| Then Click on the Macro button of Property: "datasetProjectId" and set the value to: "bqDatasetProjectId" | ||
| Then Click on the Macro button of Property: "serviceAccountType" and set the value to: "serviceAccountType" | ||
| Then Click on the Macro button of Property: "serviceAccountFilePath" and set the value to: "serviceAccount" | ||
| Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset" | ||
| Then Click plugin property: "truncateTable" | ||
| Then Click plugin property: "allowSchema" | ||
| Then Click on the Validate button | ||
| And Close the Plugin Properties page | ||
| Then Connect plugins: "Multiple Database Tables" and "BigQuery Multi Table" to establish connection | ||
| Then Save the pipeline | ||
| Then Preview and run the pipeline | ||
| Then Enter runtime argument value "projectId" for key "bqProjectId" | ||
| Then Enter runtime argument value "projectId" for key "bqDatasetProjectId" | ||
| Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType" | ||
| Then Enter runtime argument value "serviceAccount" for key "serviceAccount" | ||
| Then Enter runtime argument value "dataset" for key "bqDataset" | ||
| Then Run the preview of pipeline with runtime arguments | ||
| Then Wait till pipeline preview is in running state | ||
| # Then Open and capture pipeline preview logs | ||
| # Then Verify the preview of pipeline is "succeeded" | ||
| # Then Close the pipeline logs | ||
| Then Close the preview | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Enter runtime argument value "projectId" for key "bqProjectId" | ||
| Then Enter runtime argument value "projectId" for key "bqDatasetProjectId" | ||
| Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType" | ||
| Then Enter runtime argument value "serviceAccount" for key "serviceAccount" | ||
| Then Enter runtime argument value "dataset" for key "bqDataset" | ||
| Then Run the Pipeline in Runtime with runtime arguments | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate the values of records transferred to BQMT sink is equal to the value from source MultiDatabase table | ||
|
|
||
| @MULTIPLEDATABASETABLE_SOURCE_TEST @BQMT_Required | ||
| Scenario:Verify data is getting transferred from Multiple Database Tables to BQMT sink with split field using macros | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Expand Plugin group in the LHS plugins list: "Source" | ||
| When Select plugin: "Multiple Database Tables" from the plugins list as: "Source" | ||
| When Expand Plugin group in the LHS plugins list: "Sink" | ||
| When Select plugin: "BigQuery Multi Table" from the plugins list as: "Sink" | ||
| Then Navigate to the properties page of plugin: "Multiple Database Tables" | ||
| Then Replace input plugin property: "referenceName" with value: "ref" | ||
| Then Enter input plugin property: "connectionString" with value: "connectionString" for Credentials and Authorization related fields | ||
| Then Replace input plugin property: "jdbcPluginName" with value: "mysql" | ||
| Then Replace input plugin property: "user" with value: "user" for Credentials and Authorization related fields | ||
| Then Replace input plugin property: "password" with value: "pass" for Credentials and Authorization related fields | ||
| And Select radio button plugin property: "dataSelectionMode" with value: "sql-statements" | ||
| Then Click on the Add Button of the property: "sqlStatements" with value: | ||
| | selectQuery| | ||
| Then Click on the Validate button | ||
| And Close the Plugin Properties page | ||
| Then Navigate to the properties page of plugin: "BigQuery Multi Table" | ||
| And Enter input plugin property: "referenceName" with value: "Reference" | ||
| Then Click on the Macro button of Property: "projectId" and set the value to: "bqProjectId" | ||
| Then Click on the Macro button of Property: "datasetProjectId" and set the value to: "bqDatasetProjectId" | ||
| Then Click on the Macro button of Property: "serviceAccountType" and set the value to: "serviceAccountType" | ||
| Then Click on the Macro button of Property: "serviceAccountFilePath" and set the value to: "serviceAccount" | ||
| Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset" | ||
| Then Click on the Macro button of Property: "splitField" and set the value to: "bqmtSplitField" | ||
| Then Click plugin property: "truncateTable" | ||
| Then Click plugin property: "allowSchema" | ||
| Then Click on the Validate button | ||
| And Close the Plugin Properties page | ||
| Then Connect plugins: "Multiple Database Tables" and "BigQuery Multi Table" to establish connection | ||
| Then Save the pipeline | ||
| Then Preview and run the pipeline | ||
| Then Enter runtime argument value "projectId" for key "bqProjectId" | ||
| Then Enter runtime argument value "projectId" for key "bqDatasetProjectId" | ||
| Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType" | ||
| Then Enter runtime argument value "serviceAccount" for key "serviceAccount" | ||
| Then Enter runtime argument value "dataset" for key "bqDataset" | ||
| Then Enter runtime argument value "bqmtSplitField" for key "bqmtSplitField" | ||
| Then Run the preview of pipeline with runtime arguments | ||
| Then Wait till pipeline preview is in running state | ||
| Then Open and capture pipeline preview logs | ||
| Then Verify the preview of pipeline is "succeeded" | ||
| Then Close the pipeline logs | ||
| Then Close the preview | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Enter runtime argument value "projectId" for key "bqProjectId" | ||
| Then Enter runtime argument value "projectId" for key "bqDatasetProjectId" | ||
| Then Enter runtime argument value "serviceAccountType" for key "serviceAccountType" | ||
| Then Enter runtime argument value "serviceAccount" for key "serviceAccount" | ||
| Then Enter runtime argument value "dataset" for key "bqDataset" | ||
| Then Enter runtime argument value "bqmtSplitField" for key "bqmtSplitField" | ||
| Then Run the Pipeline in Runtime with runtime arguments | ||
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Close the pipeline logs | ||
| Then Validate the values of records transferred to BQMT sink is equal to the value from source MultiDatabase table |
Oops, something went wrong.