Skip to content

Commit

Permalink
debug
Browse files Browse the repository at this point in the history
  • Loading branch information
AnkitCLI committed Oct 24, 2024
1 parent ea12d50 commit 15c1689
Show file tree
Hide file tree
Showing 49 changed files with 83 additions and 83 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@
# License for the specific language governing permissions and limitations under
# the License.

@BigQueryMultiTable_Sink
@BigQueryMultiTable_Sinks
Feature: BigQueryMultiTable sink -Verification of BigQuery to BigQueryMultiTable successful data transfer

@BQ_TWO_SOURCE_BQMT_TEST @BQ_DELETE_TABLES_TEST
Scenario: Verify data successfully transferred from BigQuery To BigQueryMultiTable in two new tables
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
When Expand Plugin group in the LHS plugins list: "Source"s
When Select plugin: "BigQuery" from the plugins list as: "Source"
Then Navigate to the properties page of plugin: "BigQuery"
Then Replace input plugin property: "project" with value: "projectId"
Expand Down
4 changes: 2 additions & 2 deletions src/e2e-test/features/bigquery/sink/BigQuerySinkError.feature
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
@BigQuery_Sink
@BigQuery_Sinks
Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios

@BigQuery_Sink_Required
Scenario Outline:Verify BigQuery Sink properties validation errors for mandatory fields
Given Open Datafusion Project to configure pipeline
Given Open Datafusion Project to configure pipelines
When Sink is BigQuery
Then Open BigQuery sink properties
Then Enter the BigQuery properties with blank property "<property>"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@
# License for the specific language governing permissions and limitations under
# the License.

@BigQuery_Sink
@BigQuery_Sinks
Feature: BigQuery sink - Verification of BigQuery to BigQuery successful data transfer

@BQ_UPSERT_SOURCE_TEST @BQ_UPSERT_SINK_TEST @EXISTING_BQ_CONNECTION
Scenario:Validate successful records transfer from BigQuery source to BigQuery sink with Upsert operation by updating destination table schema and destination table exists with records in it.
Given Open Datafusion Project to configure pipeline
Given Open Datafusion Project to configure pipelines
When Expand Plugin group in the LHS plugins list: "Source"
When Select plugin: "BigQuery" from the plugins list as: "Source"
When Expand Plugin group in the LHS plugins list: "Sink"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@
# License for the specific language governing permissions and limitations under
# the License.

@BigQuery_Sink
@BigQuery_Sinks
Feature: BigQuery sink - Verification of BigQuery to BigQuery successful data transfer

@BQ_SOURCE_DATATYPE_TEST @BQ_SINK_TEST
Scenario:Validate successful records transfer from BigQuery to BigQuery with partition type TIME with Partition field and require partitioned filter true
Given Open Datafusion Project to configure pipeline
When Expand Plugin group in the LHS plugins list: "Source"
When Select plugin: "BigQuery" from the plugins list as: "Source"
When Select plugin: "BigQuery" from the plugins list as: "Source"s
When Expand Plugin group in the LHS plugins list: "Sink"
When Select plugin: "BigQuery" from the plugins list as: "Sink"
Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
Expand Down
4 changes: 2 additions & 2 deletions src/e2e-test/features/bigquery/sink/GCSToBigQuery.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Sink
@BigQuery_Sinks
Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfer

@CMEK @GCS_CSV_TEST @BQ_SINK_TEST @BigQuery_Sink_Required
Expand All @@ -22,7 +22,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
Then Toggle BigQuery sink property truncateTable to true
Then Toggle BigQuery sink property updateTableSchema to true
Then Validate "BigQuery" plugin properties
Then Close the BigQuery properties
Then Close the BigQuery propertiess
Then Connect source as "GCS" and sink as "BigQuery" to establish connection
Then Save the pipeline
Then Preview and run the pipeline
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Sink
@BigQuery_Sinks
Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfer with macro arguments

@CMEK @GCS_CSV_TEST @BQ_SINK_TEST @BigQuery_Sink_Required
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Validate BigQuery source plugin error scenarios

Scenario Outline:Verify BigQuery Source properties validation errors for mandatory fields
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Verification of BigQuery to BigQuery successful data transfer

@BQ_SOURCE_TEST @BQ_SINK_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Verification of BigQuery to BigQuery successful data transfer using connections

@BQ_SOURCE_TEST @BQ_SINK_TEST @BQ_CONNECTION @BigQuery_Source_Required
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Verification of BigQuery to GCS successful data transfer

@CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Verification of BigQuery to GCS successful data transfer with macro arguments

@CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BigQuery_Source
@BigQuery_Sources
Feature: BigQuery source - Verification of BigQuery to Multiple sinks successful data transfer

@CMEK @BQ_SOURCE_TEST @GCS_SINK_TEST @BQ_SINK_TEST @PUBSUB_SINK_TEST @BigQuery_Source_Required @CMEK_Required
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/bigqueryexecute/BQExecute.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BQExecute
@BQExecutes
Feature: BigQueryExecute - Verify data transfer using BigQuery Execute plugin

@BQ_SOURCE_TEST @BQ_SINK_TEST @BQ_EXECUTE_SQL @BQExecute_Required
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BQExecute
@BQExecutes
Feature: BigQueryExecute - Verify BigQueryExecute plugin error scenarios

Scenario: Verify BigQueryExecute validation error for mandatory field SQL Query
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@BQExecute
@BQExecutes
Feature: BigQueryExecute - Verify data transfer using BigQuery Execute plugin with macro arguments

@BQ_SOURCE_TEST @BQ_SINK_TEST @BQ_EXECUTE_SQL
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/bigtable/BigTableToBigTable.feature
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.
@BigTable @BIGTABLE_SOURCE_TEST
@BigTables @BIGTABLE_SOURCE_TEST
Feature: BigTable source - Verification of BigTable to BigTable Successful Data Transfer

@BIGTABLE_SINK_TEST @bigtable_Required
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/datastore/runtime.feature
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# License for the specific language governing permissions and limitations under
# the License.

@DataStore
@DataStores
Feature: DataStore - Verification of Datastore to Datastore Successful Data Transfer

@DATASTORE_SOURCE_ENTITY @datastore_Required
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/gcs/sink/GCSSink.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Sink
@GCS_Sinks
Feature: GCS sink - Verification of GCS Sink plugin

@CMEK @GCS_SINK_TEST @BQ_SOURCE_TEST
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/gcs/sink/GCSSinkError.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Sink
@GCS_Sinks
Feature: GCS sink - Verify GCS Sink plugin error scenarios

Scenario Outline:Verify GCS Sink properties validation errors for mandatory fields
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/gcs/source/GCSSourceError.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Verify GCS Source plugin error scenarios

Scenario Outline:Verify GCS Source properties validation errors for mandatory fields
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Verification of GCS to BQ successful data transfer

@GCS_CSV_TEST @BQ_SINK_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Verification of GCS to BQ successful data transfer

@BQ_SINK_TEST @GCS_READ_RECURSIVE_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Verification of GCS to GCS Additional Tests successful

@GCS_AVRO_FILE @GCS_SINK_TEST @GCS_Source_Required @ITN_TEST
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Verification of GCS to GCS successful data transfer using connections

@GCS_CSV_TEST @GCS_SINK_TEST @GCS_CONNECTION
Expand Down
12 changes: 6 additions & 6 deletions src/e2e-test/features/gcs/source/GCSourceSchema.feature
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
@GCS_Source
@GCS_Sources
Feature: GCS source - Validate GCS plugin output schema for different formats

Scenario Outline:GCS Source output schema validation for csv and tsv format
Expand All @@ -7,7 +7,7 @@ Feature: GCS source - Validate GCS plugin output schema for different formats
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Override Service account details if set in environment variables
Then Enter GCS source property path "<GcsPath>"
Then Enter GCS source property path "<GcsPath>"s
Then Select GCS property format "<FileFormat>"
Then Toggle GCS source property skip header to true
Then Validate output schema with expectedSchema "<ExpectedSchema>"
Expand All @@ -28,7 +28,7 @@ Feature: GCS source - Validate GCS plugin output schema for different formats
Then Enter GCS property projectId and reference name
Then Override Service account details if set in environment variables
Then Enter GCS source property path "<GcsPath>"
Then Select GCS property format "<FileFormat>"
Then Select GCS property format "<FileFormat>"s
Then Validate output schema with expectedSchema "<ExpectedSchema>"
@GCS_BLOB_TEST
Examples:
Expand All @@ -52,7 +52,7 @@ Feature: GCS source - Validate GCS plugin output schema for different formats
Given Open Datafusion Project to configure pipeline
When Source is GCS
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Enter GCS property projectId and reference names
Then Override Service account details if set in environment variables
Then Enter GCS source property path "<GcsPath>"
Then Select GCS property format "<FileFormat>"
Expand All @@ -70,7 +70,7 @@ Feature: GCS source - Validate GCS plugin output schema for different formats
Given Open Datafusion Project to configure pipeline
When Source is GCS
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Enter GCS property projectId and reference names
Then Override Service account details if set in environment variables
Then Enter GCS source property path "gcsDelimitedFile"
Then Select GCS property format "delimited"
Expand All @@ -82,7 +82,7 @@ Feature: GCS source - Validate GCS plugin output schema for different formats
Given Open Datafusion Project to configure pipeline
When Source is GCS
Then Open GCS source properties
Then Enter GCS property projectId and reference name
Then Enter GCS property projectId and reference names
Then Enter GCS source property path "<GcsPath>"
Then Select GCS property format "<FileFormat>"
Then Toggle GCS source property skip header to true
Expand Down
4 changes: 2 additions & 2 deletions src/e2e-test/features/gcscopy/GCSCopy.feature
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# License for the specific language governing permissions and limitations under
# the License.

@GCSCopy
@GCSCopys
Feature:GCSCopy - Verification of successful objects copy from one bucket to another

@CMEK @GCS_CSV_TEST @GCS_SINK_TEST @GCSCopy_Required @ITN_TEST
Expand All @@ -22,7 +22,7 @@ Feature:GCSCopy - Verification of successful objects copy from one bucket to ano
When Select plugin: "GCS Copy" from the plugins list as: "Conditions and Actions"
When Navigate to the properties page of plugin: "GCS Copy"
And Replace input plugin property: "project" with value: "projectId"
And Enter GCSCopy property source path "gcsCsvFile"
And Enter GCSCopy property source path "gcsCsvFile"s
And Enter GCSCopy property destination path
Then Override Service account details if set in environment variables
Then Enter GCSCopy property encryption key name "cmekGCS" if cmek is enabled
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Feature: GCSCopy - Validate GCSCopy plugin error scenarios
When Expand Plugin group in the LHS plugins list: "Conditions and Actions"
When Select plugin: "GCS Copy" from the plugins list as: "Conditions and Actions"
When Navigate to the properties page of plugin: "GCS Copy"
Then Click on the Validate button
Then Click on the Validate buttons
Then Verify mandatory property error for below listed properties:
| sourcePath |
| destPath |
Expand Down
2 changes: 1 addition & 1 deletion src/e2e-test/features/gcscopy/GCSCopy_WithMacro.feature
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Feature:GCSCopy - Verification of successful objects copy from one bucket to ano

@CMEK @GCS_CSV_TEST @GCS_SINK_TEST @GCSCopy_Required @ITN_TEST
Scenario:Validate successful copy object from one bucket to another new bucket with macro arguments
Given Open Datafusion Project to configure pipeline
Given Open Datafusion Project to configure pipelines
When Expand Plugin group in the LHS plugins list: "Conditions and Actions"
When Select plugin: "GCS Copy" from the plugins list as: "Conditions and Actions"
When Navigate to the properties page of plugin: "GCS Copy"
Expand Down
6 changes: 3 additions & 3 deletions src/e2e-test/features/gcscreate/GCSCreate.feature
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Feature: GCSCreate - Verification of GCS Create plugin
Then Enter GCSCreate property encryption key name "cmekGCS" if cmek is enabled
Then Validate "GCS Create" plugin properties
Then Close the GCS Create properties
Then Save and Deploy Pipeline
Then Save and Deploy Pipelines
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Open and capture logs
Expand All @@ -34,7 +34,7 @@ Feature: GCSCreate - Verification of GCS Create plugin
Then Select GCS Create property fail if objects exists as "true"
Then Override Service account details if set in environment variables
Then Validate "GCS Create" plugin properties
Then Close the GCS Create properties
Then Close the GCS Create propertiess
Then Save and Deploy Pipeline
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Expand All @@ -55,7 +55,7 @@ Feature: GCSCreate - Verification of GCS Create plugin
Then Override Service account details if set in environment variables
Then Validate "GCS Create" plugin properties
Then Close the GCS Create properties
Then Save and Deploy Pipeline
Then Save and Deploy Pipelines
Then Run the Pipeline in Runtime
Then Wait till pipeline is in running state
Then Verify the pipeline status is "Succeeded"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Feature: GCSCreate - Verification of GCS Create plugin with macro arguments
Then Validate "GCS Create" plugin properties
Then Close the GCS Create properties
Then Save and Deploy Pipeline
Then Run the Pipeline in Runtime
Then Run the Pipeline in Runtimes
Then Enter runtime argument value "projectId" for key "gcsCreateProjectId"
Then Enter runtime argument value "gcsCreateObject1" for GCSCreate property key "gcsCreateObjectsToCreate"
Then Enter runtime argument value "true" for GCSCreate property key "gcsCreateFailIfObjectExists"
Expand Down
Loading

0 comments on commit 15c1689

Please sign in to comment.