new scenarios e2e bq #26

priyabhatnagar25 · 2023-10-06T06:40:32Z

PR consists of scenarios from ITN Classes

itsmekumari · 2023-10-09T11:12:12Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

@@ -941,4 +941,259 @@ public static void createBucketWithMultipleTestFilesWithRegex() throws IOExcepti
    gcsSourceBucketName = createGCSBucketWithMultipleFiles(PluginPropertyUtils.pluginProp(
      "gcsMultipleFilesFilterRegexPath"));
  }
+
+  @Before(order = 1, value = "@BQ_UPSERT_SOURCE_TEST")


I don't see the After hooks for all before hooks, Please Add the after hooks for all of them, to cleanup the created tables after the tests execution.

Added the after hooks for all the before hooks

rahuldash171 · 2023-10-09T11:38:46Z

Please change the description .

itsmekumari · 2023-10-09T11:44:25Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Click on the Browse button inside plugin properties
+    Then Select connection data row with name: "dataset"
+    Then Select connection data row with name: "bqSourceTable"
+    Then Wait till connection data loading completes with a timeout of 60 seconds


Please check in all the scenarios, why do we need this step? If not required please remove this step.

This step is used to for the connection is created and used in all the scenarios that are using use connection functionality . This is need to check the connection is loaded properly or not.In earlier 5 scenarios for BQ I have used this step and got approved

itsmekumari · 2023-10-09T11:57:33Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Open and capture logs
+    Then Close the pipeline logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Validate the data transferred from BigQuery to BigQuery with actual And expected file for: "bqUpdatededupeExpectedFile"


Use camelcase for Parameter.

Changed to camelcase

itsmekumari · 2023-10-09T12:05:58Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Verify input plugin property: "dataset" contains value: "dataset"
+    Then Enter input plugin property: "table" with value: "bqTargetTable"
+    And Select radio button plugin property: "operation" with value: "update"
+    Then Enter Value for plugin property table key : "relationTableKey" with values: "Name"


Mention the value in plugin parameter file and use from there.

added the value in plugin file

itsmekumari · 2023-10-09T12:06:14Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    And Select radio button plugin property: "operation" with value: "update"
+    Then Enter Value for plugin property table key : "relationTableKey" with values: "Name"
+    Then Select dropdown plugin property: "dedupeBy" with option value: "dedupeByOrder"
+    Then Enter key for plugin property: "dedupeBy" with values: "ID"


Mention the value in plugin parameter file and use from there.

itsmekumari · 2023-10-09T12:24:51Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

+    PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable);
+    BeforeActions.scenario.write("BQ source table name - " + bqSourceTable);
+    BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " +
+                                        "(ID STRING, transaction_date DATE, Firstname STRING)");


Add Datetime and Timestamp columns to use them to cover the Datetime and Timestamp column scenarios.

Added the steps

itsmekumari · 2023-10-09T12:31:00Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

+    BeforeActions.scenario.write("BQ Target Table " + bqTargetTable + " updated successfully");
+  }
+
+  @Before(order = 1, value = "@BQ_TIME_STAMP_SOURCE_TEST")


Please change the Hook name @BQ_TIME_SOURCE_TEST (which is to be common hook to cover Date/Datetime/Timestamp columns)

Added the separate steps

itsmekumari · 2023-10-09T12:37:34Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Click on the Add Button of the property: "relationTableKey" with value:
+      | TableKeyDedupe |
+    Then Select dropdown plugin property: "dedupeBy" with option value: "dedupeBy"
+    Then Enter key for plugin property: "dedupeBy" with values: "Price"


Mention in Parameter file and use it from there.

Done, added in plugin parameter

itsmekumari · 2023-10-09T12:51:36Z

src/e2e-test/java/io/cdap/plugin/bigquery/runners/sinkrunner/TestRunner.java

@@ -27,7 +27,8 @@
  features = {"src/e2e-test/features"},
  glue = {"io.cdap.plugin.bigquery.stepsdesign", "io.cdap.plugin.gcs.stepsdesign",
    "stepsdesign", "io.cdap.plugin.common.stepsdesign"},
-  tags = {"@BigQuery_Sink"},
+  tags = {"@BigQuery_Sink not @CDAP-20830"},


it should be 'and not'

Changed to and not

rahuldash171

Please go though all the comments and change as per them

rahuldash171 · 2023-10-09T11:41:49Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Click on the Browse button inside plugin properties
+    Then Select connection data row with name: "dataset"
+    Then Select connection data row with name: "bqSourceTable"
+    Then Wait till connection data loading completes with a timeout of 60 seconds


Why do we need this , default timeout is already there for 180 sec .

This is already used in the use connection scenarios for BQ

rahuldash171 · 2023-10-09T11:43:11Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Enter input plugin property: "referenceName" with value: "BQSinkReferenceName"
+    Then Click on the Browse button inside plugin properties
+    Then Click SELECT button inside connection data row with name: "dataset"
+    Then Wait till connection data loading completes with a timeout of 60 seconds


if not needed then please remove , and what happens if it load the data before 60 seconds , does it wait till 1 min gets over or go to the next step . ?

This is already used in the use connection scenarios for BQ

rahuldash171 · 2023-10-09T11:46:47Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Validate the data transferred from BigQuery to BigQuery with actual And expected file for: "bqUpsertExpectedFile"
+
+  @BQ_NULL_MODE_SOURCE_TEST @BQ_SINK_TEST @EXISTING_BQ_CONNECTION
+  Scenario: Validate Successful record transfer from BigQuery source plugin to BigQuery sink plugin with all null values in one column and few null values in different column.


Please change it to "Validate Successful record transfer from BigQuery source plugin to BigQuery sink plugin having all null values in one column and few null values in another column in Source table ."

Changed to above

rahuldash171 · 2023-10-09T11:52:40Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
+
+  @BQ_UPDATE_SOURCE_DEDUPE_TEST @BQ_UPDATE_SINK_DEDUPE_TEST @EXISTING_BQ_CONNECTION
+  Scenario: Verify successful record transfer from BigQuery source to BigQuery sink using advance operation update with dedupe property.


change it to with 'Dedupe By property'.

changed to Dedupe By property'

rahuldash171 · 2023-10-09T11:56:32Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Validate "BigQuery" plugin properties
+    And Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "BigQuery2"
+    Then Click plugin property: "useConnection"


Why are we using different values in step definition for selecting connection in source and sink ? It is present in every scenario .

This is already used in the use connection scenarios for BQ

@bharatgulati If switch-useConnection is used in Sink will it work seamlessly ? I believe you have used these before in both source and sink for recent merged GCS tests .

rahuldash171 · 2023-10-09T12:26:39Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Wait till connection data loading completes with a timeout of 60 seconds
+    Then Verify input plugin property: "dataset" contains value: "dataset"
+    Then Enter input plugin property: "table" with value: "bqTargetTable"
+    Then Enter input plugin property: "partitionByField" with value: "bqPartitionFieldTime"


this value is not present in pluginParam.properties , is it an old key/value which was previously used ?

This value is a old key value

rahuldash171 · 2023-10-09T12:33:06Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Validate the data transferred from BigQuery to BigQuery with actual And expected file for: "bqTimeStampExpectedFile"
+
+  @BQ_UPSERT_DEDUPE_SOURCE_TEST @BQ_UPSERT_DEDUPE_SINK_TEST @EXISTING_BQ_CONNECTION
+  Scenario:Validate successful records transfer from BigQuery source to BigQuery sink with Upsert operation with dedupe source data and destination table already exists and update table schema is false


change to " Validate successful records transfer from BigQuery source to BigQuery sink with Upsert operation with dedupe source data and existing destination table where update table schema is set to false"

rahuldash171 · 2023-10-09T12:50:31Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Click on the Get Schema button
+    Then Click on the Validate button
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "BigQuery3"


Why are we not following the above steps for bigQuery2?

added BQ 2 source with hook

added the BQ 2 hook

rahuldash171 · 2023-10-09T13:01:20Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+   *
+   * @param json The JSON object to search for the ID key.
+   */
+  private static String getIdKey(JsonObject json) {


why is this method is needed ? getIdkey() ? @bharatgulati Can you please look at this

This we used earlier also in wrangler, and BQ 5 scenarios, PR got merged

rahuldash171 · 2023-10-09T13:07:18Z

src/e2e-test/resources/testdata/BigQuery/BigQueryToBigQuerySink.feature

Delete this file from testData , please avoid these type of mistakes .

deleted the file

itsmekumari · 2023-10-09T13:12:08Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Open and capture logs
+    Then Close the pipeline logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Validate the data transferred from BigQuery to BigQuery with actual And expected file for: "bqTimeStampExpectedFile"


Since the test is for 'Date' datatype use relevant date related parameter value. And similarly for other Datetime an Timestamp for other tests to be added.

renamed the file for date, date time and timestamp

rahuldash171

Reviewed , please look into .

rahuldash171 · 2023-10-12T13:08:18Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Validate "BigQuery" plugin properties
+    And Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "BigQuery2"
+    Then Click plugin property: "useConnection"


@bharatgulati If switch-useConnection is used in Sink will it work seamlessly ? I believe you have used these before in both source and sink for recent merged GCS tests .

rahuldash171 · 2023-10-12T13:14:02Z

src/e2e-test/resources/pluginParameters.properties

 dedupeBy=DESC
 TableKeyDedupe=Name
 Directive_Drop=testdata/BigQuery/test_diffschema_record-cdap-data-pipeline.json
 bqUpsertDedupeFile=testdata/BigQuery/BQUpsertDedupeFile
 bqDifferentRecordFile=testdata/BigQuery/BQDifferentRecordNameFile
+bqDateExpectedFile=testdata/BigQuery/BQTimeStampFile


Use proper naming convention , Change the file name from 'BQTimeStampFile' to 'BQDateFile'

Resolved changed the name testdata/BigQuery/BQDateFile

rahuldash171 · 2023-10-12T13:27:19Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

-  @BQ_TIME_STAMP_SOURCE_TEST @BQ_SINK_TEST @EXISTING_BQ_CONNECTION
-  Scenario: Verify successful record transfer for the Insert operation from BigQuery source plugin to BigQuery sink with partition type Time.
+  @BQ_TIME_SOURCE_TEST @BQ_SINK_TEST @EXISTING_BQ_CONNECTION
+  Scenario: Verify successful record transfer for the Insert operation from BigQuery source plugin to BigQuery sink with partition type Time and partition field is date.


bqPartitionFieldTime is used in line 435 but the scenario is related to partition filed is date , Can you add a value like 'bqPartitionFieldDate' in pluginParam properties and use the same for this scenario .

Added the field bqPartitionFieldDate

rahuldash171 · 2023-10-12T13:37:19Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

@@ -270,7 +271,11 @@ public static void createTempSourceBQTable() throws IOException, InterruptedExce
  }

  @After(order = 1, value = "@BQ_SOURCE_TEST or @BQ_PARTITIONED_SOURCE_TEST or @BQ_SOURCE_DATATYPE_TEST or " +
-    "@BQ_INSERT_SOURCE_TEST or @BQ_UPDATE_SINK_TEST")
+    "@BQ_INSERT_SOURCE_TEST or @BQ_UPDATE_SINK_TEST or @BQ_UPSERT_SOURCE_TEST or @BQ_UPSERT_SINK_TEST or " +


The before and after hooks counts are not matching . After are 15 but before count is 13 . Can you please reverify ?

reverified the after hooks

priyabhatnagar25 requested a review from itsmekumari October 6, 2023 13:30

itsmekumari reviewed Oct 9, 2023

View reviewed changes

rahuldash171 reviewed Oct 9, 2023

View reviewed changes

itsmekumari reviewed Oct 9, 2023

View reviewed changes

priyabhatnagar25 force-pushed the new_bq_e2e_test branch from 0818101 to 985c4a8 Compare October 11, 2023 14:12

rahuldash171 reviewed Oct 12, 2023

View reviewed changes

new scenarios e2e bq

62e6a10

priyabhatnagar25 force-pushed the new_bq_e2e_test branch from 62edd14 to 62e6a10 Compare October 13, 2023 10:00

new scenarios e2e bq review comments addressed

1c7df35

NavdeepCS force-pushed the develop branch 4 times, most recently from fc2efdf to 5f4f137 Compare March 1, 2024 06:20

vikasrathee-cs force-pushed the develop branch from 5f4f137 to 008f3a8 Compare March 1, 2024 06:27

new scenarios e2e bq #26

Are you sure you want to change the base?

new scenarios e2e bq #26

Conversation

priyabhatnagar25 commented Oct 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahuldash171 commented Oct 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Oct 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahuldash171 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahuldash171 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priyabhatnagar25 commented Oct 6, 2023 •

edited

Loading

itsmekumari Oct 9, 2023 •

edited

Loading