e2e BQ test scenarios new feature #17

priyabhatnagar25 · 2023-09-15T11:23:29Z

No description provided.

rahuldash171

Reviewed

rahuldash171 · 2023-09-15T12:31:12Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

+  public static void createSourceBQExistingTable() throws IOException, InterruptedException {
+    bqSourceTable = "E2E_SOURCE_" + UUID.randomUUID().toString().replaceAll("-" , "_");
+    io.cdap.e2e.utils.BigQueryClient.getSoleQueryResult("create table `" + datasetName + "." + bqSourceTable + "` " +
+                                                          "(CustomerID INT64, LastName STRING, " + "FirstName STRING," +


Table should include string, int, float, bool datatypes .(in both the tests) Go through the ITN class once and the buganizer ticket .https://buganizer.corp.google.com/issues/282611750 , https://github.com/cdapio/cdap-integration-tests/blob/develop/integration-test-remote/src/test/java/io/cdap/cdap/app/etl/gcp/GoogleBigQueryTest.java

done ...added the datatypes in the hook

itsmekumari · 2023-09-20T13:54:14Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

@@ -920,4 +920,46 @@ public static void createSourceBQUpdateTable() throws IOException, InterruptedEx
    PluginPropertyUtils.addPluginProp(" bqTargetTable",  bqTargetTable);
    BeforeActions.scenario.write("BQ Target Table " +  bqTargetTable + " updated successfully");
  }
+
+  @Before(order = 1, value = "@BQ_EXISTING_SOURCE_TEST")


Add the after hook as well.

itsmekumari · 2023-09-20T13:54:25Z

src/e2e-test/java/io/cdap/plugin/common/stepsdesign/TestSetupHooks.java

+    PluginPropertyUtils.addPluginProp("bqSourceTable", bqSourceTable);
+    BeforeActions.scenario.write("BQ Source Table " + bqSourceTable + " created successfully");
+  }
+  @Before(order = 1, value = "@BQ_EXISTING_SINK_TEST")


Add the after hook as well.

itsmekumari · 2023-09-20T13:56:31Z

src/e2e-test/resources/pluginParameters.properties

@@ -186,6 +186,7 @@ bqInvalidRefName=invalidRef&^*&&*
 bqDatatypeChange1=[{"key":"Id","value":"long"},{"key":"Value","value":"long"}]
 bqDataTypeTestFileSchema1=[{"key":"Id","value":"long"},{"key":"Value","value":"long"},\
  {"key":"UID","value":"string"}]
+bqexpectedfile=testdata/BigQuery/BQExistingTableFile


nit:camelcase

itsmekumari · 2023-09-21T12:34:11Z

src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature

@@ -262,3 +262,81 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
    Then Open and capture logs
    Then Verify the pipeline status is "Succeeded"
    Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
+
+  @BQ_EXISTING_SOURCE_TEST @BQ_EXISTING_SINK_TEST
+  Scenario: Validate successful record transfer from BigQuery source(existing table) to BigQuery sink(existing table)


All the scenarios heading needs to be updated as per ITN class for ease of scenario understanding. can be same as mentioned in ITN class.

scenarios are updated as per ITN class

rahuldash171

Reviewed , please address the comments.

rahuldash171 · 2023-09-21T12:05:52Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    And Replace input plugin property: "dataset" with value: "dataset"
+    Then Override Service account details if set in environment variables
+    And Enter input plugin property: "table" with value: "bqSourceTable"
+    Then Validate "BigQuery" plugin properties


Why are we not using 'get schema' before validating ? In case of tables this is an ideal step.

Resolved.Added schema step.

rahuldash171 · 2023-09-21T12:21:16Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    When Select plugin: "BigQuery" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
+    Then Navigate to the properties page of plugin: "BigQuery"
+    And Enter input plugin property: "referenceName" with value: "Reference"


We should use "useConnection:"true" here as per the ITN description .

Resolved. Scenarios are updated and use connection is used in all the scenarios

rahuldash171 · 2023-09-21T12:22:01Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "BigQuery" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
+    Then Navigate to the properties page of plugin: "BigQuery"


We should use "useConnection:"true" here as per the ITN description .

Resolved. Scenarios are updated and use connection is used in all the scenarios

rahuldash171 · 2023-09-21T12:22:55Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "BigQuery" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
+    Then Navigate to the properties page of plugin: "BigQuery"


We should use "useConnection:"true" here as per the ITN description .

rahuldash171 · 2023-09-21T12:26:22Z

src/e2e-test/features/bigquery/sink/BigQueryToBigQuerySink.feature

+    Then Verify the pipeline status is "Succeeded"
+    Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
+
+  @BQ_INSERT_SOURCE_TEST @BQ_SINK_TEST


Please use correct hook , as per the ITN the sink table is an existing table with data .

This is the upsert button, so it will insert the data in the new table created

rahuldash171 · 2023-09-21T12:43:05Z

src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature

@@ -262,3 +262,81 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
    Then Open and capture logs
    Then Verify the pipeline status is "Succeeded"
    Then Validate the values of records transferred to BQ sink is equal to the values from source BigQuery table
+
+  @BQ_EXISTING_SOURCE_TEST @BQ_EXISTING_SINK_TEST
+  Scenario: Validate successful record transfer from BigQuery source(existing table) to BigQuery sink(existing table)


for this scenario use connection is true for both source and sink in ITN , please use .

rahuldash171 · 2023-09-21T12:47:52Z

src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature

+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "BigQuery" from the plugins list as: "Sink"
+    Then Connect plugins: "BigQuery" and "BigQuery2" to establish connection
+    Then Navigate to the properties page of plugin: "BigQuery"


useConnection should be true as per ITN.

rahuldash171 · 2023-09-21T12:54:03Z

src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature

+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Validate the data transferred from BQ to BQ with actual And expected file for: "bqexpectedfile"


This will fail as it does not match the key in plugin properties bqExpectedFile , Have you executed in local ?

rahuldash171 · 2023-09-21T12:56:04Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+/**
+ * BigQuery Plugin Existing Table validation.
+ */
+public class BQValidationExistingTables {


@bharatgulati Please review the below logic for validation .

Change the class name to something relevant like BQTableValidation or something like that.

rahuldash171 · 2023-09-21T12:57:18Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+        }
+      }
+    } catch (IOException e) {
+      System.err.println("Error reading the file: " + e.getMessage());


add logger.error instead of sysout

bharatgulati · 2023-09-22T07:28:26Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+public class BQValidationExistingTables {
+
+  private static final Logger LOG = LoggerFactory.getLogger(BQValidationExistingTables.class);
+  static Gson gson = new Gson();


Add access modifier of Gson

added modifier

bharatgulati · 2023-09-22T07:29:18Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+    return isMatched;
+  }
+
+  public static void getFileData(String fileName, Map<String, JsonObject> fileMap) {


Add a java doc for this function

bharatgulati · 2023-09-22T07:29:28Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+
+  private static final Logger LOG = LoggerFactory.getLogger(BQValidationExistingTables.class);
+  static Gson gson = new Gson();
+  public static boolean validateActualDataToExpectedData(String table, String fileName) throws IOException,


Add a java doc for this function

added javadoc

bharatgulati · 2023-09-22T07:30:28Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+    getBigQueryTableData(table, bigQueryMap);
+    getFileData(importbqexpectedfile.toString(), fileMap);
+
+    boolean isMatched = matchJsonMaps(bigQueryMap, fileMap);


remove matchJsonMaps function and match bigqueryMap and fileMap with equals function.

bharatgulati · 2023-09-22T07:30:42Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+    if (!map1.keySet().equals(map2.keySet())) {
+      return false;
+    }
+    for (String key : map1.keySet()) {


Remove this function

bharatgulati · 2023-09-22T07:31:20Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BigQuerySource.java

@@ -160,4 +161,13 @@ public void enterTheBigQuerySourcePropertyForViewMaterializationDataset(String v
  public void enterBigQuerySourcePropertyTableNameAsView() {
    CdfBigQueryPropertiesActions.enterBigQueryTable(TestSetupHooks.bqSourceView);
  }
+
+  @Then("Validate the data transferred from BQ to BQ with actual And expected file for: {string}")
+  public void validateTheDataFromBQToBQWithActualAndExpectedFileFor(String expectedFile) throws IOException,


Rename it to BigQuery from BQ

bharatgulati · 2023-09-22T07:35:18Z

src/e2e-test/java/io/cdap/plugin/bigquery/stepsdesign/BQValidationExistingTables.java

+    InterruptedException, URISyntaxException {
+    Map<String, JsonObject> bigQueryMap = new HashMap<>();
+    Map<String, JsonObject> fileMap = new HashMap<>();
+    Path importbqexpectedfile = Paths.get(BQValidationExistingTables.class.getResource("/" + fileName).toURI());


Rename importbqexpectedfile to bqExpectedFilePath or just path

rahuldash171

Approved

e2e BQ new tests scenarios

rahuldash171 reviewed Sep 15, 2023

View reviewed changes

itsmekumari reviewed Sep 20, 2023

View reviewed changes

priyabhatnagar25 force-pushed the e2e_bq_tests branch from 8905a21 to aca8121 Compare September 21, 2023 05:34

itsmekumari reviewed Sep 21, 2023

View reviewed changes

rahuldash171 reviewed Sep 21, 2023

View reviewed changes

bharatgulati reviewed Sep 22, 2023

View reviewed changes

priyabhatnagar25 force-pushed the e2e_bq_tests branch 2 times, most recently from 1c1b2b5 to dc8764b Compare September 25, 2023 14:20

rahuldash171 approved these changes Sep 26, 2023

View reviewed changes

itsmekumari approved these changes Sep 26, 2023

View reviewed changes

priyabhatnagar25 force-pushed the e2e_bq_tests branch 3 times, most recently from 4cbaecd to 6b78689 Compare September 26, 2023 17:42

e2e BQ new tests scenarios

c5fb04d

e2e BQ new tests scenarios

priyabhatnagar25 force-pushed the e2e_bq_tests branch from 6b78689 to c5fb04d Compare September 27, 2023 06:56

NavdeepCS force-pushed the develop branch 4 times, most recently from fc2efdf to 5f4f137 Compare March 1, 2024 06:20

vikasrathee-cs force-pushed the develop branch from 5f4f137 to 008f3a8 Compare March 1, 2024 06:27

e2e BQ test scenarios new feature #17

Are you sure you want to change the base?

e2e BQ test scenarios new feature #17

Conversation

priyabhatnagar25 commented Sep 15, 2023

rahuldash171 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahuldash171 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rahuldash171 left a comment

Choose a reason for hiding this comment

itsmekumari Sep 21, 2023 •

edited

Loading