Added e2e tests with snapshot and cdc- Draft PR #103

yjhawar · 2023-04-04T18:47:58Z

No description provided.

sumitjnn · 2023-04-05T12:26:48Z

src/e2e-test/features/Pipeline.feature

+#  Scenario: Sanity test from Oracle to Big Query
+#    Given Open DataFusion Project with replication to configure pipeline
+#    When Enter input plugin property: "name" with value: "pipelineName"
+#    And Click on the Next button
+#    And Select Oracle as Source
+#    Then Replace input plugin property: "host" with value: "host" for Credentials and Authorization related fields
+#    Then Replace input plugin property: "port" with value: "port" for Credentials and Authorization related fields
+#    Then Click plugin property: "region"
+#    Then Click plugin property: "regionOption"
+#    Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
+#    Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
+#    Then Replace input plugin property: "sid" with value: "database" for Credentials and Authorization related fields
+#    Then Click on the Next button
+#    Then Replace input plugin property: "loadInterval" with value: "loadInterval"
+#    Then Click on the Next button
+#    Then Validate Table is available and can be selected "ABC.E2E-sanity"
+#    And Click on the Next button
+#    And Click on the Next button
+#    And Click on the Next button
+#    Then Deploy the replication pipeline
+#    And Run the replication Pipeline
+#    Then Open the logs
+#    And Wait till pipeline is in running state and check if no errors occurred
+#    Then Verify expected Oracle records in target BigQuery table
+#    And Capture raw logs
+#    Then Close the pipeline logs and stop the pipeline


Will this comment block be removed?

sumitjnn · 2023-04-05T12:27:14Z

src/e2e-test/features/Pipeline.feature

+#    And Capture raw logs
+#    Then Close the pipeline logs and stop the pipeline
+
+#  @ORACLE_SOURCE


What's the significance of this annotation? Why is it commented out?

sumitjnn · 2023-04-05T12:27:56Z

src/e2e-test/features/Pipeline.feature

+#    Then Close the pipeline logs and stop the pipeline
+
+#  @ORACLE_SOURCE
+  Scenario: To verify snapshot and cdc from Oracle to Big Query successfully


Suggested change

Scenario: To verify snapshot and cdc from Oracle to Big Query successfully

Scenario: To verify replication of snapshot and cdc data from Oracle to Big Query successfully

sumitjnn · 2023-04-05T12:33:16Z

src/e2e-test/resources/pluginParameters.properties

+datatypeColumnsForSanityTesting=('USER1', 'M','ABCDEF','ABC','ABC','�','� 好�?�?','� 好�?�?','AAAAaoAATAAABrXAAA',1234,1234.56789,\
+  1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.5679,\
+  1234.56789,1234.5679,1234.5679,1234.56789,TIMESTAMP'2023-01-01 2:00:00',TIMESTAMP'2023-01-01 2:00:00',\
+  TIMESTAMP '2023-01-01 00:00:00.000000')
+
+datatypeValuesForSanityTesting=('USER1', 'M','ABCDEF','ABC','ABC','�','� 好�?�?','� 好�?�?',\
+  'AAAAaoAATAAABrXAAA',1234,1234.56789,\
+  1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.56789,1234.5679,\
+ 1234.56789,1234.5679,1234.5679,1234.56789,TIMESTAMP'2023-01-01 2:00:00',TIMESTAMP'2023-01-01 2:00:00',\
+  TIMESTAMP '2023-01-01 00:00:00.000000')


What's the reason for storing these in property file? Are we storing rows and types like this for existing e2e tests?

Yes.For connectors, it is done this way.

sumitjnn · 2023-04-05T12:37:31Z

src/e2e-test/java/io.cdap.plugin/actions/ReplicationActions.java

+
+    public static void waitTillPipelineIsRunningAndCheckForErrors() throws InterruptedException {
+        //wait for datastream to startup
+        int defaultTimeout = Integer.parseInt(PluginPropertyUtils.pluginProp("datastream.timeout"));


Rename it to more general term like pipeline initialization time instead of datastream.timeout

sumitjnn · 2023-04-05T13:11:36Z

src/e2e-test/java/io.cdap.plugin/utils/ValidationHelper.java

+            bqRow.remove("_is_deleted");
+            bqRow.remove("_sequence_num");
+
+            ValidationHelper.compareBothResponses( bqRow, oracleRecord);


Suggested change

ValidationHelper.compareBothResponses( bqRow, oracleRecord);

compareRecords( bqRow, oracleRecord);

sumitjnn · 2023-04-05T13:14:49Z

src/e2e-test/java/io.cdap.plugin/hooks/TestSetUpHooks.java

+        if (port!= null && !port.isEmpty()) {
+            PluginPropertyUtils.addPluginProp("port", port);
+        }
+        String sapHost = System.getenv("ORACLE_HOST");


sumitjnn · 2023-04-05T13:15:27Z

src/e2e-test/java/io.cdap.plugin/hooks/TestSetUpHooks.java

+    @Before(order = 1, value = "@ORACLE_SOURCE")
+    public static void overridePropertiesFromEnvVarsIfProvided() {
+        String username = System.getenv("ORACLE_USERNAME");
+        if (username != null && !username.isEmpty()) {


what happens if property is null?

test will not run like username is used for creating tables, that will fail thereby ending tests

sumitjnn · 2023-04-05T13:17:14Z

src/e2e-test/java/io.cdap.plugin/hooks/TestSetUpHooks.java

+
+    @Before(order = 4, value = "@ORACLE_SOURCE")
+    public static void getOracleRecordsAsMap() throws SQLException, ClassNotFoundException {
+        sourceOracleRecords = OracleClient.getOracleRecordsAsMap(tableName, schemaName);


This is for logging expected records before test starts?

sumitjnn · 2023-04-05T13:17:47Z

src/e2e-test/java/io.cdap.plugin/hooks/TestSetUpHooks.java

+
+  @After(order = 1, value = "@ORACLE_SOURCE_TEMP")
+  public static void dropTables() throws SQLException, ClassNotFoundException {
+    OracleClient.deleteTables(schemaName, tableName);


Should we drop BQ table as well?

Yes we can. But having target table is good for validation after e2e tests are done

sumitjnn · 2023-04-10T12:48:11Z

src/e2e-test/features/Pipeline.feature

@@ -0,0 +1,50 @@
+#


I think it would be better to rename the file to SanityTests.feature to reflect the intent

yjhawar requested a review from sumitjnn April 4, 2023 18:48

yjhawar added the build label Apr 4, 2023

Added e2e tests with snapshot and cdc

fd593c4

yjhawar force-pushed the e2e branch from 491d0ff to fd593c4 Compare April 5, 2023 04:59

sumitjnn reviewed Apr 5, 2023

View reviewed changes

yjhawar and others added 3 commits April 6, 2023 19:05

Refactored the code.

c173246

Add build helper plugin to configure test sources for e2e profile

6442266

Add build helper plugin to configure test sources for e2e profile

700cf89

sumitjnn reviewed Apr 10, 2023

View reviewed changes

resolved comments

28bcdf6

yjhawar force-pushed the e2e branch from 20bf406 to 28bcdf6 Compare April 11, 2023 09:19

Add github workflow

db19820

yjhawar force-pushed the e2e branch from 3a98914 to db19820 Compare April 11, 2023 09:50

yjhawar added 2 commits April 11, 2023 16:36

fixed build issues

8106d03

Fixed checkstyle issues and minor fix

5294941

yjhawar force-pushed the e2e branch from 3cde584 to 5294941 Compare April 11, 2023 12:47

corrected path for feature

38df210

yjhawar force-pushed the e2e branch from 9781d4f to 38df210 Compare April 12, 2023 09:04

yjhawar and others added 3 commits April 12, 2023 16:08

Removing runner path

3fed35f

add correct tag to runner file

3755349

Fix cucumber tests not executing due to junit 5

9891050

sumitjnn force-pushed the e2e branch from a1e8cb0 to 9891050 Compare April 13, 2023 04:16

yjhawar added 2 commits April 13, 2023 12:55

added secret manager values

62167e0

added different project id

8472625

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added e2e tests with snapshot and cdc- Draft PR #103

Added e2e tests with snapshot and cdc- Draft PR #103

yjhawar commented Apr 4, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

yjhawar Apr 6, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

sumitjnn Apr 5, 2023

yjhawar Apr 6, 2023

sumitjnn Apr 5, 2023

yjhawar Apr 6, 2023

sumitjnn Apr 5, 2023

yjhawar Apr 6, 2023

sumitjnn Apr 10, 2023

	Scenario: To verify snapshot and cdc from Oracle to Big Query successfully
	Scenario: To verify replication of snapshot and cdc data from Oracle to Big Query successfully

	ValidationHelper.compareBothResponses( bqRow, oracleRecord);
	compareRecords( bqRow, oracleRecord);

Added e2e tests with snapshot and cdc- Draft PR #103

Are you sure you want to change the base?

Added e2e tests with snapshot and cdc- Draft PR #103

Conversation

yjhawar commented Apr 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment