Additional Steps for BQ_Source #1472

Praveena2607 · 2024-12-11T07:18:20Z

Please verify the Scenarios.

google-cla · 2024-12-11T07:18:25Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

masivesky

Please update the PR description to reflect details and context about the changes. For eg: #1444 (comment)

masivesky

The BUTs seems to be failing, please take a look and fix the errors.

…ugin: Added and modified.

Praveena2607 · 2024-12-12T06:04:51Z

Description:

Updated the BigQuery plugin to include additional steps for better handling.
Modified property locators in Cdfplugin property locators.java to support new BigQuery functionality.
Updated the errorMessage.properties and pluginParameters.properties file to ensure proper error handling and parameter management for the new configuration.

Modified Files:

Cdfplugin property locators.java
errorMessage.properties
pluginParameters.properties

File Modified:

src/e2e-test/java/io/cdap/plugin/utils/CdfPluginPropertyLocator.java

Unit Tests:

No changes made to unit tests.
Verified that all new test cases pass successfully.

masivesky · 2024-12-12T09:48:08Z

Build with unit tests is still failing, there are a bunch of checkstyle errors, for eg:

/tmp/google-cloud/google-cloud/src/e2e-test/java/io/cdap/plugin/utils/E2ETestConstants.java:20:68: '=' is not preceded with whitespace. [WhitespaceAround]

Please check and fix the same

…fully.

itsankit-google · 2024-12-16T06:02:18Z

pom.xml

@@ -86,7 +86,7 @@
    <google.cloud.storage.version>2.3.0</google.cloud.storage.version>
    <google.cloud.datastore.version>1.105.1</google.cloud.datastore.version>
    <google.protobuf.java.version>3.19.4</google.protobuf.java.version>
-    <google.tink.version>1.3.0-rc3</google.tink.version>
+    <google.tink.version>1.5.0</google.tink.version>


why this change?

I encountered dependency errors while running mvn clean install after resolving all the checkstyle issues. The build was failing due to the outdated version of google.tink.version. Updating it to 1.5.0 resolved the issue, and no other changes were made.

please do not make dependency changes in the e2e tests PR.

Refactor CdfPluginPropertyLocator alignment and update build dependencies.
-Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema.
-Updated pom.xml as it is.

itsankit-google · 2024-12-16T06:03:12Z

src/e2e-test/java/io/cdap/plugin/utils/CdfPluginPropertyLocator.java

+            .put("projectId", CdfPluginPropertyLocator.PROJECT_ID)
+            .put("datasetProjectId", CdfPluginPropertyLocator.DATASET_PROJECT_ID)
+            .put("dataset", CdfPluginPropertyLocator.DATASET)
+            .put("table", CdfPluginPropertyLocator.TABLE)
+            .put("format", CdfPluginPropertyLocator.FORMAT)
+            .put("path", CdfPluginPropertyLocator.PATH)
+            .put("sampleSize", CdfPluginPropertyLocator.SAMPLE_SIZE)
+            .put("delimiter", CdfPluginPropertyLocator.DELIMITER)
+            .put("skipHeader", CdfPluginPropertyLocator.SKIP_HEADER)
+            .put("pathSuffix", CdfPluginPropertyLocator.SUFFIX)
+            .put("encryptionKeyName", CdfPluginPropertyLocator.CMEK_KEY)
+            .put("serviceAccountType", CdfPluginPropertyLocator.SERVICE_ACCOUNT_TYPE)
+            .put("serviceAccountFilePath", CdfPluginPropertyLocator.SERVICE_ACCOUNT_PATH)
+            .put("serviceAccountJSON", CdfPluginPropertyLocator.SERVICE_ACCOUNT_JSON)
+            .put("truncateTable", CdfPluginPropertyLocator.TRUNCATE_TABLE)
+            .put("updateTableSchema", CdfPluginPropertyLocator.UPDATE_TABLE_SCHEMA)
+            .put("topic", CdfPluginPropertyLocator.PUBSUB_TOPIC)
+            .put("maximumBatchCount", CdfPluginPropertyLocator.PUBSUB_MAXIMUM_BATCH_COUNT)
+            .put("maximumBatchSize", CdfPluginPropertyLocator.PUBSUB_MAXIMUM_BATCH_SIZE)
+            .put("publishDelayThreshold", CdfPluginPropertyLocator.PUBSUB_PUBLISH_DELAY_THRESHOLD)
+            .put("retryTimeout", CdfPluginPropertyLocator.PUBSUB_RETRY_TIMEOUT)
+            .put("errorThreshold", CdfPluginPropertyLocator.PUBSUB_ERROR_THRESHOLD)
+            .put("outputSchema", CdfPluginPropertyLocator.OUTPUT_SCHEMA_MACRO_INPUT)
+            .put("objectsToDelete", CdfPluginPropertyLocator.GCS_DELETE_OBJECTS_TO_DELETE)
+            .put("objectsToCreate", CdfPluginPropertyLocator.GCS_CREATE_OBJECTS_TO_CREATE)
+            .put("createFailIfObjectExists", CdfPluginPropertyLocator.GCS_CREATE_FAIL_IF_OBJECT_EXISTS)
+            .put("gcsMoveSourcePath", CdfPluginPropertyLocator.GCS_MOVE_SOURCE_PATH)
+            .put("gcsMoveDestinationPath", CdfPluginPropertyLocator.GCS_MOVE_DESTINATION_PATH)
+            .put("partitionFrom", CdfPluginPropertyLocator.PARTITION_START_DATE)
+            .put("partitionTo", CdfPluginPropertyLocator.PARTITION_END_DATE)
+            .put("filter", CdfPluginPropertyLocator.FILTER)
+            .put("Output Schema-macro-input", CdfPluginPropertyLocator.OUTPUT_SCHEMA)
+            .build();


why this change?

I was unable to add my steps properly in this field because the alignments were incorrect. To fix this, I copied and pasted the same content and added my steps without making any other changes.

please fix the code style in your editor.

Refactor CdfPluginPropertyLocator alignment and update build dependencies.
-Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema.
-Updated pom.xml as it is.

itsmekumari · 2024-12-16T08:08:00Z

src/e2e-test/features/bigquery/source/BigQuerySourceError.feature

+    Then Replace input plugin property: "table" with value: "bqSourceTable"
+    Then Click on the Get Schema button
+    Then Enter BigQuery source properties partitionFrom and partitionTo
+    Then Validate BigQuery source incorrect property error for Partition Start date "<property>" value "<value>"


@Praveena2607 I don't think the internal review comments have been addressed. Please address all the internal review comments provided earlier in this PR as well. Ref Praveena2607#1

…cies. -Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema. -Updated pom.xml for as it is.

Additional Steps for BQ.

2f6ad0f

itsankit-google added the build Trigger unit test build label Dec 11, 2024

itsankit-google requested a review from masivesky December 11, 2024 07:25

masivesky reviewed Dec 11, 2024

View reviewed changes

Updated CdfPluginPropertyLocator and associated fiels for BigQuery Pl…

60fb70f

…ugin: Added and modified.

Fixed Checkstyle issues and ensured all scenarios are running success…

7e71bc2

…fully.

itsankit-google reviewed Dec 16, 2024

View reviewed changes

itsmekumari reviewed Dec 16, 2024

View reviewed changes

Refactor CdfPluginPropertyLocator alignment and update build dependen…

0870576

…cies. -Fixed indentation alignment for partitionFrom, partitionTo, filter and Output Schema. -Updated pom.xml for as it is.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Additional Steps for BQ_Source #1472

Additional Steps for BQ_Source #1472

Praveena2607 commented Dec 11, 2024 •

edited

Loading

google-cla bot commented Dec 11, 2024

masivesky left a comment

masivesky left a comment

Praveena2607 commented Dec 12, 2024

masivesky commented Dec 12, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsankit-google Dec 16, 2024

Praveena2607 Dec 16, 2024

itsmekumari Dec 16, 2024 •

edited

Loading

Additional Steps for BQ_Source #1472

Are you sure you want to change the base?

Additional Steps for BQ_Source #1472

Conversation

Praveena2607 commented Dec 11, 2024 • edited Loading

google-cla bot commented Dec 11, 2024

masivesky left a comment

Choose a reason for hiding this comment

masivesky left a comment

Choose a reason for hiding this comment

Praveena2607 commented Dec 12, 2024

masivesky commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itsmekumari Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Praveena2607 commented Dec 11, 2024 •

edited

Loading

itsmekumari Dec 16, 2024 •

edited

Loading