E2E GCS Sink additional test scenarios. #1478
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
| @@ -265,3 +265,175 @@ Feature: GCS sink - Verification of GCS Sink plugin | |||
| Then Open and capture logs | |||
| Then Verify the pipeline status is "Succeeded" | |||
| Then Verify data is transferred to target GCS bucket | |||
|
|
|||
| #Added new scenarios for GCS Sink - Bijay | |||
There was a problem hiding this comment.
Remove the commented line here.
|
|
||
| #Added new scenarios for GCS Sink - Bijay | ||
| @BQ_SOURCE_TEST @GCS_SINK_TEST | ||
| Scenario:Validate successful records transfer from BigQuery to GCS with macro enabled at sink |
There was a problem hiding this comment.
Add the macro scenario in a separate feature file with name macro, refer other plugins feature file for naming convention.
| Then Verify data is transferred to target GCS bucket | ||
|
|
||
| @GCS_SINK_TEST @BQ_SOURCE_TEST | ||
| Scenario Outline: To verify data is getting transferred successfully from BigQuery to GCS with contenttype selection |
There was a problem hiding this comment.
Add the validation for file format as well
| | tsv | text/plain | | ||
|
|
||
| @BQ_SOURCE_TEST @GCS_SINK_TEST | ||
| Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field |
There was a problem hiding this comment.
Why are we adding macro here again, It is already covered in macro enabled scenario. It should be for without macro enabled
| @BQ_SOURCE_TEST @GCS_SINK_TEST | ||
| Scenario: To verify data is getting transferred successfully from BigQuery to GCS using advanced file system properties field | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Source is BigQuery |
There was a problem hiding this comment.
Use the latest existing steps. This is a common review comment across all scenarios.
| Then Close the GCS properties | ||
| Then Save the pipeline | ||
| Then Preview and run the pipeline | ||
| Then Enter runtime argument value "gcsFileSysProperty" for key "FileSystemPr" |
There was a problem hiding this comment.
I don't see any value added in parameter file for file system property
| Then Close the preview | ||
| Then Deploy the pipeline | ||
| Then Run the Pipeline in Runtime | ||
| Then Enter runtime argument value "projectId" for key "gcsProjectId" |
There was a problem hiding this comment.
Remove the properties from the macro which are already covered in the scenarios . for eg-projectID is already covered.
| @@ -33,4 +33,5 @@ errorMessageMultipleFileWithoutClearDefaultSchema=Found a row with 4 fields when | |||
| errorMessageInvalidSourcePath=Invalid bucket name in path 'abc@'. Bucket name should | |||
| errorMessageInvalidDestPath=Invalid bucket name in path 'abc@'. Bucket name should | |||
| errorMessageInvalidEncryptionKey=CryptoKeyName.parse: formattedString not in valid format: Parameter "abc@" must be | |||
| errorMessageInvalidBucketNameSink=Spark program 'phase-1' failed with error: Errors were encountered during validation. Error code: 400, Unable to read or access GCS bucket. Bucket names must be at least 3 characters in length, got 2: 'gg'. Please check the system logs for more details. | |||
There was a problem hiding this comment.
Add only relevant error message.
| Then Verify data is transferred to target GCS bucket | ||
|
|
||
| @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required | ||
| Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true |
There was a problem hiding this comment.
This scenario should be from GCS source to GCS sink right? Re-check and change accordingly. And why are we making it a macro scenarios, it is already covered in macro enabled scenario anyways.
| @GCS_SINK_TEST @BQ_SOURCE_TEST @GCS_Sink_Required | ||
| Scenario Outline: To verify successful data transfer from BigQuery to GCS for different formats with write header true | ||
| Given Open Datafusion Project to configure pipeline | ||
| When Source is BigQuery |
There was a problem hiding this comment.
Use the latest existing steps from framework. Change in all the scenarios
| Then Wait till pipeline is in running state | ||
| Then Open and capture logs | ||
| Then Verify the pipeline status is "Succeeded" | ||
| Then Verify data is transferred to target GCS bucket |
There was a problem hiding this comment.
Add the validation steps in all the scenarios.
| @@ -65,3 +65,37 @@ Feature: GCS sink - Verify GCS Sink plugin error scenarios | |||
| Then Select GCS property format "csv" | |||
| Then Click on the Validate button | |||
| Then Verify that the Plugin Property: "format" is displaying an in-line error message: "errorMessageInvalidFormat" | |||
|
|
|||
| @GCS_SINK_TEST @BQ_SOURCE_TEST | |||
There was a problem hiding this comment.
Change the tag order, for ease of understanding.
| Then Enter runtime argument value "gcsInvalidBucketNameSink" for key "gcsSinkPath" | ||
| Then Run the Pipeline in Runtime with runtime arguments | ||
| Then Wait till pipeline is in running state | ||
| Then Verify the pipeline status is "Failed" |
There was a problem hiding this comment.
Add open and capture logs step
No description provided.