WIP: resolve bigquery cache datetime format issue #331
Conversation
WalkthroughWalkthroughThe new scripts Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Salesforce
participant BigQuery
participant Airbyte
participant SecretManager
User->>SecretManager: Request Credentials
SecretManager-->>User: Provide Salesforce and BigQuery Credentials
User->>Airbyte: Initiate Data Transfer
Airbyte->>Salesforce: Fetch Source Configuration
Airbyte->>Salesforce: Validate Configuration
Airbyte->>Salesforce: Retrieve Data Streams
Airbyte->>BigQuery: Create Temporary Credential File
Airbyte->>BigQuery: Initialize Cache
Airbyte->>Salesforce: Read Data Streams
Airbyte->>BigQuery: Cache Data
Airbyte->>User: Provide Data Transfer Summary
Does this summary and the table effectively capture the changes made? Let me know if you'd like to adjust anything! Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
…-fails-due-to-datetime-formats
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
aaronsteers
changed the title
There was a problem hiding this comment.
Actionable comments posted: 0
Outside diff range, codebase verification and nitpick comments (1)
examples/run_salesforce_to_bigquery_destination.py (1)
1-5: Consider updating the usage comment.The usage comment references
run_salesforce_to_bigquery_cache.py, but this file isrun_salesforce_to_bigquery_destination.py. Should this be updated to reflect the correct script name? Wdyt?- poetry run python examples/run_salesforce_to_bigquery_cache.py + poetry run python examples/run_salesforce_to_bigquery_destination.py
|
|
|
|
|
|
|
|
|
|
|
Reported in slack:
Log from repro in new "example" script:
google.api_core.exceptions.BadRequest: 400 Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.; reason: invalid, message: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.; reason: invalid, message: Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0; reason: invalid, message: Error while reading data, error message: JSON parsing error in row starting at position 0: Couldn't convert value to timestamp: Could not parse '2020-10-22T21:03:23.000+0000' as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]] or YYYY/MM/DD HH:MM[:SS[.SSSSSS]] Field: createddate; Value: 2020-10-22T21:03:23.000+0000BigQuery expects a colon in timezone part:
+00:00instead of+0000.Summary by CodeRabbit