Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rest_api: add an example to the incremental load section #1502

Merged
merged 2 commits into from
Jun 24, 2024

Conversation

burnash
Copy link
Collaborator

@burnash burnash commented Jun 20, 2024

Adds an example and elaborates on incremental configuration following up the Slack community question:
https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1718889819509209

@burnash burnash added the documentation Improvements or additions to documentation label Jun 20, 2024
@burnash burnash requested a review from sh-rp June 20, 2024 17:14
@burnash burnash self-assigned this Jun 20, 2024
Copy link

netlify bot commented Jun 20, 2024

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit 70e43af
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/667993a392484c0008f86668
😎 Deploy Preview https://deploy-preview-1502--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@burnash burnash requested a review from rudolfix June 20, 2024 17:16
```

- `type`: The type of the incremental parameter. Set to `incremental`.
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response.
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response.
`{
"responses": [
{
"submissionId": "ggg",
"submissionTime": "2024-04-17T20:09:38.716Z",
"lastUpdatedAt": "2024-04-17T20:09:38.716Z",
},
{
"submissionId": "yyy",
"submissionTime": "2024-04-17T20:11:38.716Z",
"lastUpdatedAt": "2024-04-17T20:11:38.716Z",
}
]
}`
For example, in the response above, and the cursor path is inside the array.
Thus, we should modify the `data_selector` parameter to "responses", and the `cursor_path` to "submissionTime"
` "endpoint": {
"data_selector": "responses",
"params": {
"since": {
"type": "incremental",
"cursor_path": "submissionTime",
"initial_value": "2022-01-01T11:21:28Z",
},
},
},`

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @omri-antman-imagen, thanks for the suggestion. I believe introducing a second JSON example would be confusing for the reader, so I suggest using only one example (e.g. "posts"). Nevertheless, I think I get your point, so I adjusted the text to really stress that the cursor_path is within the data item and added an example to illustrate that in commit 70e43af. Please take a look.

@AstrakhantsevaAA AstrakhantsevaAA self-assigned this Jun 21, 2024
@burnash burnash force-pushed the feat/docs/rest-api-add-example-to-incremental branch from fc3324f to 9b1004d Compare June 24, 2024 14:43
@burnash burnash requested a review from AstrakhantsevaAA June 24, 2024 15:49
Copy link
Contributor

@AstrakhantsevaAA AstrakhantsevaAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good Anton!

@burnash burnash merged commit ace102e into master Jun 24, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants