-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rest_api: add an example to the incremental load section #1502
rest_api: add an example to the incremental load section #1502
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
``` | ||
|
||
- `type`: The type of the incremental parameter. Set to `incremental`. | ||
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response. | |
- `cursor_path`: The JSONPath to the field within each item in the list that will be used as the cursor value. In this case, it's `created_at`. Note that the path starts from the root of the item (dict) and not from the root of the response. | |
`{ | |
"responses": [ | |
{ | |
"submissionId": "ggg", | |
"submissionTime": "2024-04-17T20:09:38.716Z", | |
"lastUpdatedAt": "2024-04-17T20:09:38.716Z", | |
}, | |
{ | |
"submissionId": "yyy", | |
"submissionTime": "2024-04-17T20:11:38.716Z", | |
"lastUpdatedAt": "2024-04-17T20:11:38.716Z", | |
} | |
] | |
}` | |
For example, in the response above, and the cursor path is inside the array. | |
Thus, we should modify the `data_selector` parameter to "responses", and the `cursor_path` to "submissionTime" | |
` "endpoint": { | |
"data_selector": "responses", | |
"params": { | |
"since": { | |
"type": "incremental", | |
"cursor_path": "submissionTime", | |
"initial_value": "2022-01-01T11:21:28Z", | |
}, | |
}, | |
},` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @omri-antman-imagen, thanks for the suggestion. I believe introducing a second JSON example would be confusing for the reader, so I suggest using only one example (e.g. "posts"). Nevertheless, I think I get your point, so I adjusted the text to really stress that the cursor_path
is within the data item and added an example to illustrate that in commit 70e43af. Please take a look.
fc3324f
to
9b1004d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very good Anton!
Adds an example and elaborates on incremental configuration following up the Slack community question:
https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1718889819509209