Add a topic guide for the RESTClient class
#1151
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| @@ -0,0 +1,259 @@ | |||
| # Reading data from RESTful APIs | |||
|
|
|||
| RESTful APIs are a common way to interact with web services. They are based on the HTTP protocol and are used to read and write data from and to a web server. dlt provides a simple way to read data from RESTful APIs using two helper methods: [a wrapper around the `requests`](../reference/performance#using-the-built-in-requests-client) library and a `RESTClient` class. | |||
There was a problem hiding this comment.
I would suggest splitting this in two pages:
- RESTClient -- takes care of pagination, authentification, customization
- requests module (migrate stuff from performance to separate page) -- takes care of retries
Maybe even abstract the requests helper and make retry customization available through RESTClient so users have one entry point.
|
|
||
| ## Quick example | ||
|
|
||
| Here's a simple pipeline that reads issues from the [dlt GitHub repository](https://github.com/dlt-hub/dlt/issues). The API endpoint is `https://api.github.com/repos/dlt-hub/dlt/issues`. The result is "paginated", meaning that the API returns a limited number of issues per page. The `paginate()` iterates over all pages and yields the results which are then processed by the pipeline. |
There was a problem hiding this comment.
I would add that in this example authentification and pagination are guessed automatically. And after this example put another one that shows how you can specify the paginator and auth if they are special/not guessed correctly.
Honestly, these two examples would be enough to give users an understanding of how to use RESTClient.
So overall I suggest:
- Quick example simple
- Quick example with specific paginator/auth
- Available Paginators
- How to write my own if nothing fits
- Available Auths
- How to write my own if nothing fits
- Advanced usage
In this case user WOW with this doc would be:
- Copy-past code which guesses everything automatically
- If it doesn't work, copy-past code with a specification of pagination and auth.
- Search in docs which auth and paginator is right for you
- If nothing fits -> How-to write custom auth/paginator.
rudolfix
added
the
sprint
@rudolfix which one? |
There was a problem hiding this comment.
We do not explain data_selector. It requires a separate chapter because it does a lot of detection magic. this must be explained - at least briefly
IMO we should do a followup ticket on this:
- I agree with @VioletM - we should explain the pagination detection better and say when it works (links paginator, next field at known places) and when it does not (offset pagination using non standard names) and what happens if we cannot detect.
- more in depth on
data_selectorresponse detections - I'd add an example with incremental loading. this will be the most common "advanced" task people have
…elector and custom auth
burnash
force-pushed
the
enh/docs/rest_client
branch
from
ae5508d to
9a3d9bb
Compare
|
|
||
| `JSONResponsePaginator` is designed for APIs where the next page URL is included in the response's JSON body. This paginator uses a JSON path to locate the next page URL within the JSON response. | ||
|
|
||
| ##### Parameters: |
There was a problem hiding this comment.
NIT:
The 5th and 6th headings looks very strange, smaller than a regular text :D
Maybe just use bold?
Co-authored-by: VioletM <[email protected]>
rudolfix
removed
the
sprint
Description
This PR adds documentation for the
RESTClientclass, including a brief example and instructions on how to handle paginated endpoints in APIs.Related Issues