Add RESTClient

[burnash]

Description

This pull request introduces a RESTClient class along with supporting components to interact with RESTful APIs. The RESTClient provides:

• an interface for making HTTP requests,
• handling pagination (via a set of paginator classes: e.g. HeaderLinkPaginator, JSONResponsePaginator.
• managing authentication (via BearerTokenAuth, HttpBasicAuth, etc.)

Users are able to customize request handling, response parsing, and data extraction according to the specifics of the target API. RESTClient also features automatic paginator detection with the PaginatorFactory

Example usage:

import dlt
from dlt.sources.helpers.rest_client import RESTClient

github_client = RESTClient(base_url="https://api.github.com")

@dlt.resource(
    table_name="issues",
    write_disposition="merge",
    primary_key="id",
)
def get_issues(
    updated_at=dlt.sources.incremental(
        "updated_at", initial_value="1970-01-01T00:00:00Z"
    )
):
    for page in github_client.paginate(
        "/repos/dlt-hub/dlt/issues",
        params={
            "since": updated_at.last_value,
            "per_page": 100,
            "sort": "updated",
            "direction": "desc",
            "state": "open",
        },
    ):
        yield page


# The rest of the pipeline remains the same
pipeline = dlt.pipeline(
    pipeline_name="github_issues_merge",
    destination="duckdb",
    dataset_name="github_data_merge",
)
load_info = pipeline.run(get_issues)
print(load_info)

Related Issues

• This is an extract from the rest_api source from Implement a basic generic API client verified-sources#313

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	b05b20c
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/6601c12cc8e5b100080976d6

[burnash]

@rudolfix I've put it in dlt.sources.helpers.rest_client module. Open for suggestions.

[rudolfix]

@burnash what I think is missing for internal testing is a simplified request like interface to access the rest client. the idea is to expose it from request helper.

from source.helpers import requests

requests.paginate(url, method, auth=str | AuthBase | AuthBase(), paginator = str | Paginator | Paginator(), ....)

auth and paginators follow the same convention as destinations, progress, naming convention

str - a short hand alias ie or type
type or instance of a type
types have aliases identical to short hand ie basic is basic auth shorthand alias and a type. basic(user, password) creates BasicAuthenticator

auth and paginators can be configured

[source.chargebee.paginator]
cursor_field="..."
cursor_param="cursor"

[source.chargebee.auth]
api_key="...."

where to put REST client: current location is good. let's write some code and maybe we'll change it
I'll review the actual code tomorrow

[rudolfix]

this is good! we are missing

1. tests for requests.paginated shorthand
2. some initial documentation (maybe page in docs but not linked to index yet)

[rudolfix]

LGTM! let' work on it in followup PRs

[burnash]

@rudolfix

1. tests for requests.paginated shorthand

There are tests for that in the test_client.py but I think it's best to have a separate file for that.

[burnash]

@rudolfix

tests for requests.paginated shorthand

Refactored from test_client.py in b1ee49a

[rudolfix]

pls see one comment

[rudolfix]

LGTM!