Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(RESTClient): add hint about connection pooling #2160

Closed
wants to merge 2 commits into from
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,3 +693,30 @@ for page in client.paginate(
):
print(page)
```

## Performance

If you have a substantial amount of requests going to a single or smaller set of hosts, it can be worth to use connection pooling.
You can do this by leveraging the custom `session` parameter like so:

```py
from dlt.sources.helpers.rest_client.client import RESTClient
from dlt.sources.helpers.requests.session import Session
from requests.adapters import HTTPAdapter

base_url = "https://api.my-service.com"

session = Session(raise_for_status=False)
adapter = HTTPAdapter(
# We're only using one host for all requests in our example
pool_connections=1,
# But we're making a lot of requests and want to reuse connections
pool_maxsize=100,
)
session.mount(api_base, adapter)
joscha marked this conversation as resolved.
Show resolved Hide resolved

client = RESTClient(
base_url=base_url,
session=session,
)
```
Loading