Skip to content

Commit

Permalink
docs: add hint about connection pooling
Browse files Browse the repository at this point in the history
  • Loading branch information
joscha authored Dec 17, 2024
1 parent 38d0dab commit a53c911
Showing 1 changed file with 27 additions and 0 deletions.
27 changes: 27 additions & 0 deletions docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,3 +693,30 @@ for page in client.paginate(
):
print(page)
```

## Performance

If you have a substantial amount of requests going to a single or smaller set of hosts, it can be worth to use connection pooling.
You can do this by leveraging the custom `session` parameter like so:

```py
from dlt.sources.helpers.rest_client.client import RESTClient
from dlt.sources.helpers.requests.session import Session
from requests.adapters import HTTPAdapter

base_url = "https://api.my-service.com"

session = Session(raise_for_status=False)
adapter = HTTPAdapter(
# We're only using one host for all requests in our example
pool_connections=1,
# But we're making a lot of requests and want to reuse connections
pool_maxsize=100,
)
session.mount(api_base, adapter)

client = RESTClient(
base_url=base_url,
session=session,
)
```

0 comments on commit a53c911

Please sign in to comment.