Skip to content

Commit

Permalink
docs/505 updates documentation on custom hooks in response_actions (#…
Browse files Browse the repository at this point in the history
  • Loading branch information
willi-mueller authored Jul 18, 2024
1 parent 9db6650 commit 501df76
Show file tree
Hide file tree
Showing 2 changed files with 60 additions and 9 deletions.
67 changes: 59 additions & 8 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,13 +751,24 @@ If you encounter issues with incremental loading, see the [troubleshooting secti

### Response actions

The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses from the API based on status codes or content substrings. This is useful for handling edge cases like ignoring responses on specific conditions.
The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses or all responses from the API. For example, responses with specific status codes or content substrings can be ignored.
Additionally, all responses or only responses with specific status codes or content substrings can be transformed with a custom callable, such as a function. This callable is passed on to the requests library as a [response hook](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks). The callable can modify the response object and has to return it for the modifications to take effect.

:::caution Experimental Feature
This is an experimental feature and may change in future releases.
:::

#### Example
**Fields:**

- `status_code` (int, optional): The HTTP status code to match.
- `content` (str, optional): A substring to search for in the response content.
- `action` (str or Callable or List[Callable], optional): The action to take when the condition is met. Currently supported actions:
- `"ignore"`: Ignore the response.
- a callable accepting and returning the response object.
- a list of callables, each accepting and returning the response object.


#### Example A

```py
{
Expand All @@ -772,12 +783,52 @@ This is an experimental feature and may change in future releases.

In this example, the source will ignore responses with a status code of 404, responses with the content "Not found", and responses with a status code of 200 _and_ content "some text".

**Fields:**
#### Example B

```py
def set_encoding(response, *args, **kwargs):
# sets the encoding in case it's not correctly detected
response.encoding = 'windows-1252'
return response

def add_and_remove_fields(response: Response, *args, **kwargs) -> Response:
payload = response.json()
for record in payload["data"]:
record["custom_field"] = "foobar"
record.pop("email", None)
modified_content: bytes = json.dumps(payload).encode("utf-8")
response._content = modified_content
return response

{
"path": "issues",
"response_actions": [
set_encoding
{"status_code": 200, "action": add_and_remove_fields},
],
}
```

In this example, the resource will set the correct encoding for all responses first. Thereafter, for all responses that have the status code 200, we will add a field `custom_field` and remove the field `email`.

#### Example C

```py
def set_encoding(response, *args, **kwargs):
# sets the encoding in case it's not correctly detected
response.encoding = 'windows-1252'
return response

{
"path": "issues",
"response_actions": [
set_encoding,
],
}
```

In this example, the resource will set the correct encoding for all responses. More callables can be added to the list of response_actions.

- `status_code` (int, optional): The HTTP status code to match.
- `content` (str, optional): A substring to search for in the response content.
- `action` (str): The action to take when the condition is met. Currently supported actions:
- `ignore`: Ignore the response.

## Troubleshooting

Expand Down Expand Up @@ -866,4 +917,4 @@ If experiencing 401 (Unauthorized) errors, this could indicate:

The `rest_api` source uses the [RESTClient](../../general-usage/http/rest-client.md) class for HTTP requests. Refer to the RESTClient [troubleshooting guide](../../general-usage/http/rest-client.md#troubleshooting) for debugging tips.

For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help!
For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help!
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,7 @@ client = RESTClient(

## Advanced usage

`RESTClient.paginate()` allows to specify a custom hook function that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully:
`RESTClient.paginate()` allows to specify a [custom hook function](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks) that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully:

```py
def custom_response_handler(response):
Expand Down

0 comments on commit 501df76

Please sign in to comment.