Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs/505 updates documentation on custom hooks in response_actions #1524

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 59 additions & 8 deletions docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -751,13 +751,24 @@ If you encounter issues with incremental loading, see the [troubleshooting secti

### Response actions

The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses from the API based on status codes or content substrings. This is useful for handling edge cases like ignoring responses on specific conditions.
The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses or all responses from the API. For example, responses with specific status codes or content substrings can be ignored.
Additionally, all responses or only responses with specific status codes or content substrings can be transformed with a custom callable, such as a function. This callable is passed on to the requests library as a [response hook](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks). The callable can modify the response object and has to return it for the modifications to take effect.

:::caution Experimental Feature
This is an experimental feature and may change in future releases.
:::

#### Example
**Fields:**

- `status_code` (int, optional): The HTTP status code to match.
- `content` (str, optional): A substring to search for in the response content.
- `action` (str or Callable or List[Callable], optional): The action to take when the condition is met. Currently supported actions:
- `"ignore"`: Ignore the response.
- a callable accepting and returning the response object.
- a list of callables, each accepting and returning the response object.


#### Example A

```py
{
Expand All @@ -772,12 +783,52 @@ This is an experimental feature and may change in future releases.

In this example, the source will ignore responses with a status code of 404, responses with the content "Not found", and responses with a status code of 200 _and_ content "some text".

**Fields:**
#### Example B

```py
def set_encoding(response, *args, **kwargs):
# sets the encoding in case it's not correctly detected
response.encoding = 'windows-1252'
return response

def add_and_remove_fields(response: Response, *args, **kwargs) -> Response:
payload = response.json()
for record in payload["data"]:
record["custom_field"] = "foobar"
record.pop("email", None)
modified_content: bytes = json.dumps(payload).encode("utf-8")
response._content = modified_content
return response

{
"path": "issues",
"response_actions": [
set_encoding
{"status_code": 200, "action": add_and_remove_fields},
],
}
```

In this example, the resource will set the correct encoding for all responses first. Thereafter, for all responses that have the status code 200, we will add a field `custom_field` and remove the field `email`.

#### Example C

```py
def set_encoding(response, *args, **kwargs):
# sets the encoding in case it's not correctly detected
response.encoding = 'windows-1252'
return response

{
"path": "issues",
"response_actions": [
set_encoding,
],
}
```

In this example, the resource will set the correct encoding for all responses. More callables can be added to the list of response_actions.

- `status_code` (int, optional): The HTTP status code to match.
- `content` (str, optional): A substring to search for in the response content.
- `action` (str): The action to take when the condition is met. Currently supported actions:
- `ignore`: Ignore the response.

## Troubleshooting

Expand Down Expand Up @@ -866,4 +917,4 @@ If experiencing 401 (Unauthorized) errors, this could indicate:

The `rest_api` source uses the [RESTClient](../../general-usage/http/rest-client.md) class for HTTP requests. Refer to the RESTClient [troubleshooting guide](../../general-usage/http/rest-client.md#troubleshooting) for debugging tips.

For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help!
For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help!
2 changes: 1 addition & 1 deletion docs/website/docs/general-usage/http/rest-client.md
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,7 @@ client = RESTClient(

## Advanced usage

`RESTClient.paginate()` allows to specify a custom hook function that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully:
`RESTClient.paginate()` allows to specify a [custom hook function](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks) that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully:

```py
def custom_response_handler(response):
Expand Down
Loading