diff --git a/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md b/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md index 8d43e471c8..c752748c69 100644 --- a/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md +++ b/docs/website/docs/dlt-ecosystem/verified-sources/rest_api.md @@ -751,13 +751,24 @@ If you encounter issues with incremental loading, see the [troubleshooting secti ### Response actions -The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses from the API based on status codes or content substrings. This is useful for handling edge cases like ignoring responses on specific conditions. +The `response_actions` field in the endpoint configuration allows you to specify how to handle specific responses or all responses from the API. For example, responses with specific status codes or content substrings can be ignored. +Additionally, all responses or only responses with specific status codes or content substrings can be transformed with a custom callable, such as a function. This callable is passed on to the requests library as a [response hook](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks). The callable can modify the response object and has to return it for the modifications to take effect. :::caution Experimental Feature This is an experimental feature and may change in future releases. ::: -#### Example +**Fields:** + +- `status_code` (int, optional): The HTTP status code to match. +- `content` (str, optional): A substring to search for in the response content. +- `action` (str or Callable or List[Callable], optional): The action to take when the condition is met. Currently supported actions: + - `"ignore"`: Ignore the response. + - a callable accepting and returning the response object. + - a list of callables, each accepting and returning the response object. + + +#### Example A ```py { @@ -772,12 +783,52 @@ This is an experimental feature and may change in future releases. In this example, the source will ignore responses with a status code of 404, responses with the content "Not found", and responses with a status code of 200 _and_ content "some text". -**Fields:** +#### Example B + +```py +def set_encoding(response, *args, **kwargs): + # sets the encoding in case it's not correctly detected + response.encoding = 'windows-1252' + return response + +def add_and_remove_fields(response: Response, *args, **kwargs) -> Response: + payload = response.json() + for record in payload["data"]: + record["custom_field"] = "foobar" + record.pop("email", None) + modified_content: bytes = json.dumps(payload).encode("utf-8") + response._content = modified_content + return response + +{ + "path": "issues", + "response_actions": [ + set_encoding + {"status_code": 200, "action": add_and_remove_fields}, + ], +} +``` + +In this example, the resource will set the correct encoding for all responses first. Thereafter, for all responses that have the status code 200, we will add a field `custom_field` and remove the field `email`. + +#### Example C + +```py +def set_encoding(response, *args, **kwargs): + # sets the encoding in case it's not correctly detected + response.encoding = 'windows-1252' + return response + +{ + "path": "issues", + "response_actions": [ + set_encoding, + ], +} +``` + +In this example, the resource will set the correct encoding for all responses. More callables can be added to the list of response_actions. -- `status_code` (int, optional): The HTTP status code to match. -- `content` (str, optional): A substring to search for in the response content. -- `action` (str): The action to take when the condition is met. Currently supported actions: - - `ignore`: Ignore the response. ## Troubleshooting @@ -866,4 +917,4 @@ If experiencing 401 (Unauthorized) errors, this could indicate: The `rest_api` source uses the [RESTClient](../../general-usage/http/rest-client.md) class for HTTP requests. Refer to the RESTClient [troubleshooting guide](../../general-usage/http/rest-client.md#troubleshooting) for debugging tips. -For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help! \ No newline at end of file +For further assistance, join our [Slack community](https://dlthub.com/community). We're here to help! diff --git a/docs/website/docs/general-usage/http/rest-client.md b/docs/website/docs/general-usage/http/rest-client.md index d3a06a1d28..7d75a79c13 100644 --- a/docs/website/docs/general-usage/http/rest-client.md +++ b/docs/website/docs/general-usage/http/rest-client.md @@ -566,7 +566,7 @@ client = RESTClient( ## Advanced usage -`RESTClient.paginate()` allows to specify a custom hook function that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully: +`RESTClient.paginate()` allows to specify a [custom hook function](https://requests.readthedocs.io/en/latest/user/advanced/#event-hooks) that can be used to modify the response objects. For example, to handle specific HTTP status codes gracefully: ```py def custom_response_handler(response):