Add Rate Limit Logic and Cache Authenticator
This PR contains three features to deliver required functionality for a new API.
- It contains enhanced logic in the tap discovery to cache credentials to avoid having to re-authenticate for each stream. The API was erroring due to too many OAuth Requests in a quick succession.
- Optional backoff logic has been built in allowing tap-rest-api-msdk to respond to http retry-after messages. This is configurable to use either the header or message responses. There is also a new setting which adds some additional time because sometime the requested wait time is longer enough. This feature was built in to meet the API's backoff requirements.
- Optionally provides the ability to store the whole raw message. The feature is enabled by setting the store_raw_json_message to true. This is useful if you wish to offload, the flattening functionality to the likes of dbt. Where I have used this feature I tend to select only the primary key, the replication key, and the _sdc_raw_json field.
The use case for this was a dynamic schema with optional fields/columns which were not available in every record. In this situation the schema discovery did not pick up every field leading to missing data, it was elected that storing the raw json record was important - to ensure all data is preserved.
Points 2-3 are optional, and so the tap's behaviour does not change. For Point 1, the caching will speed up discovery and ingestion as credentials are cached.