Feat/fetch retry and error handling

[TheMagoo73]

A slightly naive implementation of retry logic to mitigate transient errors communicating with the intake. Can be improved down the line by filtering HTTP response codes such as 401 that should not cause a retry as they're not transient.

[TheMagoo73]

Found some time today to improve the retry logic slightly. It now only retries HTTP 500 and greater errors. Anything in the 400 range is considered a 'persistent' error and isn't retried - the POST bails immediately. There are a couple of test cases to cover each scenario.

[itsfadnis]

@TheMagoo73 My apologies for the delay in getting back at this PR

While I think retrying requests is a good idea, a better idea would be to expose a callback that the consumer of datadog-winston could call if a request fails.

This way the consumer can decide what they want to do with a log when it fails, rather than us trying to resend it to datadog.

We could probably expose something like an onFailure hook, so it could be used like:

const transport = new DatadogTransport({
  // options
  onFailure: (data) => {
	// Log transport failed,
    // Do whatever you want with the failed log
  }
})

What are your thoughts on this?

[TheMagoo73]

Apologies for the delay in getting back - holiday season!

I like the idea of a callback, it definitely adds more flexibility, I'm playing with some ideas around it at the moment. The key for me, I think, is to allow the caller to have enough information to handle retries, back offs etc. easily. My thinking is that an enriched version of the original log request that includes the number of failures would probably no the job.

I think supplying an 'out-of-the-box' retry implementation that can be used for the hook handler is also something we should consider, so users have a really simple, zero(ish) config to just get something reliable working.

Will try and get something up this weekend!