Feature: Retry errors from CCI 3 times in order to sidestep retryable errors

[AlexanderMann]

Motivation

#70

^ seeing the same errors as above...when running this...from inside CCI. This is the simplest solution I could whip up. I'm not a Go native developer, but figured this would be a good enough first pass! Happy to try and make changes as suggested etc.

Thanks for making this awesome provider! Our ops team is really happy with it 😄

Suggested Musical Pairing

https://soundcloud.com/madonna/hung-up-album-version

[bendrucker]

Thanks, https://github.com/hashicorp/go-retryablehttp would be a good fit for this. That will handle potentially transient server response errors in addition to client errors like connection failures. I'm not sure this code all requests:

The main difference from net/http is that requests which take a request body (POST/PUT et. al) can have the body provided in a number of ways (some more or less efficient) that allow "rewinding" the request body if the initial request fails so that the full request can be attempted again.

It also offers logging hooks which would seem like a useful thing to implement:

https://www.terraform.io/plugin/sdkv2/debugging#log-based-debugging

[bendrucker]

Also, from #70:

When I try to apply again the resources fails because it already exists.

This would suggest that retrying would not be the answer. A client is left in a hard spot here. It needs to fetch the context by ID. Reads can easily be retried, but if that write fails what can the client do?

In theory, yes, you can list and match by name, but that is not the provider's job. Terraform intentionally makes import an explicit operation.

Retrying is reasonable in general but I wonder if adding it just results in a new error for the original case described in #70, where upon retry (manual or automatic) Circle will return a 4xx, hopefully 409, indicating a conflict.