External provider curl

README.md

@@ -4,26 +4,22 @@
 
 This module polls a series of fairly well known but occasionally unreliable 'what is my ip' services. It then produces the most common response that is a valid ipv4 or ipv6 address (both outputs are separately provided).
 
-We support two providers - `curl2` and `http`. `curl2` is the default provider because it has better failure handling whereas `http` provider will fail a plan/apply if the endpoint doesn't respond.
-
-## Limitations
-
-Neither the `curl2` nor `http` providers are perfect. The `curl2` provider is slightly more reliable than the `http` provider. We should ideally check DNS prior to polling, or find a provider that won't poll unless dns resolves - or won't fail if it doesn't resolve.
-
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Requirements
 
 | Name | Version |
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | ~> 1.0 |
 | <a name="requirement_curl2"></a> [curl2](#requirement\_curl2) | ~> 1.6 |
+| <a name="requirement_external"></a> [external](#requirement\_external) | ~> 2.3.1 |
 | <a name="requirement_http"></a> [http](#requirement\_http) | ~> 3 |
 
 ## Providers
 
 | Name | Version |
 |------|---------|
 | <a name="provider_curl2"></a> [curl2](#provider\_curl2) | 1.6.1 |
+| <a name="provider_external"></a> [external](#provider\_external) | 2.3.1 |
 | <a name="provider_http"></a> [http](#provider\_http) | 3.4.0 |
 
 ## Modules
@@ -35,16 +31,17 @@ No modules.
 | Name | Type |
 |------|------|
 | [curl2_curl2.myip](https://registry.terraform.io/providers/mehulgohil/curl2/latest/docs/data-sources/curl2) | data source |
+| [external_external.external_curl](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/external) | data source |
 | [http_http.myip](https://registry.terraform.io/providers/hashicorp/http/latest/docs/data-sources/http) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
-| <a name="input_data_provider"></a> [data\_provider](#input\_data\_provider) | `curl2` or `http` providers are both supported - we recommend `curl2` | `string` | `"curl2"` | no |
+| <a name="input_data_provider"></a> [data\_provider](#input\_data\_provider) | `curl2` or `http` providers are also supported - we recommend `external_curl` because it handles failure better | `string` | `"external_curl"` | no |
 | <a name="input_extra_service_urls"></a> [extra\_service\_urls](#input\_extra\_service\_urls) | Put your own in here if you want extra ones, this gets merged with the `service_urls` list | `list(string)` | `[]` | no |
 | <a name="input_request_timeout"></a> [request\_timeout](#input\_request\_timeout) | Request timeout in milliseconds | `number` | `500` | no |
-| <a name="input_retry_attempts"></a> [retry\_attempts](#input\_retry\_attempts) | Request retries | `number` | `0` | no |
+| <a name="input_retry_attempts"></a> [retry\_attempts](#input\_retry\_attempts) | Request retries | `number` | `1` | no |
 | <a name="input_service_urls"></a> [service\_urls](#input\_service\_urls) | List of urls to use for getting our IP | `list(string)` | <pre>[<br>  "https://api.seeip.org",<br>  "https://ipinfo.io/ip",<br>  "https://ifconfig.co",<br>  "https://icanhazip.com",<br>  "https://api.ipify.org",<br>  "https://ifconfig.me",<br>  "https://ipecho.net/plain",<br>  "https://ifconfig.io",<br>  "https://ident.me",<br>  "https://ipv4.ident.me"<br>]</pre> | no |
 
 ## Outputs
@@ -57,6 +54,18 @@ No modules.
 | <a name="output_ipv6_all_matches"></a> [ipv6\_all\_matches](#output\_ipv6\_all\_matches) | List of all the ipv6 matches (informational/testing) |
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 
+## Providers and their Limitations : An explanation.
+
+The goal of this module is to provide a valid-enough answer even when the internet does what the internet does, which is to be flaky and broken somewhere, sometime.
+
+Neither the `curl2` nor `http` providers are perfectly suited to the internet since they will fail a run if the url doesn't respond and/or the url doesn't resolve. The `curl2` provider is slightly more reliable than the `http` provider. This failure mode makes sense if it's a critical part of your terraform..
+
+Both of `curl2` and `http` providers are provided as a matter of them being possibly better in the future if/when they have some `ignore_failure` options.
+
+Since we're aggregating results to achieve a 'most common' response it frankly shouldn't matter if one of the endpoints in the list fails to respond - we will have gathered enough data to make a good response. Let the build roll on!
+
+As such we implement an `external_curl` using the `external` provider and a shim script `external_curl.sh`, which will survive a truly non-resolving non-responsive endpoint by faking the response data, which will be filtered out later by our `ipv4_matches` and `ipv6_matches` filters.
+
 ## Authors
 
 Alex Trull ([email protected])

external_curl.sh

@@ -0,0 +1,8 @@
+# Exit if any of the intermediate steps fail
+set -e
+
+export BODY=`curl -s --retry $2 --connect-timeout $3 $1 || echo -n noresponse`
+# Safely produce a JSON object containing the result value.
+# jq will ensure that the value is properly quoted
+# and escaped to produce a valid JSON string.
+jq -n --arg BODY "$BODY" '{"body":$BODY}'

main.tf

@@ -1,6 +1,10 @@
 terraform {
   required_version = "~> 1.0"
   required_providers {
+    external = {
+      version = "~> 2.3.1"
+      source  = "hashicorp/external"
+    }
     curl2 = {
       version = "~> 1.6"
       source  = "mehulgohil/curl2"
@@ -19,6 +23,15 @@ provider "curl2" {
   }
 }
 
+# this is the most reliable option due to the fact we can fake response for a broken service url
+data "external" "external_curl" {
+  for_each = var.data_provider == "external_curl" ? toset(local.service_urls) : []
+  program  = ["/bin/sh", "${path.module}/external_curl.sh", each.key, var.retry_attempts, var.request_timeout / 1000]
+  query = {
+    id = ""
+  }
+}
+
 # curl2 is the default method
 data "curl2" "myip" {
   for_each    = var.data_provider == "curl2" ? toset(local.service_urls) : []
@@ -41,8 +54,13 @@ locals {
   # merge extra with primary list and make sure entries are unique
   service_urls = distinct(concat(var.service_urls, var.extra_service_urls))
 
+  # pick whichever responses we based on the chosen data_provider option
+  external_curl_responses = var.data_provider == "external_curl" ? values(data.external.external_curl)[*].result.body : []
+  curl2_responses         = var.data_provider == "curl2" ? values(data.curl2.myip)[*].response.body : []
+  http_responses          = var.data_provider == "http" ? values(data.http.myip)[*].response_body : []
+
   # build a list of responses
-  service_response_bodies = var.data_provider == "curl2" ? values(data.curl2.myip)[*].response.body : values(data.http.myip)[*].response_body
+  service_response_bodies = coalesce(local.external_curl_responses, local.curl2_responses, local.http_responses)
 
   # remunge it without whitespace as a list of strings
   split_output = split(",", replace(trimspace(join(",", local.service_response_bodies)), "/\\s/", ""))

variables.tf

@@ -34,7 +34,7 @@ variable "retry_attempts" {
 }
 
 variable "data_provider" {
-  default     = "curl2"
+  default     = "external_curl"
   type        = string
-  description = "`curl2` or `http` providers are both supported - we recommend `curl2`"
+  description = "`curl2` or `http` providers are also supported - we recommend `external_curl` because it handles failure better"
 }