keyvault: support for creating items over a nested endpoint

[cb900rr2000]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

0.14.0 / 2.39.0

Affected Resource(s)

• azurerm_key_vault_key

Terraform Configuration Files

provider "azurerm" {
  version          = "=2.39.0"
  subscription_id  = "5ab29627-1b06-47b1-bf94-251a9e6c74c7"
  tenant_id        = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
  skip_provider_registration = false
  features {}

  use_msi = true
}

provider "azurerm" {
 version           = "=2.39.0"
  alias            = "dnsprod"
  subscription_id  = "ed330c6f-5b25-4d9a-8db2-13a6b612c317"
  tenant_id        = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
  skip_provider_registration = true
  features {}

  use_msi = true
}

data "azurerm_client_config" "current" {}

data "azurerm_virtual_machine" "vm" {

   name                 =  "mangementvm"
   resource_group_name  = "development"
}

data "azurerm_subnet" "rg" {
  name                 = "Storage"
  virtual_network_name = "Policytest"
  resource_group_name  = "development"

}

resource "azurerm_resource_group" "rg" {
  name     = "TFtest2"
  location = "uksouth"
}

resource "azurerm_key_vault" "main" {
  name                = "newkekvault215"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name = "premium"
  purge_protection_enabled = true
  soft_delete_enabled      = true

     network_acls {
     default_action = "Deny"
     bypass         = "AzureServices"
}
access_policy {
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id   = data.azurerm_virtual_machine.vm.identity.0.principal_id

  key_permissions    = ["get", "create", "delete", "list", "restore", "recover", "unwrapkey", "wrapkey", "purge", "encrypt", "decrypt", "sign", "verify",]
  }
}

  data "azurerm_private_dns_zone" "key_vault_dns_private_zone" {
  name     = "privatelink.vaultcore.azure.net"
  provider = azurerm.dnsprod
  resource_group_name = "NewTestNetwork"
}

resource "azurerm_private_endpoint" "keyvault" {
   name                = "key_vault-terraform-endpoint"
   location            = azurerm_resource_group.rg.location
   resource_group_name = azurerm_resource_group.rg.name
   subnet_id           = "${data.azurerm_subnet.rg.id}"

  private_service_connection {
    name                           = "key_vault-terraform-privateserviceconnection"
    private_connection_resource_id = azurerm_key_vault.main.id
    subresource_names              = [ "vault" ]
    is_manual_connection           = false
  }

 private_dns_zone_group {
    name = data.azurerm_private_dns_zone.key_vault_dns_private_zone.name
   private_dns_zone_ids = [data.azurerm_private_dns_zone.key_vault_dns_private_zone.id]
  }
}

resource "azurerm_key_vault_key" "key" {
  name         = "Storage-KEK200"
  key_vault_id = azurerm_key_vault.main.id
  key_type     = "RSA-HSM"
  key_size     = 2048
  expiration_date = "2050-01-01T00:00:00Z"

 key_opts = [
   "decrypt",
   "encrypt",
   "sign",
   "unwrapKey",
   "verify",
   "wrapKey",
  ]
}

Debug Output

Panic Output

Expected Behaviour

The Key gets created.

Actual Behaviour

Error: Error checking for presence of existing Key "Storage-KEK200" (Key Vault "https://newkekvault215.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 10.0.6.5 from unknown subnet\r\nCaller: appid=feb4f312-1ff0-4b05-94b0-933b72f199d5;oid=8257275c-a014-4140-ba4f-b3b656dc9f6d;iss=https://sts.windows.net/90ad1900-2019-4ffa-b845-96f012d0dc5a/;xms_mirid=/subscriptions//resourcegroups/development/providers/Microsoft.Compute/virtualMachines/mangementvm\r\nVault: newkekvault215;location=uksouth" InnerError={"code":"ForbiddenByFirewall"}

• At this point the Private End Point, KV is created correctly and from the VM with the Managed Service Identity I can create a key with az cli for example over the Private End Point because the KV has no firewalls.

• If without any change I do terraform apply again the key is deployed.

I've tried to put the Key Vault access policy via a separate resource and note this as a dependency for the key Vault Key resource and put in a timer delay but this didn't help.

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[tombuildsstuff]

👋

At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.

Thanks!

[do87]

Things that helped me workaround this issue until a fix is released:
assigning certificate_permissions = [ "ManageContacts" ] to the Service Principal that controls terraform
i also encountered context timeout, this was because i had private DNS for keyvault linked to the vnet and it prevented getting the public IP for the vault i was trying to modify. Removing the private DNS fixed the issue.

[sebader]

In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

@tombuildsstuff do you have any pointers to this? I couldnt find any API docs around that. Thanks for any pointers!

[zhangweikop]

Generally, I think the KeyVault implementation should have one option to turn off the data plane access (and disable those related minor feature if needed)
Include:
Do not ping the keyvault URL for checking existence.
Do not make any API such as get contacts.

Making data plane access always cause additional trouble when we have private network endpoint.

[roy-work]

At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.

I am also getting this error (suddenly, too; to my knowledge, nothing has changed, and I can't find anything in the activity logs in Azure to indicate it).

But the KV for which I'm getting this on is set to "Allow access from: All networks" on the Networking / Firewalls and virtual networks page, and has no connections under "Private endpoint connections"…

[oliviergaumond]

Multiple issues have been closed as duplicates of this one. But I am not sur the following error is the exact same context as described here.
Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

We get this error when trying to refresh the state even if there are no data plane operations in the Terraform file, as described in may of the linked issues.

Unless you tell me that the access policy is using the data plane?

[AndreasMWalter]

We have a similar issue right now in our current environment the keyvault is created before peerings are created and state refresh before peering creation will fail due to the required dataplane access to set these values. We can workaround this issue by deploying the keyvault after peerings and DNS are done but that might cause other issues.

[dstratman-fi]

Agree with @oliviergaumond

@tombuildsstuff

Can you clarify if you are including all the problems mentioned in - #10501
within this issue now as well even though this issue is labeled very differently above?

Thanks!

[jpmicrosoft]

In Azure DevOps I use an Azure DevOps Private Agent and ensure that it has access to the subnet where the key vault private endpoint resides. This is my work around. It works without issues.

What happens is that once Private Endpoints are enabled in the Key Vault, it shuts off public access. Terraform reaches out to the Key Vault to confirm that it exists and/or to make a change, etc... If the access from Terraform to the Key Vault is public, it will not be able to confirm access it thus it fails. I have seen a couple of different errors that are a symptom of the same issue.

I hope this helps someone.

[pwolthausen]

If I try a wget from the runner where terraform is being run:

azureuser@GitLabRunner-UbuntuServer22-VM-01:~$ wget  https://myvault.vault.azure.net
--2022-12-21 00:20:59--  https://myvault.vault.azure.net/
Resolving myvault.vault.azure.net (myvault.vault.azure.net)... 20.48.197.105
Connecting to myvault.vault.azure.net (myvault.vault.azure.net)|20.48.197.105|:443... connected.
OpenSSL: error:0A000126:SSL routines::unexpected eof while reading
Unable to establish SSL connection.

[magodo]

Providing a workaround using azapi provider (so that we can avoid calling the contact list data plane API), e.g.:

resource "azapi_resource" "vault" {
  type      = "Microsoft.KeyVault/vaults@2021-10-01"
  parent_id = azurerm_resource_group.test.id
  name      = "vault230119153000994165"
  location  = azapi_resource.resourceGroup.location
  body = jsonencode({
    properties = {
      accessPolicies = [
        {
          objectId = "xxxx"
          permissions = {
            certificates = [
              "ManageContacts",
            ]
            keys = [
              "Create",
            ]
            secrets = [
              "Set",
            ]
            storage = [
            ]
          }
          tenantId = "xxxx"
        },
      ]
      enableRbacAuthorization      = false
      enableSoftDelete             = true
      enabledForDeployment         = false
      enabledForDiskEncryption     = false
      enabledForTemplateDeployment = false
      publicNetworkAccess          = "Enabled"
      sku = {
        family = "A"
        name   = "standard"
      }
      softDeleteRetentionInDays = 7
      tenantId                  = "xxxx"
    }
  })
}

[johhess40]

Ran into this same issue @tombuildsstuff and was able to add the contact field to the ignore_changes field inside of the lifecycle block for the key vault resource:

  lifecycle {
    ignore_changes = [
      tags["CreationDate"],
      tags["creationDate"],
      tags["Project"],
      contact
    ]
  }

I think that the code mentioned above by @Lddeiva is the root of the issue as the vault's uri is used to query this property, and an addition to that function which can check the status of networking/private endpoints for the vault may be useful. Either way this was my workaround so that my pipelines could proceed when checking against state. I also don't usually use the ignore_changes functionality, but in this instance it is inconsequential since the field can only be set once and cannot be change per the documentation here.

[reshmav18]

Hi @tombuildsstuff , I'm reaching out w.r.t to #21876

My customer whitelisted IP address as per the suggestion. however, issue still persist. Looking for an alternative.

Thanks

[joakimlemb]

Since this has been a issue since at least Dec 7, 2020 with multiple duplicate reports can we at least get a update to the docs at "https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/key_vault" with some information that explains that "terraform refresh" operations needs network connectivity to the keyvault dataplane?

Maybe also a minor note that the "-refresh=false" parameter circumvents the issue...

The error message "Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" is not descriptive enough on it's own to easily identify the issue...

[vxchil]

Hello, we are seeing this as well when refreshing keyvault with private end point. Any help on this is appreciated. We are using latest azurerm provider (3.70.0)

[glloyd2010f]

Seeing this on the latest provider still, 3.75.0.

[XristophD]

running into the same issue with the current provider version

Workarounds mentioned did not work:

• include 'contact' within lifecycle ignore_changes as mentioned by @johhess40
• add Public IP of DevOps Agent to allowed IP addresses within KeyVault network settings

Our design currently does not support the Azure DevOps Agent to connect to the KeyVault via Private Network for different reasons. Any help on working around this issue is highly appreciated.

[phoehnel]

Had the same error using hashicorp/azurerm v3.89.0. Though in my case, the reason seems to be that the private endpoint connection was not working correctly (although there was no visible sign of this in terraform nor the portal).
Trying to reach the Key Vault URL e.g. using curl or a browser timed out as well.

Resolved it by manually deleting the Private Endpoint Connection in the Networking tab of the Key Vault and then re-applying terraform.

[nomadramanujan]

I had the same issue with keyvault and storage resources. I switched to microsoft/azapi and that provider can managed these resources while the privates endpoints exist, so it is a workaround I guess. But we need a fix eventually as azurerm is a much better provider to manage (lack of jsonencode stuff)

[joakimlemb]

I had the same issue with keyvault and storage resources. I switched to microsoft/azapi and that provider can managed these resources while the privates endpoints exist, so it is a workaround I guess. But we need a fix eventually as azurerm is a much better provider to manage (lack of jsonencode stuff)

@nomadramanujan
Could you share your azapi workaround?

Looking a little closer at the issue, the function "GetCertificateContacts" is what requires data plane access to the keyvault and creates this problem when using private endpoints without having tcp port 443 access to the keyvault from the host machine that is running terraform.
(Due to how microsoft separates mgmt plane and data plane operations: https://learn.microsoft.com/en-us/rest/api/keyvault/certificates/get-certificate-contacts/get-certificate-contacts )

But, there already exists a dedicated terraform resource for this: azurerm_key_vault_certificate_contacts

The GetCertificateContacts function could then be removed from "azurerm_key_vault", and those that need that function could use "azurerm_key_vault_certificate_contacts" instead.

I'm guessing most who use "azurerm_key_vault" does not use the information provided by GetCertificateContacts anyway, so the percentage of impacted users should be at a minimum for this proposed breaking change.

[nomadramanujan]

I basically removed the state of the azurerm keyvault resource and imported them to terraform as azapi resources to my codebase after declaring keyvault azapi resource there with minimal config and adjusted the configuration using this documentation after import until plan reflected the current infra. But to remove the state and import I disconnected the private endpoint/link temporarily. I had the privilege of having some downtime.

[roy-work]

I think #23823 addresses the "Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" error. It is merged in v3.93.0, one would need to upgrade to at least that.

Upgrading to that version for me appears to alleviate the problem, though the upgrade from v2.99 → v3.93 introduces a frankly comical amount of other breakage, so be prepared to fight a bunch of small fires when you make this upgrade. They're manageable, though.

[asnell-frtservices]

The "Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded" issue does not appear to be fixed with latest available provider version 3.115.0

[heitorPB]

This error also hits me with azurerm 4.11.0

[joakimlemb]

This error also hits me with azurerm 4.11.0

Did you set "public_network_access_enabled = false"?

"public_network_access_enabled = true" will still contact the dataplane: #23823

[audunsolemdal]

This error also hits me with azurerm 4.11.0

Did you set "public_network_access_enabled = false"?

"public_network_access_enabled = true" will still contact the dataplane: #23823

That is interesting and a bit unfortunate behaviour, no?. In some cases I have key vaults with public network access enabled but with default action "deny", bypass "none" and specific IP whitelists.

[joakimlemb]

This error also hits me with azurerm 4.11.0

Did you set "public_network_access_enabled = false"?
"public_network_access_enabled = true" will still contact the dataplane: #23823

That is interesting and a bit unfortunate behaviour, no?. In some cases I have key vaults with public network access enabled but with default action "deny", bypass "none" and specific IP whitelists.

Indeed, #23823 does not solve separation between data and controlplane in all scenarios...

Perhaps the solution used in storage_account could be reused for keyvaults as well: #27818

Tldr; storage_accounts can now be configured to only use control plane operations via provider config:

provider "azurerm" {
  features {
    storage {
        data_plane_available = false 
    }
  }
}

Could be useful with this as a keyvault feature in the future?

    keyvault {
        data_plane_available = false 
    }

[heitorPB]

This error also hits me with azurerm 4.11.0

Did you set "public_network_access_enabled = false"?

"public_network_access_enabled = true" will still contact the dataplane: #23823

I did.

Here's what I have:

resource "azurerm_key_vault" "default" {
  name                      = "kv-${var.name}-${var.environment}"
  location                  = azurerm_resource_group.ml_rg.location
  resource_group_name       = azurerm_resource_group.ml_rg.name
  tenant_id                 = var.tenant_id
  sku_name                  = "premium"
  purge_protection_enabled  = true
  enable_rbac_authorization = true

  public_network_access_enabled = false

  # Allow Azure Services to access resource without explicit Private Endpoint
  network_acls {
    default_action             = "Allow"
    bypass                     = "AzureServices"
    virtual_network_subnet_ids = [azurerm_subnet.workspace.id]
  }
}

And the error:

│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
│
│   with azurerm_key_vault.default,
│   on foo.tf line 1, in resource "azurerm_key_vault" "default":
│    1: resource "azurerm_key_vault" "default" {

[dan-dimitrov]

We experienced a similar issue in a secured vWAN scenario (with Azure Firewall). Ultimately this issue is caused by lack of data plane connectivity between the build agents and the key vault. In our scenario, both are private. We could hit private key vaults in the same region (vWAN Hub), but we could not hit the key vaults in the other regions (other connected Hubs).

We discovered that this appears to be in the way that the CNAME resolution for *.privatelink.vaultcore.azure.net to *.vault.azure.net is handled in a multi-hub scenario with vWAN (with routing intent enabled). Part of this operation is to update the source IP address as the traffic is passed through from source hub firewall to the destination hub firewall. The source IP is updated to reflect a random IP address in the source hub address space (potentially an outbound Firewall IP as this appears to be in the second available /24 of this address space in our case).

With the above, we could see no traffic being blocked on the source firewall (as we already had rules in place for build agents), but we could see blocked traffic on the destination hub firewall for random addresses in the hub address space. We allowed the traffic for this range to the key vault endpoints, and our private connectivity/deployment issues were resolved.

We used "curl -i https://fabrikam.vault.azure.net/healthstatus" command from our ubuntu agents to test connectivity to the key vaults and validate this. The diagnostics page came in handy for various test commands - https://learn.microsoft.com/en-us/azure/key-vault/general/private-link-diagnostics#7-validate-that-requests-to-key-vault-use-private-link .