AWS IAM Auth seems to ignore X-Vault-AWS-IAM-Server-ID header in VaultAuth CR

[joshbench]

Describe the bug

When vault-secrets-operator is configured to authenticate to Vault with AWS IAM, with X-Vault-AWS-IAM-Server-ID set as a header in the VaultAuth custom resource, a VaultStaticSecret that is configured to use the configured VaultAuth fails to authenticate with the following error:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

It seems like when the VaultStaticSecret attempts to refresh, or to authenticate before refreshing, it is not attempting to use this header.

To Reproduce
Steps to reproduce the behavior:

1. Deploy application with the following yaml file with the following VSO custom resources.

helm install vault-secrets-operator hashicorp/vault-secrets-operator -n vault-secrets-operator --create-namespace -f values.yaml --version 0.8.1

• values.yml:

---
controller:
  manager:
    extraEnv:
    # Required for Pod Identity (https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)
    - name: AWS_SDK_LOAD_CONFIG
      value: "'true'"

defaultVaultConnection:
  enabled: true
  address: "https://my.vault.address:8200"
  tlsServerName: "my.vault.address"
  headers:
    X-Vault-AWS-IAM-Server-ID: "my.vault.address"

defaultAuthMethod:
  enabled: true
  method: aws
  mount: us-east-1/aws
  aws:
    role: my-role
    region: us-east-1
  headers:
    X-Vault-AWS-IAM-Server-ID: "my.vault.address"

• example_secret.yml

---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: example-secret
  namespace: my-app-namespace
spec:
  type: kv-v2
  mount: secretmount
  path: my/secret/path
  destination:
    name: example-secret
    create: true
    overwrite: true
  refreshAfter: 1h

3. See error (vault-secrets-operator logs, application logs, etc.)

• kubectl describe vaultstaticsecret example-secret

Name:         example-secret
Namespace:    my-app-namespace
Labels:       <none>
Annotations:  <none>
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultStaticSecret
Metadata:
  Creation Timestamp:  2024-09-12T19:52:52Z
  Generation:          1
  Resource Version:    56303
  UID:                 c2470e10-ac7e-427e-a269-ffa2d031ba8f
Spec:
  Destination:
    Create:          true
    Name:           example-secret
    Overwrite:       true
  Hmac Secret Data:  true
  Mount:             secretmount
  Path:              my/secret/path
  Refresh After:     1h
  Type:              kv-v2
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Warning  VaultClientConfigError  4m6s (x203 over 19m)  VaultStaticSecret  Failed to get Vault auth login: Error making API request.

URL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login
Code: 400. Errors:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

• kubectl describe vaultauth default

Name:         default
Namespace:    vault-secrets-operator
Labels:       app.kubernetes.io/component=controller-manager
              app.kubernetes.io/instance=vault-secrets-operator
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=vault-secrets-operator
              app.kubernetes.io/version=0.8.1
              component=controller-manager
              control-plane=controller-manager
              helm.sh/chart=vault-secrets-operator-0.8.1
Annotations:  meta.helm.sh/release-name: vault-secrets-operator
              meta.helm.sh/release-namespace: vault-secrets-operator
API Version:  secrets.hashicorp.com/v1beta1
Kind:         VaultAuth
Metadata:
  Creation Timestamp:  2024-09-12T19:52:35Z
  Finalizers:
    vaultauth.secrets.hashicorp.com/finalizer
  Generation:        1
  Resource Version:  56218
  UID:               7c6e004a-c2fd-4480-9a70-06f56a2f1a5c
Spec:
  Aws:
    Region:  us-east-1
    Role:    dataplane-dev
  Headers:
    X - Vault - AWS - IAM - Server - ID:  my-vault-address
  Method:                                 aws
  Mount:                                  us-east-1/aws
Status:
  Spec Hash:  1145c37dee10f76d30c2d7356dadfc0decc1fe4dff6401d4f47966d6e7d66fee
  Valid:      true
Events:
  Type    Reason    Age   From       Message
  ----    ------    ----  ----       -------
  Normal  Accepted  22m   VaultAuth  Successfully handled VaultAuth resource request
  Normal  Accepted  21m   VaultAuth  Successfully handled VaultAuth resource request

• Relevant logs:

vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager 2024/09/12 20:17:37 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil>
vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager 2024/09/12 20:17:37 Ignoring, HTTP credential provider invalid endpoint host, "169.254.170.23", only loopback hosts are allowed. <nil>
vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager 2024-09-12T20:17:37.277Z [DEBUG] added environment variable credential provider
vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager 2024-09-12T20:17:37.277Z [DEBUG] added shared credential provider
vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager {"level":"info","ts":"2024-09-12T20:17:37Z","msg":"Starting workers","controller":"vaultpkisecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultPKISecret","worker count":100}
vault-secrets-operator-controller-manager-6c8457bdc-kd4dm manager {"level":"error","ts":"2024-09-12T20:17:37Z","logger":"cachingClientFactory","msg":"Failed to get NewClientWithLogin","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","VaultStaticSecret":{"name":"example-secret","namespace":"my-app-namespace"},"namespace":"my-app-namespace","name":"example-secret","reconcileID":"7337504d-73d4-4ff5-a468-083eae9e1c61","cacheKey":"aws-db8afa8d0fdab12eb1a7cf","error":"Error making API request.\n\nURL: PUT https://my.vault-address:8200/v1/us-east-1/aws/login\nCode: 400. Errors:\n\n* error validating X-Vault-AWS-IAM-Server-ID header: missing header \"X-Vault-AWS-IAM-Server-ID\""}

Expected behavior

1. VaultAuth is configured with the correct IAM server header
2. A VaultStaticSecret is created
3. The controller manager sees that a new secret is created
4. The controller manager authenticates to Vault to fetch the secret. The controller manager uses all of the VaultAuth config, including any specified headers, to log in to Vault.
5. The secret is pulled and stored in the specified location

Environment

• Kubernetes version: 1.30
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): AWS EKS
  • Other configuration options or runtime services (istio, etc.): AWS IAM EKS Pod Identity is enabled
• vault-secrets-operator version: 0.8.1

Additional context
Add any other context about the problem here.

[benashz]

Hi @joshbench,

Thanks reporting this issue. Would you mind setting the value on the VaultAuth's .spec.aws.headerValue, rather than in .spec.headers. See https://developer.hashicorp.com/vault/docs/platform/k8s/vso/api-reference#vaultauthconfigaws for more info.

Please let us know if that helps to resolve the issue.

Thanks,

Ben

[joshbench]

I @benashz,

Thank you for responding!

I reconfigured my values.yml like so:

---
controller:
  manager:
    extraEnv:
    # Required for Pod Identity (https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)
    - name: AWS_SDK_LOAD_CONFIG
      value: "'true'"

defaultVaultConnection:
  enabled: true
  address: "https://my.vault.address:8200"
  tlsServerName: "my.vault.address"

defaultAuthMethod:
  enabled: true
  method: aws
  mount: us-east-1/aws
  aws:
    role: my-role
    region: us-east-1
    headerValue: my.vault.address

And I'm still getting an error, but now it's "missing Authorization header":

{"level":"error","ts":"2024-09-13T18:27:09Z","logger":"cachingClientFactory","msg":"Failed to get NewClientWithLogin","controller":"vaultstaticsecret","controllerGroup":"secrets.hashicorp.com","controllerKind":"VaultStaticSecret","VaultStaticSecret":{"name":"example-secret","namespace":"my-app-namespace"},"namespace":"my-app-namespace","name":"example-secret","reconcileID":"8b72bed4-4bd3-4a37-82f7-ae91f4b8995a","cacheKey":"aws-6bd141d8ace9805f592ec3","error":"Error making API request.\n\nURL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login\nCode: 400. Errors:\n\n* error validating X-Vault-AWS-IAM-Server-ID header: missing Authorization header"}

• kubectl describe vaultstaticsecret example-secret

  Warning  VaultClientConfigError  3m51s (x13 over 4m45s)  VaultStaticSecret  Failed to get Vault auth login: Error making API request.

URL: PUT https://my.vault.address:8200/v1/auth/us-east-1/aws/login
Code: 400. Errors:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"
  Warning  VaultClientConfigError  2m57s (x12 over 3m47s)  VaultStaticSecret  Failed to get Vault auth login: Error making API request.

URL: PUT [https://my.vault.address:8200/v1/auth/us-east-1/aws/login](https://my.vault.address:8200/v1/auth/us-east-1/aws/login)
Code: 400. Errors:

* error validating X-Vault-AWS-IAM-Server-ID header: missing Authorization header

Seems like the Authorization header is possibly getting overwritten?

[joshbench]

Looking further into this, and also reading up on https://developer.hashicorp.com/vault/docs/auth/aws#iam-auth-method, I think I'm just not authenticating to AWS correctly, so the Authorization header is not being set. I think the version of the package used for AWS authentication uses an older version of the AWS SDK that doesn't support Pod Identity.

• vault-secrets-operator/go.mod

  Line 19 in 76be1ee

  github.com/hashicorp/go-secure-stdlib/awsutil v0.3.0

• https://github.com/hashicorp/go-secure-stdlib/blob/45cce4374f8e37351cea21b9974f29a01b086ab1/awsutil/go.mod#L6

Where newer versions of awsutil seem to use aws-sdk-go-v2, which does support pod identity:

• https://github.com/hashicorp/go-secure-stdlib/blob/awsutil/v2.0.0/awsutil/go.mod#L6

So maybe instead, this should be a feature request to support authentication via Pod Identity?