-
Notifications
You must be signed in to change notification settings - Fork 888
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mutation webhook failing to inject vault sidecars #163
Comments
Hi @gopisaba, I just deployed this on EKS in different namespaces, but could not reproduce what you're seeing. Can you provide me with:
|
global:
enabled: true
tlsDisable: false
injector:
certs:
secretName: vault-tls
server:
auditStorage:
accessMode: ReadWriteOnce
enabled: true
size: 10Gi
storageClass: null
authDelegator:
enabled: true
dataStorage:
enabled: false
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-tls/tls.ca
extraVolumes:
- name: vault-tls
type: secret
ha:
config: |
ui = true
listener "tcp" {
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
tls_key_file = "/vault/userconfig/vault-tls/tls.key"
tls_client_ca_file = "/vault/userconfig/vault-tls/tls.ca"
}
storage "dynamodb" {
ha_enabled = "true"
region = "eu-west-1"
table = "vault-backend"
}
seal "awskms" {
region = "eu-west-1"
kms_key_id = "1ee6b01a-1d8a-4cfb-abcd-12bdc43ab8d2"
endpoint = "https://vpce-01234567890-6abcdef.kms.eu-west-1.vpce.amazonaws.com"
}
enabled: true
replicas: 3
ingress:
enabled: false
nodeSelector: |
nodeType: grp1
standalone:
enabled: false
ui:
enabled: true
serviceNodePort: 32582
serviceType: NodePort Kube Version = 1.14 (EKS)
|
@gopisaba could be the same problem I had hashicorp/vault-k8s#46 |
@krep-dr - That's it. After opening the port |
@gopisaba what is the EKS cluster IP range? Or how can I find out the range? I do have the same issue. Thanks! |
@DongshengXiong - Allowing EKS cluster security group to EKS worker nodes security group over the port 8080 fixed the issue for me. |
@gopisaba thanks for your reply. Actually, I am using Weave Net CNI. My issue is fixed by this solution(hashicorp/vault-k8s#72) |
Hi @DongshengXiong what specifically did you change on the EKS security group? Did you use eksctl to set up your cluster? If so, which security group did you change and which security group was the source for the inbound rule? |
I know its been a while since this was asked and you probably know the answer by now, but for anyone else, there are two security groups you will need to change, one for inbound and one for outbound.
Read comments on inbound and outbound security rules to figure out which group is used for what. |
I ran into this issue the other day when using terraform to deploy the When defining the EKS module, you need to add the following node_security_group_additional_rules:
|
This solution works well in the EKS cluster. Thanks to @kschoche! Problem
E0610 20:50:30.214031 10 dispatcher.go:214] failed calling webhook "vault.hashicorp.com": failed to call webhook: Post "[https://vault-agent-injector-svc.vault.svc:443/mutate?timeout=30s](https://vault-agent-injector-svc.vault.svc/mutate?timeout=30s)": context deadline exceeded Environment
Solution
In official # vault-helm/values.yaml
injector:
# True if you want to enable vault agent injection.
# @default: global.enabled
enabled: "-"
replicas: 1
# Configures the port the injector should listen on
port: 8080 So add an inbound rule to the worker node security group (SG) to allow TCP 8080 with the Control Plane as the source. ---
title: Kubernetes architecture (EKS v1.30)
---
flowchart LR
subgraph Control plane
C["kube-apiserver"]
end
S["vault-agent-injector-svc"]
subgraph Worker node
P["vault-agent-injector"]
end
C --"tcp/443"--> S:::blue -. tcp/8080 .-> P
classDef blue stroke:#00f
Example in terraform using eks moduleAdd an inbound rule for tcp port 8080 to module "eks" {
# ... truncated ...
node_security_group_additional_rules = {
ingress_vault_agent_injector_mutating_webhook = {
description = "Allow ingress mutating webhook traffic from kube-apiserver to vault-agent-injector pod"
protocol = "tcp"
from_port = 8080
to_port = 8080
type = "ingress"
source_cluster_security_group = true
}
# Similar case for linkerd-viz tap pod's api service
ingress_linkerd_viz_tap_api = {
description = "Allow ingress api calling traffic from kube-apiserver to linkerd-viz tap pod"
protocol = "tcp"
from_port = 8088
to_port = 8089
type = "ingress"
source_cluster_security_group = true
}
}
# ... truncated ...
} ReferenceSimilar case Linkerd-Viz Tap FailedDiscoveryCheck while Running on EKS |
I am using the latest Vault Helm chart. The mutation webhook is failing to inject the vault-agent and consul-template sidecars.
Error messages on EKS api-server logs
I don't see any other error messages on vault or vault-agent-injector pod. I am able to resolve and connect to the vault-agent-injector-svc from test pod in different namespace.
vault svc
The text was updated successfully, but these errors were encountered: