Skip to content

Commit

Permalink
Add ability to set extra environmental variables through downward API
Browse files Browse the repository at this point in the history
  • Loading branch information
dezmodue committed Nov 25, 2024
1 parent 272d04e commit 8a0418e
Show file tree
Hide file tree
Showing 8 changed files with 255 additions and 0 deletions.
42 changes: 42 additions & 0 deletions Issue.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
This PR provides a way to set extra environmental variables through the downward API.
Why?
We manage mutiple clusters where the differences between helm values boils down to the `authPath` which is set to a path including the cluster unique ID.
Each deployment has the cluster ID information provided by ArgoCD via the `argocd.argoproj.io/tracking-id` annotation.
Leveraging the downward API we can programmatically set the `authPath` for all clusters without the need for a cluster specific value file.

Example:
Generate the injector deployment:
```
helm template --show-only templates/injector-deployment.yaml --set 'injector.annotations.argocd\.argoproj\.io/tracking-id=cluster-1234' --set "injector.extraEnvironmentVarsFieldPath.CLUSTER_ID=metadata.annotations['argocd.argoproj.io/tracking-id']" --set 'injector.authPath=/auth/Kubernetes/$(CLUSTER_ID)' .
```

This will produce the yaml:
```
...
template:
metadata:
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: release-name
component: webhook
annotations:
argocd.argoproj.io/tracking-id: cluster-1234
...
env:
- name: "CLUSTER_ID"
valueFrom:
fieldRef:
fieldPath: metadata.annotations['argocd.argoproj.io/tracking-id']
...
- name: AGENT_INJECT_VAULT_AUTH_PATH
value: /auth/Kubernetes/$(CLUSTER_ID)
```

The resulting env in the running pod shows:
```
CLUSTER_ID=cluster-1234
AGENT_INJECT_VAULT_AUTH_PATH=/auth/Kubernetes/cluster-1234
```

This configuration is valid for any cluster removing the need for cluster specific values
136 changes: 136 additions & 0 deletions helm-test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
---
# Source: vault/templates/injector-deployment.yaml
# Deployment for the injector
apiVersion: apps/v1
kind: Deployment
metadata:
name: release-name-vault-agent-injector
namespace: default
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: release-name
app.kubernetes.io/managed-by: Helm
component: webhook
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: release-name
component: webhook

template:
metadata:
labels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: release-name
component: webhook
annotations:
argocd.argoproj.io/tracking-id: cluster-1234
spec:

affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: vault-agent-injector
app.kubernetes.io/instance: "release-name"
component: webhook
topologyKey: kubernetes.io/hostname




# serviceAccountName: "release-name-vault-agent-injector"

securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
hostNetwork: false
containers:
- name: sidecar-injector

image: "busybox"
imagePullPolicy: "IfNotPresent"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
env:

- name: "CLUSTER_ID"
valueFrom:
fieldRef:
fieldPath: metadata.annotations['argocd.argoproj.io/tracking-id']
- name: AGENT_INJECT_LISTEN
value: :8080
- name: AGENT_INJECT_LOG_LEVEL
value: info
- name: AGENT_INJECT_VAULT_ADDR
value: http://release-name-vault.default.svc:8200
- name: AGENT_INJECT_VAULT_AUTH_PATH
value: /auth/Kubernetes/$(CLUSTER_ID)
- name: AGENT_INJECT_VAULT_IMAGE
value: "hashicorp/vault:1.18.1"
- name: AGENT_INJECT_TLS_AUTO
value: release-name-vault-agent-injector-cfg
- name: AGENT_INJECT_TLS_AUTO_HOSTS
value: release-name-vault-agent-injector-svc,release-name-vault-agent-injector-svc.default,release-name-vault-agent-injector-svc.default.svc
- name: AGENT_INJECT_LOG_FORMAT
value: standard
- name: AGENT_INJECT_REVOKE_ON_SHUTDOWN
value: "false"
- name: AGENT_INJECT_CPU_REQUEST
value: "250m"
- name: AGENT_INJECT_CPU_LIMIT
value: "500m"
- name: AGENT_INJECT_MEM_REQUEST
value: "64Mi"
- name: AGENT_INJECT_MEM_LIMIT
value: "128Mi"
- name: AGENT_INJECT_DEFAULT_TEMPLATE
value: "map"
- name: AGENT_INJECT_TEMPLATE_CONFIG_EXIT_ON_RETRY_FAILURE
value: "true"

- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- /bin/sleep
- "600"
# livenessProbe:
# httpGet:
# path: /health/ready
# port: 8080
# scheme: HTTPS
# failureThreshold: 2
# initialDelaySeconds: 5
# periodSeconds: 2
# successThreshold: 1
# timeoutSeconds: 5
# readinessProbe:
# httpGet:
# path: /health/ready
# port: 8080
# scheme: HTTPS
# failureThreshold: 2
# initialDelaySeconds: 5
# periodSeconds: 2
# successThreshold: 1
# timeoutSeconds: 5
# startupProbe:
# httpGet:
# path: /health/ready
# port: 8080
# scheme: HTTPS
# failureThreshold: 12
# initialDelaySeconds: 5
# periodSeconds: 5
# successThreshold: 1
# timeoutSeconds: 5
27 changes: 27 additions & 0 deletions pod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
labels:
vault: downwardAPI
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox
command: [ "/bin/sh", "-c", "env" ]
env:
- name: MY_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: INPUT_ENV_VAR_FROM_FIELD_PATH
valueFrom:
fieldRef:
fieldPath: metadata.labels['vault']
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: OUTPUT_ENV_VAR_FROM_FIELD_PATH
value: $(INPUT_ENV_VAR_FROM_FIELD_PATH)
restartPolicy: Never
14 changes: 14 additions & 0 deletions templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -1003,6 +1003,20 @@ Inject extra environment vars in the format key:value, if populated
{{- end -}}
{{- end -}}
{{/*
Inject extra environment vars via fields available through the downward API
*/}}
{{- define "vault.extraEnvironmentVarsFieldPath" -}}
{{- if .extraEnvironmentVarsFieldPath -}}
{{- range $key, $value := .extraEnvironmentVarsFieldPath }}
- name: {{ printf "%s" $key | replace "." "_" | upper | quote }}
valueFrom:
fieldRef:
fieldPath: {{ $value }}
{{- end }}
{{- end -}}
{{- end -}}
{{/*
Inject extra environment populated by secrets, if populated
*/}}
Expand Down
1 change: 1 addition & 0 deletions templates/injector-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ spec:
imagePullPolicy: "{{ .Values.injector.image.pullPolicy }}"
{{- template "injector.securityContext.container" . }}
env:
{{- include "vault.extraEnvironmentVarsFieldPath" .Values.injector | nindent 12 }}
- name: AGENT_INJECT_LISTEN
value: {{ printf ":%v" .Values.injector.port }}
- name: AGENT_INJECT_LOG_LEVEL
Expand Down
26 changes: 26 additions & 0 deletions test/unit/injector-deployment.bats
Original file line number Diff line number Diff line change
Expand Up @@ -696,6 +696,32 @@ EOF
[ "${value}" = "sanitized" ]
}

#--------------------------------------------------------------------
# extraEnvironmentVarsFieldPath

@test "injector/deployment: set extraEnvironmentVarsFieldPath" {
cd `chart_dir`
local object=$(helm template \
--show-only templates/injector-deployment.yaml \
--set "injector.extraEnvironmentVarsFieldPath.FOO=metadata.labels['test']" \
--set 'injector.extraEnvironmentVarsFieldPath.FOOBAR=spec.nodeName' \
--set 'injector.extraEnvironmentVarsFieldPath.lower\.case=sanitized' \
. | tee /dev/stderr |
yq -r '.spec.template.spec.containers[0].env' | tee /dev/stderr)

local value=$(echo $object |
yq -r 'map(select(.name=="FOO")) | .[] .valueFrom.fieldRef.fieldPath' | tee /dev/stderr)
[ "${value}" = "metadata.labels['test']" ]

local value=$(echo $object |
yq -r 'map(select(.name=="FOOBAR")) | .[] .valueFrom.fieldRef.fieldPath' | tee /dev/stderr)
[ "${value}" = "spec.nodeName" ]

local value=$(echo $object |
yq -r 'map(select(.name=="LOWER_CASE")) | .[] .valueFrom.fieldRef.fieldPath' | tee /dev/stderr)
[ "${value}" = "sanitized" ]
}

#--------------------------------------------------------------------
# extra annotations

Expand Down
3 changes: 3 additions & 0 deletions values.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -374,6 +374,9 @@
"extraEnvironmentVars": {
"type": "object"
},
"extraEnvironmentVarsFieldPath": {
"type": "object"
},
"extraLabels": {
"type": "object"
},
Expand Down
6 changes: 6 additions & 0 deletions values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,12 @@ injector:
extraEnvironmentVars: {}
# KUBERNETES_SERVICE_HOST: kubernetes.default.svc

# extraEnvironmentVarsFieldPath is a list of extra environment variables to set in the
# injector deployment using the downward API.
extraEnvironmentVarsFieldPath: {}
# CLUSTERID: metadata.labels['clusterID']
# CLUSTERENVIRONMENT: metadata.annotations['clusterEnvironment']

# Affinity Settings for injector pods
# This can either be a multi-line string or YAML matching the PodSpec's affinity field.
# Commenting out or setting as empty the affinity variable, will allow
Expand Down

0 comments on commit 8a0418e

Please sign in to comment.