An error occurred in the init container(Operation not permitted)

[sdYoo]

Describe the bug
After executing the helm chart, the following error occurs when running the init container.

➜ ~ kc logs -f vault-0 -c busybox -n security
chown: /vault/logs: Operation not permitted
chown: /vault/logs: Operation not permitted

To Reproduce
Steps to reproduce the behavior:

1. Install chart
2. Run vault command
3. See error (vault logs, etc.)

vault-0 0/1 Init:CrashLoopBackOff 5 (60s ago) 4m13s 10.252.6.12 node01
vault-1 0/1 Init:CrashLoopBackOff 5 (65s ago) 4m11s 10.252.26.87 node02

Other useful info to include: vault pod logs, kubectl describe statefulset vault and kubectl get statefulset vault -o yaml output

Name: vault
Namespace: security
CreationTimestamp: Mon, 30 Oct 2023 16:41:15 +0900
Selector: app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault,component=server
Labels: app.kubernetes.io/instance=production-retail-mgmt-security-vault
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=vault
Annotations:
Replicas: 2 desired | 2 total
Update Strategy: OnDelete
Pods Status: 0 Running / 2 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
helm.sh/chart=vault-0.24.1
Annotations: kubectl.kubernetes.io/restartedAt: 2023-11-06T08:02:25Z
Service Account: vault
Init Containers:
busybox:
Image: docker-hub.com/finalspy/busybox-curl-jq
Port:
Host Port:
Command:
sh
-c
chown -R 1000:1000 /vault/logs
Environment:
Mounts:
/vault/logs from logs (rw)
Containers:
vault:
Image: docker-hub.com/hashicorp/vault:1.13.1-jqcurl
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl

Limits:
  cpu:     250m
  memory:  256Mi
Requests:
  cpu:     250m
  memory:  256Mi
Environment:
  HOST_IP:               (v1:status.hostIP)
  POD_IP:                (v1:status.podIP)
  VAULT_K8S_POD_NAME:    (v1:metadata.name)
  VAULT_K8S_NAMESPACE:   (v1:metadata.namespace)
  VAULT_ADDR:           http://127.0.0.1:8200
  VAULT_API_ADDR:       http://$(POD_IP):8200
  SKIP_CHOWN:           true
  SKIP_SETCAP:          true
  HOSTNAME:              (v1:metadata.name)
  VAULT_CLUSTER_ADDR:   https://$(HOSTNAME).vault-internal:8201
  HOME:                 /home/vault
  VAULT_LOG_LEVEL:      info
  VAULT_LOG_FORMAT:     json
Mounts:
  /etc/localtime from localtime (ro)
  /etc/timezone from timezone (ro)
  /home/vault from home (rw)
  /vault/config from config (rw)
  /vault/file from script (rw)
  /vault/logs from logs (rw)

Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
logs:
Type: HostPath (bare host directory volume)
Path: /home/logs/security/vault
HostPathType:
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
timezone:
Type: HostPath (bare host directory volume)
Path: /etc/timezone
HostPathType:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-auto-unseal
Optional: false
Volume Claims:
Events:
Type Reason Age From Message

---

Normal SuccessfulCreate 8m6s (x24 over 7d3h) statefulset-controller create Pod vault-0 in StatefulSet vault successful
Normal SuccessfulCreate 8m4s (x24 over 7d3h) statefulset-controller create Pod vault-1 in StatefulSet vault successful

Expected behavior
A clear and concise description of what you expected to happen.

Environment

• Kubernetes version:
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.):
  • Other configuration options or runtime services (istio, etc.):
• vault-helm version:
  apiVersion: v2
  name: vault
  version: 0.24.1
  appVersion: 1.13.1

Chart values:

values.yaml

server:
  # Configure the logging format for the Vault server.
  # Supported log formats include: standard, json
  logFormat: "json"

  extraInitContainers:
  - name: busybox
    image: "docker-hub-custom.com/finalspy/busybox-curl-jq"
    command: [ "sh", "-c", "chown -R 1000:1000 /vault/logs" ]
    volumeMounts:
      - name: logs
        mountPath: /vault/logs

Additional context
Add any other context about the problem here.

[sdYoo]

apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels": ---------------
creationTimestamp: "2023-10-30T07:41:15Z"
generation: 27
labels:
app.kubernetes.io/instance: security-vault
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: vault
name: vault
namespace: security
resourceVersion: "49245057"
uid: a08381cc-4a96-4543-90cd-dd8eeb79f22b
spec:
podManagementPolicy: Parallel
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: vault
app.kubernetes.io/name: vault
component: server
serviceName: vault-internal
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2023-11-06T08:02:25Z"
creationTimestamp: null
labels:
app.kubernetes.io/instance: vault
app.kubernetes.io/name: vault
component: server
helm.sh/chart: vault-0.24.1
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: vault
app.kubernetes.io/name: vault
component: server
topologyKey: kubernetes.io/hostname
containers:
- args:
- "cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;\n[
-n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;\n[
-n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;\n[
-n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;\n[
-n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;\n[
-n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g"
/tmp/storageconfig.hcl;\n[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g"
/tmp/storageconfig.hcl;\n/usr/local/bin/docker-entrypoint.sh vault server
-config=/tmp/storageconfig.hcl \n"
command:
- /bin/sh
- -ec
env:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: VAULT_K8S_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: VAULT_K8S_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: VAULT_ADDR
value: http://127.0.0.1:8200
- name: VAULT_API_ADDR
value: http://$(POD_IP):8200
- name: SKIP_CHOWN
value: "true"
- name: SKIP_SETCAP
value: "true"
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: VAULT_CLUSTER_ADDR
value: https://$(HOSTNAME).vault-internal:8201
- name: HOME
value: /home/vault
- name: VAULT_LOG_LEVEL
value: info
- name: VAULT_LOG_FORMAT
value: json
image: docker-hub.com/hashicorp/vault:1.13.1-jqcurl
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- /vault/file/auto-unseal.sh
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 5 && kill -SIGTERM $(pidof vault)
name: vault
ports:
- containerPort: 8200
name: http
protocol: TCP
- containerPort: 8201
name: https-internal
protocol: TCP
- containerPort: 8202
name: http-rep
protocol: TCP
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 250m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /vault/config
name: config
- mountPath: /vault/logs
name: logs
- mountPath: /home/vault
name: home
- mountPath: /etc/localtime
name: localtime
readOnly: true
- mountPath: /etc/timezone
name: timezone
readOnly: true
- mountPath: /vault/file
name: script
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- -c
- chown -R 1000:1000 /vault/logs
image: docker-hub.com/finalspy/busybox-curl-jq
imagePullPolicy: Always
name: busybox
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /vault/logs
name: logs
nodeSelector:
wallga/node-group: app
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 100
serviceAccount: vault
serviceAccountName: vault
terminationGracePeriodSeconds: 10
volumes:
- configMap:
defaultMode: 420
name: vault-config
name: config
- hostPath:
path: /home/logs/security/vault
type: ""
name: logs
- emptyDir: {}
name: home
- hostPath:
path: /etc/timezone
type: ""
name: timezone
- hostPath:
path: /etc/localtime
type: ""
name: localtime
- configMap:
defaultMode: 493
name: vault-auto-unseal
name: script
updateStrategy:
type: OnDelete
status:
availableReplicas: 0
collisionCount: 0
currentRevision: vault-6d68c9f774
observedGeneration: 27
replicas: 2
updateRevision: vault-684dcf5489
updatedReplicas: 2

[sdYoo]

I solved the permission issue, but an error occurs as below and the pod is pending.

===========
state:
terminated:
containerID: containerd://484566bf23bc63fe7f3f0c9474dd52226da130fd29137ee3f095a92569b17fe9
exitCode: 0
finishedAt: "2023-11-08T01:04:55Z"
reason: Completed
startedAt: "2023-11-08T01:04:55Z"
phase: Pending
podIP: 10.252.23.241
podIPs:

• ip: 10.252.23.241
  qosClass: Burstable
  startTime: "2023-11-08T01:03:18Z"
  ==================
  2023-11-08T11:08:24.474036169+09:00 stderr F {"@Level":"info","@message":"security barrier not initialized","@module":"core","@timestamp":"2023-11-08T11:08:24.473898+09:00"}
  2023-11-08T11:08:24.966796527+09:00 stderr F {"@Level":"info","@message":"security barrier not initialized","@module":"core","@timestamp":"2023-11-08T11:08:24.966662+09:00"}
  2023-11-08T11:08:25.463967073+09:00 stderr F {"@Level":"info","@message":"security barrier not initialized","@module":"core","@timestamp":"2023-11-08T11:08:25.463859+09:00"}