[KYUUBI #6521] [K8S][HELM] Implement new configuration approach

# 🔍 Description ## Issue References 🔗 This pull request changes Helm chart configuration approach as discussed in #6123 ## Describe Your Solution 🔧 Suggested implementation makes chart configuration more general and more flexible. It allows to configure Kyuubi (and its engines) by setting configuration file contents to chart properties or by providing configuration files through existing ConfigMaps and Secrets. Also users are not limited by predefined number of files and can put any files to configuration directories. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 ### 1. Test suite "Kyuubi configuration" Property file `values-kyuubi-files.yaml` ```yaml kyuubiConf: dir: /opt/kyuubi/conf files: 'kyuubi-env.sh': | #!/usr/bin/env bash export KYUUBI_TEST=true 'kyuubi-custom.properties': | kyuubi.custom=true filesFrom: - configMap: name: kyuubi-configs ``` ConfigMap `kyuubi-configs` from `configmap-kyuubi-configs.yaml` ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kyuubi-configs data: 'kyuubi-test.properties': | kyuubi.config.test=true ``` #### Rendered templates are correct ```shell $ helm template charts/kyuubi -f values-kyuubi-files.yaml -s templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml ``` #### Configuration files are in place ```shell $ kubectl create -f configmap-kyuubi-configs.yaml $ helm install kyuubi charts/kyuubi -f values-kyuubi-files.yaml $ kubectl exec kyuubi-0 -- ls conf kyuubi-custom.properties kyuubi-env.sh kyuubi-test.properties ``` ### 2. Test suite "Spark configuration" Property file `values-spark-files.yaml` ```yaml sparkConf: dir: /opt/spark/conf files: 'spark-env.sh': | #!/usr/bin/env bash export SPARK_TEST=true 'spark-custom.properties': | spark.custom=true filesFrom: - configMap: name: spark-configs ``` ConfigMap `spark-configs` from `configmap-spark-configs.yaml` ```yaml apiVersion: v1 kind: ConfigMap metadata: name: spark-configs data: 'spark-test.properties': | spark.config.test=true ``` #### Rendered templates are correct ```shell $ helm template charts/kyuubi -f values-spark-files.yaml -s templates/kyuubi-spark-configmap.yaml -s templates/kyuubi-statefulset.yaml ``` #### Configuration files are in place ```shell $ kubectl create -f configmap-spark-configs.yaml $ helm install kyuubi charts/kyuubi -f values-spark-files.yaml $ kubectl exec kyuubi-0 -- ls ../spark/conf spark-custom.properties spark-env.sh spark-test.properties ``` 3. Test suite "Custom kyuubi-defaults.conf from existing ConfigMap overwrites kyuubi-defaults.conf from values" Property file `values-kyuubi-defaults.yaml` ```yaml kyuubiConf: dir: /opt/kyuubi/conf files: 'kyuubi-defaults.conf': | custom.from.values=true filesFrom: - configMap: name: kyuubi-defaults-config ``` ConfigMap `kyuubi-defaults-config` from `configmap-kyuubi-defaults.yaml` ```yaml apiVersion: v1 kind: ConfigMap metadata: name: kyuubi-defaults-config data: 'kyuubi-defaults.conf': | custom.from.configmap=true ``` #### Rendered templates are correct ```shell $ helm template charts/kyuubi -f values-kyuubi-defaults.yaml -s templates/kyuubi-configmap.yaml -s templates/kyuubi-statefulset.yaml ``` #### Content of `kyuubi-defaults.conf` comes from ConfigMap ```shell $ kubectl create -f configmap-kyuubi-defaults.yaml $ helm install kyuubi charts/kyuubi -f values-kyuubi-defaults.yaml $ kubectl exec kyuubi-0 -- ls conf kyuubi-defaults.conf $ kubectl exec kyuubi-0 -- cat conf/kyuubi-defaults.conf custom.from.configmap=true ``` --- # Checklist 📝 - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** Closes #6521 from dnskr/implement-new-helm-chart-configuration-approach. Closes #6521 452dca3 [dnskr] Fix empty value type 14829f3 [dnskr] Revert "[REVERT BEFORE MERGE] Use 'master-snapshot' image tag" 8d90f42 [dnskr] Move default properties from 'kyuubi-defaults.conf' to --conf args 6b3c77f [dnskr] [REVERT BEFORE MERGE] Use 'master-snapshot' image tag fe7c17a [dnskr] [K8S][HELM] Implement new configuration approach Authored-by: dnskr <[email protected]> Signed-off-by: Kent Yao <[email protected]>
apache · Nov 22, 2024 · b0036fd · b0036fd
1 parent 160bf58
commit b0036fd
Show file tree

Hide file tree

Showing 4 changed files with 79 additions and 98 deletions.
diff --git a/charts/kyuubi/templates/kyuubi-configmap.yaml b/charts/kyuubi/templates/kyuubi-configmap.yaml
@@ -22,30 +22,6 @@ metadata:
   labels:
     {{- include "kyuubi.labels" . | nindent 4 }}
 data:
-  {{- with .Values.kyuubiConf.kyuubiEnv }}
-  kyuubi-env.sh: |
-    {{- tpl . $ | nindent 4 }}
-  {{- end }}
-  kyuubi-defaults.conf: |
-    ## Helm chart provided Kyuubi configurations
-    kyuubi.kubernetes.namespace={{ .Release.Namespace }}
-    kyuubi.frontend.connection.url.use.hostname=false
-    kyuubi.frontend.thrift.binary.bind.port={{ .Values.server.thriftBinary.port }}
-    kyuubi.frontend.thrift.http.bind.port={{ .Values.server.thriftHttp.port }}
-    kyuubi.frontend.rest.bind.port={{ .Values.server.rest.port }}
-    kyuubi.frontend.mysql.bind.port={{ .Values.server.mysql.port }}
-    kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
-
-    # Kyuubi Metrics
-    kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
-    kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
-    kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
-
-    ## User provided Kyuubi configurations
-    {{- with .Values.kyuubiConf.kyuubiDefaults }}
-      {{- tpl . $ | nindent 4 }}
-    {{- end }}
-  {{- with .Values.kyuubiConf.log4j2 }}
-  log4j2.xml: |
-    {{- tpl . $ | nindent 4 }}
+  {{- with .Values.kyuubiConf.files }}
+    {{- tpl (toYaml .) $ | nindent 2 }}
   {{- end }}
diff --git a/charts/kyuubi/templates/kyuubi-spark-configmap.yaml b/charts/kyuubi/templates/kyuubi-spark-configmap.yaml
@@ -22,19 +22,6 @@ metadata:
   labels:
     {{- include "kyuubi.labels" . | nindent 4 }}
 data:
-  {{- with .Values.sparkConf.sparkEnv }}
-  spark-env.sh: |
-    {{- tpl . $ | nindent 4 }}
-  {{- end }}
-  {{- with .Values.sparkConf.sparkDefaults }}
-  spark-defaults.conf: |
-    {{- tpl . $ | nindent 4 }}
-  {{- end }}
-  {{- with .Values.sparkConf.log4j2 }}
-  log4j2.properties: |
-    {{- tpl . $ | nindent 4 }}
-  {{- end }}
-  {{- with .Values.sparkConf.metrics }}
-  metrics.properties: |
-    {{- tpl . $ | nindent 4 }}
+  {{- with .Values.sparkConf.files }}
+    {{- tpl (toYaml .) $ | nindent 2 }}
   {{- end }}
diff --git a/charts/kyuubi/templates/kyuubi-statefulset.yaml b/charts/kyuubi/templates/kyuubi-statefulset.yaml
@@ -59,14 +59,28 @@ spec:
           {{- with .Values.command }}
           command: {{- tpl (toYaml .) $ | nindent 12 }}
           {{- end }}
-          {{- with .Values.args }}
+          {{- if .Values.args }}
           args: {{- tpl (toYaml .) $ | nindent 12 }}
+          {{- else }}
+          args:
+            - ./bin/kyuubi
+            - run
+            - --conf kyuubi.kubernetes.namespace={{ .Release.Namespace }}
+            - --conf kyuubi.frontend.connection.url.use.hostname=false
+            - --conf kyuubi.frontend.thrift.binary.bind.port={{ .Values.server.thriftBinary.port }}
+            - --conf kyuubi.frontend.thrift.http.bind.port={{ .Values.server.thriftHttp.port }}
+            - --conf kyuubi.frontend.rest.bind.port={{ .Values.server.rest.port }}
+            - --conf kyuubi.frontend.mysql.bind.port={{ .Values.server.mysql.port }}
+            - --conf kyuubi.frontend.protocols={{ include "kyuubi.frontend.protocols" . }}
+            - --conf kyuubi.metrics.enabled={{ .Values.metrics.enabled }}
+            - --conf kyuubi.metrics.reporters={{ .Values.metrics.reporters }}
+            - --conf kyuubi.metrics.prometheus.port={{ .Values.metrics.prometheusPort }}
           {{- end }}
           env:
             - name: KYUUBI_CONF_DIR
-              value: {{ .Values.kyuubiConfDir }}
+              value: {{ .Values.kyuubiConf.dir }}
             - name: SPARK_CONF_DIR
-              value: {{ .Values.sparkConfDir }}
+              value: {{ .Values.sparkConf.dir }}
             {{- with .Values.env }}
               {{- tpl (toYaml .) $ | nindent 12 }}
             {{- end }}
@@ -109,9 +123,9 @@ spec:
           {{- end }}
           volumeMounts:
             - name: conf
-              mountPath: {{ .Values.kyuubiConfDir }}
+              mountPath: {{ .Values.kyuubiConf.dir }}
             - name: conf-spark
-              mountPath: {{ .Values.sparkConfDir }}
+              mountPath: {{ .Values.sparkConf.dir }}
             {{- with .Values.volumeMounts }}
               {{- tpl (toYaml .) $ | nindent 12 }}
             {{- end }}
@@ -120,11 +134,21 @@ spec:
         {{- end }}
       volumes:
         - name: conf
-          configMap:
-            name: {{ .Release.Name }}
+          projected:
+            sources:
+              - configMap:
+                  name: {{ .Release.Name }}
+              {{- with .Values.kyuubiConf.filesFrom }}
+                {{- tpl (toYaml .) $ | nindent 14 }}
+              {{- end }}
         - name: conf-spark
-          configMap:
-            name: {{ .Release.Name }}-spark
+          projected:
+            sources:
+              - configMap:
+                  name: {{ .Release.Name }}-spark
+              {{- with .Values.sparkConf.filesFrom }}
+                {{- tpl (toYaml .) $ | nindent 14 }}
+              {{- end }}
         {{- with .Values.volumes }}
           {{- tpl (toYaml .) $ | nindent 8 }}
         {{- end }}

diff --git a/charts/kyuubi/values.yaml b/charts/kyuubi/values.yaml
@@ -145,53 +145,40 @@ server:
       #    clientIP:
       #      timeoutSeconds: 10800
 
-# $KYUUBI_CONF_DIR directory
-kyuubiConfDir: /opt/kyuubi/conf
 # Kyuubi configuration files
 kyuubiConf:
-  # The value (templated string) is used for kyuubi-env.sh file
-  # See example at conf/kyuubi-env.sh.template and https://kyuubi.readthedocs.io/en/master/configuration/settings.html#environments for more details
-  kyuubiEnv: ~
-  #  kyuubiEnv: |
-  #    #!/usr/bin/env bash
-  #    export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
-  #    export SPARK_HOME=/opt/spark
-  #    export FLINK_HOME=/opt/flink
-  #    export HIVE_HOME=/opt/hive
-
-  # The value (templated string) is used for kyuubi-defaults.conf file
-  # See https://kyuubi.readthedocs.io/en/master/configuration/settings.html#kyuubi-configurations for more details
-  kyuubiDefaults: ~
-  #  kyuubiDefaults: |
+  # $KYUUBI_CONF_DIR directory
+  dir: /opt/kyuubi/conf
+  # Configuration files from the specified keys (file name) and values (file content)
+  files: ~
+  #files:
+  #  'kyuubi-defaults.conf': |
   #    kyuubi.authentication=NONE
-  #    kyuubi.frontend.bind.host=10.0.0.1
-  #    kyuubi.engine.type=SPARK_SQL
   #    kyuubi.engine.share.level=USER
-  #    kyuubi.session.engine.initialize.timeout=PT3M
-  #    kyuubi.ha.addresses=zk1:2181,zk2:2181,zk3:2181
-  #    kyuubi.ha.namespace=kyuubi
 
-  # The value (templated string) is used for log4j2.xml file
-  # See example at conf/log4j2.xml.template https://kyuubi.readthedocs.io/en/master/configuration/settings.html#logging for more details
-  log4j2: ~
-
-# $SPARK_CONF_DIR directory
-sparkConfDir: /opt/spark/conf
-# Spark configuration files
+  # Configuration files from the list of existing ConfigMaps and Secrets
+  filesFrom: []
+  #filesFrom:
+  #- configMap:
+  #    name: kyuubi-configs
+  #- secret:
+  #    name: kyuubi-secrets
+  #- secret:
+  #    name: ssl-secrets
+  #    items:
+  #      - key: key-store
+  #        path: certs/keystore.jks
+  #      - key: trust-store
+  #        path: certs/truststore.jks
+
+# Spark configuration, see https://github.com/apache/spark/tree/master/conf and Spark documentation for more details
 sparkConf:
-  # The value (templated string) is used for spark-env.sh file
-  # See example at https://github.com/apache/spark/blob/master/conf/spark-env.sh.template and Spark documentation for more details
-  sparkEnv: ~
-  #  sparkEnv: |
-  #    #!/usr/bin/env bash
-  #    export JAVA_HOME=/usr/jdk64/jdk1.8.0_152
-  #    export SPARK_LOG_DIR=/opt/spark/logs
-  #    export SPARK_LOG_MAX_FILES=5
-
-  # The value (templated string) is used for spark-defaults.conf file
-  # See example at https://github.com/apache/spark/blob/master/conf/spark-defaults.conf.template and Spark documentation for more details
-  sparkDefaults: ~
-  #  sparkDefaults: |
+  # $SPARK_CONF_DIR directory
+  dir: /opt/spark/conf
+  # Configuration files from the specified keys (file name) and values (file content)
+  files: ~
+  #files:
+  #  'spark-defaults.conf': |
   #    spark.submit.deployMode=cluster
   #    spark.kubernetes.container.image=apache/spark:3.5.0
   #    spark.kubernetes.authenticate.driver.serviceAccountName=spark
@@ -207,13 +194,20 @@ sparkConf:
   #    spark.hadoop.fs.s3a.path.style.access=true
   #    spark.hadoop.fs.s3a.fast.upload=true
 
-  # The value (templated string) is used for log4j2.properties file
-  # See example at https://github.com/apache/spark/blob/master/conf/log4j2.properties.template and Spark documentation for more details
-  log4j2: ~
-
-  # The value (templated string) is used for metrics.properties file
-  # See example at https://github.com/apache/spark/blob/master/conf/metrics.properties.template and Spark documentation for more details
-  metrics: ~
+  # Configuration files from the list of existing ConfigMaps and Secrets
+  filesFrom: []
+  #filesFrom:
+  #- configMap:
+  #    name: spark-configs
+  #- secret:
+  #    name: spark-secrets
+  #- secret:
+  #    name: ssl-secrets
+  #    items:
+  #      - key: key-store
+  #        path: certs/keystore.jks
+  #      - key: trust-store
+  #        path: certs/truststore.jks
 
 # Command to launch Kyuubi server (templated)
 command: ~