How to mount certificates to gitsync-1 initContainer of my kubernetesExecutors pods

[PaulienVa]

I have configured my kubernetes executors to mount a volume with certificates needed for the gitsync like stated in this docs: https://crds.stackable.tech/24.3.0/airflow.stackable.tech/airflowcluster/v1alpha1/#spec-webservers-podOverrides-spec-initContainers-volumeMounts

However whatever I do, my initContainer does not get this volume mounted. What do I need to do to get this done?

[adwk67]

Do you have an example custom resource you are using? (you can leave all the gitsync stuff out, just so I can try and see reproduce it).

[PaulienVa]

in my AirflowCluster I have configured the following:

kubernetesExecutors:
    podOverrides:
      spec:
        initContainers:
          - name: gitsync-1
            volumeMounts:
            - name: "ca-cert"
              mountPath: "/tmp/ca-cert"
        volumes:
          - name: "ca-cert"
            configMap:
              name: "ca-cert"
              items:
                - key: "ca.crt"
                  path: "ca.crt"

[adwk67]

Airflow reads the Secret and then writes the credentials to environment variables. This is how the database credential secret is used, for example, so you shouldn't have to mount it explicitly. But note also that the implementation for gitSync has to be (with the current implemntation) user + password. We have an issue open for allowing SSH access to gitSync (currently not planned).

[PaulienVa]

Nope we cannot deploy a redis instance

[PaulienVa]

@adwk67 could this maybe fix it : stackabletech/airflow-operator#455

[adwk67]

Thanks for tackling this! There are a few more moving parts to this, though, so I've opened up another PR and will start testing it once the helm chart has been built. You might find this section on our docs page useful if you would like to contribute in the future (which we very much encourage!). One of the components in the rust stack is the cargo tool, which is the easiest way to check low-hanging fruit (does the code compile, is it formatted, are the charts up-to-date, linting etc.).

[adwk67]

The PR is this one: stackabletech/airflow-operator#456. Just to explain a few things: there were actually two things missing - the podOverrides were not being passed through to the gitsync containers, and neither were the envOverrides in kubernetesExecutor-mode. There are actually three places where the gitsync container is created: for the two modes (kubernetesExecutor and celeryExecutor), which are different as one requires a template that is used by airflow, and the other is the pod we create ourselves, but also we need an initContainer as well as a sidecar for the celeryExecutor to avoid a race condition.

[PaulienVa]

Ah good to know, will check this out! Thank you for picking it up the change is indeed more elaborated. Thank you for explaining it 🙏

[adwk67]

You are very welcome!