External TLS cert/key not auto-updating for Hosted CE

[bbockelm]

I noticed the following in the Kubernetes docs:

Note: A container using a Secret as a subPath volume mount will not receive Secret updates.

Unfortunately, subPath is exactly how we handle the mounting of hostcert.pem and hostkey.pem. This means Let's Encrypt updates from cert-manager of the Secret does not propagate into the pod.

I think potentially the best way to do this is to mount the secret in a separate directory, not use subPath, and then point the hosted CE configuration at these files.

[LincolnBryant]

Sounds good to me

[LincolnBryant]

@brianhlin thoughts on how you want to proceed here? I see two paths..

1. We mount the secret somewhere, and then symlink to hostcert.pem and hostkey.pem. We would need to add another little script at startup to patch that together. I presume this would go into the image config scripts that we get from software base.
2. We mount the secret somewhere, and then change the HostedCE configuration appropriately. Perhaps a drop-in configmap for the CE configuration?

[brianhlin]

I think I like option 2 and we can just set the following with updated paths:

GSI_DAEMON_CERT = /etc/grid-security/hostcert.pem
GSI_DAEMON_KEY = /etc/grid-security/hostkey.pem

If we do 2, though, are we going to be able to set the permissions on the cert/key properly?

[LincolnBryant]

@brianhlin Ugh, yeah, I forgot about that whole problem. Thinking..

[LincolnBryant]

OK, will experiment and return with a PR!

[LincolnBryant]

So I did a bit of looking and thinking on this one-

We currently do the following:

      - name: osg-hosted-ce-hostcertkey-volume
        secret:
          secretName: {{ .Values.HostCredentials.HostCertKeySecret }}
          items:
          - key: tls.crt
            path: hostcert.pem
            mode: 256
          - key: tls.key
            path: hostkey.pem
            mode: 256

As far as permissions are concerned, all of this is above board as far as I can tell.

If we change the mount from

        {{ if .Values.HostCredentials.HostCertKeySecret }}
        - name: osg-hosted-ce-hostcertkey-volume
          mountPath: /etc/grid-security/hostcert.pem
          subPath: hostcert.pem
        - name: osg-hosted-ce-hostcertkey-volume
          mountPath: /etc/grid-security/hostkey.pem
          subPath: hostkey.pem

to

        {{ if .Values.HostCredentials.HostCertKeySecret }}
        - name: osg-hosted-ce-hostcertkey-volume
        mountPath: /media

(or whatever, /media is convenient, exists, and is usually empty)

You can poke the filesystem and see files are properly created with the right mode (256 dec = 0400 oct)

I made a nasty kubectl exec loop to just try to recursively look at /media while I waited between crashloops on my laziest/worst deployment of theHostedCE:

drwxrwxrwt 3 root root  120 Apr 30 20:38 .
drwxr-xr-x 1 root root 4096 Apr 30 20:42 ..
drwxr-xr-x 2 root root   80 Apr 30 20:38 ..2021_04_30_20_38_13.342530518
lrwxrwxrwx 1 root root   31 Apr 30 20:38 ..data -> ..2021_04_30_20_38_13.342530518
lrwxrwxrwx 1 root root   19 Apr 30 20:38 hostcert.pem -> ..data/hostcert.pem
lrwxrwxrwx 1 root root   18 Apr 30 20:38 hostkey.pem -> ..data/hostkey.pem

/media/..2021_04_30_20_38_13.342530518:
total 4
drwxr-xr-x 2 root root   80 Apr 30 20:38 .
drwxrwxrwt 3 root root  120 Apr 30 20:38 ..
-r-------- 1 root root    0 Apr 30 20:38 hostcert.pem
-r-------- 1 root root 1675 Apr 30 20:38 hostkey.pem

That looks like we ought to just be able to symlink to the right locations and things will be Fine (tm) ?

[brianhlin]

(or whatever, /media is convenient, exists, and is usually empty)

We should be able to pick whatever, k8s should make the dir for us, IIRC. I think @matyasselmeci has used /etc/grid-security/hostcert.orig.d/ or something to that effect.

-r-------- 1 root root 0 Apr 30 20:38 hostcert.pem

We may need this to be world-readable. I forget exactly when the SchedD drops privs and if that's after it reads in the host cert.

Otherwise I think This Is The Way so we'll have to make some minor adjustments to the container to create the symlinks if the certs live in whatever dir we choose.

[LincolnBryant]

We should be able to pick whatever, k8s should make the dir for us, IIRC. I think @matyasselmeci has used /etc/grid-security/hostcert.orig.d/ or something to that effect.

Works for me. Let's do that. We'll play along with whatever you guys have precedent for.

-r-------- 1 root root 0 Apr 30 20:38 hostcert.pem

We may need this to be world-readable. I forget exactly when the SchedD drops privs and if that's after it reads in the host cert.

OK, so mode 432 instead of 256. (thanks Kubernetes for making this so intuitive and easy 💩 )

Otherwise I think This Is The Way so we'll have to make some minor adjustments to the container to create the symlinks if the certs live in whatever dir we choose.

Sounds good.

[matyasselmeci]

This is what I use in the condor containers:

if [[ -f /etc/grid-security-orig.d/hostcert.pem && -f /etc/grid-security-orig.d/hostkey.pem ]]; then
    install -o root -g root -m 0644 /etc/grid-security-orig.d/hostcert.pem /etc/grid-security/hostcert.pem
    install -o root -g root -m 0600 /etc/grid-security-orig.d/hostkey.pem  /etc/grid-security/hostkey.pem
fi

[bbockelm]

@matyasselmeci - that probably should be executed every few hours, right?

[matyasselmeci]

Yep. I don't have cron in the original htcondor images but in OSG images I put a call to it in /etc/cron.hourly. (It's part of the update-secrets script which also updates passwords and tokens.)

[bbockelm]

Hi all! What's the next steps here? Not clear who's holding the ball on this.

[brianhlin]

Container work here: https://opensciencegrid.atlassian.net/browse/SOFTWARE-4623. I believe @LincolnBryant is going to tackle the Helm Chart work.

[LincolnBryant]

Indeed, I am trying to focus on this today/tomorrow and get this patch landed. Apologies for the slowness.

[LincolnBryant]

PR #496 should be a very simple patch that should address this specific issue, if the container is updated appropriately.