🌱 Dynamic measured boot and runtime attestation

[mudler]

This is about integrating keylime into Kairos. Besides Marble that provides a Secure environment for the user-space ( tracked in #1257 ), this is to protect the system integrity by relying on Linux IMA mechanisms, for example.

Related to #2166

Acceptance criteria

• we have documentation on how to run dynamic measured boot with Kairos
• we have documentation on how to run runtime attestation with Linux IMA with keylime
• we have bundles, or in images directly included the keylime agent to run assessments
• we have a full e2e example in our docs

[bencorrado]

For others looking at this here is what I am testing with currently for Keylime in the Dockerfile for Kairos

###############################################################
####                  Keylime Build Stage                  ####
###############################################################
# Use a separate stage to build Keylime so build dependencies don't stay in the final image
FROM ubuntu:24.04 AS keylime-build

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    build-essential \
    git \
    cargo \
    libssl-dev \
    pkg-config \
    libclang-dev \
    libssl-dev \
    libtss2-dev \
    pkg-config \
    libzmq3-dev \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Clone and build Keylime
RUN git clone https://github.com/keylime/rust-keylime.git /opt/keylime \
    && cd /opt/keylime \
    && cargo build --release

then in the FROM common AS all

# Copy the Keylime agent binary and related files from the build stage
COPY --from=keylime-build /opt/keylime/target/release/keylime_agent /usr/local/bin/keylime_agent
COPY --from=keylime-build /opt/keylime/target/release/keylime_ima_emulator /usr/local/bin/keylime_ima_emulator

# Copy configuration files and systemd service units from the build stage
COPY --from=keylime-build /opt/keylime/keylime-agent.conf /etc/keylime/agent.conf
COPY --from=keylime-build /opt/keylime/dist/systemd/system/keylime_agent.service /etc/systemd/system/keylime_agent.service
COPY --from=keylime-build /opt/keylime/dist/systemd/system/var-lib-keylime-secure.mount /etc/systemd/system/var-lib-keylime-secure.mount
COPY --from=keylime-build /opt/keylime/keylime-agent/tests/actions/shim.py /usr/libexec/keylime/shim.py

RUN systemctl enable keylime_agent.service

And add in packages PERSISTENT_STATE_PATHS \etc\keylime

[Itxaka]

we should start by providing the agent as a luet package if possible so it can be installed as part of the build process, maybe even on the framework if needed.

then setup and test if it works :D

[Itxaka]

package is up: https://packages.kairos.io/Kairos/utils/keylime-agent/0.2.6/

seems like we also need to add /var/lib/keystone to writable paths.
Still looking if it supports overrides via /etc/keystone.d/ files which would be nice :D

[Itxaka]

seems like the makefile install target creates the dir /etc/keylime/agent.conf.d so maybe its possible to override stuff by creating files there and leaving the default config as-is

[Itxaka]

yes, its possible so I got a working keylime kairos version (working in the sense that it can connect :D)

With a Dockerfile as such (with an up to date framework that has the keylime package available! Not released yet):

FROM quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.2.1 AS base
RUN luet install -y --relax utils/keylime-agent

And then a cloud config as such:

#cloud-config

install:
  auto: true
  reboot: true
  device: /dev/vda
  bind_mounts:
    - /var/lib/keylime
  grub_options:
    extra_cmdline: "rd.immucore.debug"

# Keylime needs several things to work
# User keylime which is part of the tss group. Does not need to be able to login or have password,
# its mainly used for the agent to drop pivs to when runs as a service
# the /var/lib/keylime dir writable and owned by the keylime user
# Any config dropped at /etc/keylime/agent.conf.d/
# So registrar IP for example or verifier, etc...
# any extra config needs to be dropped there
stages:
  initramfs:
    - name: "Set user and password"
      users:
        kairos:
          passwd: "kairos"
          groups:
            - "admin"
        keylime:
          groups:
            - "tss"
      hostname: kairos-{{ trunc 4 .Random }}
  boot:  # Do it on boot, at this point the bind mount for /var/lib/keylime is done and the user created
    - name: "Set keylime owner to /var/lib/keylime"
      commands:
      - chown keylime /var/lib/keylime
    - name: "Keylime config"
      files:
        - path: /etc/keylime/agent.conf.d/10-config.conf
          content: |
            [agent]
            ip = '0.0.0.0'
            registrar_ip = '<Registrar IP here>'
            uuid = '<UUID here>'
          owner_string: "keylime"

On boot you get the agent auto connected to your keylime registrar:

2024-10-25 09:00:36.127 - keylime.registrar - WARNING - Overriding ek_tpm for agent 55400648-4ad5-428b-a549-c18485ba4ef4 from ekcert
2024-10-25 09:00:36.127 - keylime.tpm - INFO - Encrypting AIK with EK for UUID 55400648-4ad5-428b-a549-c18485ba4ef4
2024-10-25 09:00:36.128 - keylime.registrar - INFO - Overwriting previous registration for this UUID.
2024-10-25 09:00:36.158 - keylime.registrar - INFO - POST returning key blob for agent_id: 55400648-4ad5-428b-a549-c18485ba4ef4
2024-10-25 09:00:36.275 - keylime.registrar - INFO - PUT activated: 55400648-4ad5-428b-a549-c18485ba4ef4

And service is up and running:

● keylime-agent.service - The Keylime compute agent
     Loaded: loaded (/etc/systemd/system/keylime-agent.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-10-25 09:00:36 UTC; 3min 39s ago
   Main PID: 1794 (keylime_agent)
      Tasks: 10 (limit: 18585)
     Memory: 12.3M (peak: 13.7M)
        CPU: 316ms
     CGroup: /system.slice/keylime-agent.service
             └─1794 /usr/bin/keylime_agent

Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  WARN  keylime_agent              > INSECURE: Keylime is currently using a software TPM emulator rather than a real hardware TPM.
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  WARN  keylime_agent              > INSECURE: The security of Keylime is NOT linked to a hardware root of trust.
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  WARN  keylime_agent              > INSECURE: Only use Keylime in this mode for testing or debugging purposes.
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent              > Loaded old AK key from /var/lib/keylime/agent_data.json
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent              > Agent UUID: 55400648-4ad5-428b-a549-c18485ba4ef4
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent::registrar_agent > Requesting agent registration from http://192.168.122.161:8890/v2.2/agents/55400648-4ad5-428b-a549-c18485ba4ef4 for 55400648-4ad5-428b-a549-c18485ba4ef4
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent                  > SUCCESS: Agent 55400648-4ad5-428b-a549-c18485ba4ef4 registered
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent::registrar_agent > Requesting agent activation from http://192.168.122.161:8890/v2.2/agents/55400648-4ad5-428b-a549-c18485ba4ef4 for 55400648-4ad5-428b-a549-c18485ba4ef4
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent                  > SUCCESS: Agent 55400648-4ad5-428b-a549-c18485ba4ef4 activated
Oct 25 09:00:36 kairos-nppf keylime_agent[1794]:  INFO  keylime_agent                  > Listening on https://0.0.0.0:9002

[Itxaka]

So I finally get an example ongoing in which I tested the attestation and the failure of it.

TBH, most of this is configuration done by the Keylime admin, not much special or specific from Kairos.

PR at #2981

Most important thing is the configuration of the user permissions and to make sure the root CA from the registrar/verfier/tenant is properly deployed into the node. Then its just normal keylime config from the tenant, adding the node to the verifier with whatever config you want.

Im not sure we have to go much further than this?

[jimmykarily]

Let's clarify what the outcome of this story should be:

We want to measure specific binaries (e.g. immucore, kairos-agent) during boot and bind the unlocking of a PCR register (11?) to the measurement of those binaries. If they change, we want the keylime server to prevent booting by not letting PCR 11 unlock (thus the disks won't be decrypted).

Probably this is only a matter of documentation. We can also ship the keylime agent by default (put it in the framework or do it here).

Another scenario (probably needs a new ticket) is that we want to constantly monitor and measure specific binaries. If they change, keylime has the ability to send over a script and execute it on that node (or all other nodes) so that the tampered node is blacklisted. What that script actually does, depends on the cluster type. E.g. in Kubernetes it should somehow remove that node (how?), in edgevpn, rotate the token, in incus there is a remote management key that can be revoked etc. For this one we should also look at spiffe.
Let's extract this second scenario to another ticket and experiment when we are done with the first part.

[Itxaka]

We want to measure specific binaries (e.g. immucore, kairos-agent) during boot and bind the unlocking of a PCR register (11?) to the measurement of those binaries. If they change, we want the keylime server to prevent booting by not letting PCR 11 unlock (thus the disks won't be decrypted).

we can do this but its a bit absurd. first, PCRs are not free as other things use them so using it ourselves to measure things onto them its ok, but it can lead to something else measuring in there and breaking our stuff.

Second, for measuring the integrity of files and such IMA is in place I think, it just measures files every time they are accessed and you can create an allowlist and so on so it will refuse to run those commands if they dont match.

Also, this is for uki or non-uki? because it seems like we are mixing things here. disk being decrypted by PCR measurements are a uki thing only. I think for UKI the PCR11 checks plus secureboot is more than enough and the way to go and expanding on that will come with time with pcrlock.

For non-uki yes we are missing something that ties the system integrity to the disk unlocking, but Im not sure if IMA is good enough for this?

For that, we would need to run the agent during initramfs, which means bundling the agent in there as well, as well as the configs and check that IMA measurements work on initramfs as expected? And then abort the initrd if they dont.

But still to just measure our binaries, we have sha files already, we dont need keylime for something like that? We are shipping an immutable system, on the time of building we can bundle the sha files and on boot have them checked before running the binaries and abort if they dont match.

Like Im not really seeing the advantage on doing just the measurements of our binaries, it should be the full system.

But still, there is no Kairos features at play here unless we want to go really deep with the keylime integration. Keylime wors for linux distros, no matter what the underlying distro is, it has been proven to work under kairos, then the implementation details depend on the consumer no?

Maybe we should have an specific meeting for this because I cant see anything that would require further kairos dev time for this.

[Itxaka]

btw once ima is enabled, everything is measured:

root@kairos-iq2u:~# cat /sys/kernel/security/ima/ascii_runtime_measurements |grep immucore
10 f626e603427b4188dbee2f3729cbbd579f48e973 ima-sig sha256:75010d51f7ddadca8ff4233166b9db9fb537d1a95714e0cbcf486a914469cd02 /usr/lib/systemd/system-generators/immucore-generator 
10 729166e6f21dda071830a65b61adde873fcd16f1 ima-sig sha256:cc81491f7df25aa38a9bf7effa9b7dfb16d45ba41a199d5bf13a11612783d733 /usr/lib/systemd/system/immucore.service 
10 e9bf9e9b4d75763cd30c9e0e2970663e413a781b ima-sig sha256:d189e52159906d85a84b3edb04a262ced42677f758e8e8aeaed450700ebd060b /usr/bin/immucore 

So thats good and all but I have no idea how this fits in our usecase or how we can provide value on top of this.

The only value I can see its providing an initramfs module or runner (part of immucore?) that enables the agent and the verification and blocks on boot, but I dont even know if this is possible. At least yoou need 1 correct boot so the tenant can communicate with the agent and validate the stuff and the verifier start verufying before being able to block the boot.

[Itxaka]

We have discussed about this and it seems that keylime does not fit into the Kairos system specially on how we want to use it for.
We provide a luet package for the agent and some examples under the Kairos repo for users to consume and have a guide for it but we wont go any further into this as it seems like implementing this is very generic and can fit not only Kairos but any other linux systems, so we dont want to expand much on tihs. Anyone using keylime will find that it works under Kairos as any other distro and that he can implement the runtime details themselves (like policies, and revokation scripts and so on)

What we are looking for its to have attestation for the initramfs, in the case of non-uki, when using encrypted partitions. so the unlocking is not automatic but done based on some system measurements, as to not have a potentially modified system unlocking the encrypted partitions.

we will follow up this on a card on trying to implement boot attestation for non-uki encrypted systems during initramfs (maybe use tpm2 to lock the key?)

@bencorrado Not sure how this fits on your usecase or if you need more info about this. Im closing this to open the follow ups, but feel free to open new tickets with specific cases or uses that specifically Kairos could improve upon

cheers!