Custom ISO: RT_DOCKER_RESTART error on locally built ISO

[willgarrett610]

What Happened?

I need to build the minikube ISO file locally so I can add a kernel module. The build looks like it runs just fine and it generates the ISO for me.

When I run minikube with that ISO I get the following error:

😄  [test] minikube v1.33.1 on Darwin 14.5 (arm64)
✨  Using the qemu2 driver based on user configuration
👍  Starting "test" primary control-plane node in "test" cluster
🔥  Creating qemu2 VM (CPUs=2, Memory=5433MB, Disk=20000MB) ...
🔥  Deleting "test" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

🔥  Creating qemu2 VM (CPUs=2, Memory=5433MB, Disk=20000MB) ...
😿  Failed to start qemu2 VM. Running "minikube delete -p test" may fix it: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.


❌  Exiting due to RT_DOCKER_RESTART: Failed to start host: creating host: create: provisioning: ssh command error:
command : sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

💡  Suggestion: Remove the invalid --docker-opt or --insecure-registry flag if one was provided
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7070

The command I'm using is:

minikube start \
  --driver=qemu \
  --network socket_vmnet \
  --nodes 3 -p test \
  --iso-url=file://$(pwd)/minikube-arm64.iso

It does start and work fine when running without the --iso-url=file... flag

This is without adding any modules, just checking out v1.33.1 and running IN_DOCKER=1 make out/minikube-aarch64.iso

I run the ISO build in a podman container which I can provide information about if needed.

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

QEMU

[medyagh]

willgarrett610

your logs show that the docker.server file is not there

err     : Process exited with status 1
output  : diff: can't stat '/lib/systemd/system/docker.service': No such file or directory

you could ssh to the minikube using "minikbue ssh"
and see what files are there and check the logs,

good luck figuring it out, if that kernel module is something that many users need you could try to make a PR to have it added to minkube

you co do a minikube ssh
btw I am curious what kernel module are you adding ?

[willgarrett610]

@medyagh

I haven't been able to ssh into the minikube because it didn't start properly.

minikube ssh -p test

❌  Exiting due to DRV_CP_ENDPOINT: Unable to get control-plane node test endpoint: failed to lookup ip for ""
💡  Suggestion:

    Recreate the cluster by running:
    minikube delete <no value>
    minikube start <no value>

We need the OVS kernel module for what I'm working on. The same build process worked just fine for minikube v1.31.2 but does not work for v1.33.1.

I'm working on trying to run the image directly with qemu so I can inspect it some more

[willgarrett610]

@medyagh

So I managed to SSH into the vm that minikube set up using the raw ssh command from the minikube logs.

It looks like the docker.service file is there so I'm not sure why it's failing.

$ ls -al /lib/systemd/system | grep docker
-rw-r--r--  1 root root 2012 Jul 24 22:02 docker.service
-rw-r--r--  1 root root  197 Jul 23 20:44 docker.socket

[medyagh]

willgarrett610
whats the output of
"sudo systemctl status docker"

[willgarrett610]

@medyagh

Output of sudo systemctl status docker:

× docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-07-26 17:17:29 UTC; 4min 26s ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 458 ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.
   Main PID: 458 (code=exited, status=1/FAILURE)
        CPU: 23ms

Jul 26 17:17:29 test systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 17:17:29 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:29 test systemd[1]: Failed to start Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jul 26 17:17:29 test systemd[1]: Stopped Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: docker.service: Start request repeated too quickly.
Jul 26 17:17:29 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:29 test systemd[1]: Failed to start Docker Application Container Engine.

Looking at the logs with journalctl -u docker.service:

Jul 26 17:17:28 test systemd[1]: Starting Docker Application Container Engine...
Jul 26 17:17:28 test dockerd[442]: time="2024-07-26T17:17:28.403255377Z" level=info msg="Starting up"
Jul 26 17:17:28 test dockerd[442]: time="2024-07-26T17:17:28.403525211Z" level=info msg="containerd not running, starting managed containerd"
Jul 26 17:17:28 test dockerd[442]: failed to start containerd: exec: "containerd": executable file not found in $PATH
Jul 26 17:17:28 test systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 17:17:28 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:28 test systemd[1]: Failed to start Docker Application Container Engine.
Jul 26 17:17:28 test systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Jul 26 17:17:28 test systemd[1]: Stopped Docker Application Container Engine.
Jul 26 17:17:28 test systemd[1]: Starting Docker Application Container Engine...
Jul 26 17:17:28 test dockerd[450]: time="2024-07-26T17:17:28.780956919Z" level=info msg="Starting up"
Jul 26 17:17:28 test dockerd[450]: time="2024-07-26T17:17:28.781222086Z" level=info msg="containerd not running, starting managed containerd"
Jul 26 17:17:28 test dockerd[450]: failed to start containerd: exec: "containerd": executable file not found in $PATH
Jul 26 17:17:28 test systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 17:17:28 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:28 test systemd[1]: Failed to start Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: docker.service: Scheduled restart job, restart counter is at 2.
Jul 26 17:17:29 test systemd[1]: Stopped Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: Starting Docker Application Container Engine...
Jul 26 17:17:29 test dockerd[458]: time="2024-07-26T17:17:29.157120544Z" level=info msg="Starting up"
Jul 26 17:17:29 test dockerd[458]: time="2024-07-26T17:17:29.157429044Z" level=info msg="containerd not running, starting managed containerd"
Jul 26 17:17:29 test dockerd[458]: failed to start containerd: exec: "containerd": executable file not found in $PATH
Jul 26 17:17:29 test systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 17:17:29 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:29 test systemd[1]: Failed to start Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jul 26 17:17:29 test systemd[1]: Stopped Docker Application Container Engine.
Jul 26 17:17:29 test systemd[1]: docker.service: Start request repeated too quickly.
Jul 26 17:17:29 test systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 26 17:17:29 test systemd[1]: Failed to start Docker Application Container Engine.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten