systemctl reload caddy.service fails with log output dial fd: unknown network fd when using socket activation

[eriksjolund]

I built git commit a211c65 (master branch as of today)

1. Install caddy executable to /usr/local/bin/caddy
2. Run command

   sudo groupadd --system caddy
   

3. Run command

   sudo useradd --system \
      --gid caddy \
      --create-home \
      --home-dir /var/lib/caddy \
      --shell /usr/sbin/nologin \
      --comment "Caddy web server" \
      caddy`
   

4. Create file /etc/caddy/Caddyfile containing

   {
   	auto_https disable_redirects
   	admin fd/4
   }
   
   http://test.example.com {
   	bind fd/3 {
   		protocols h1
   	}
   	log
   	respond "hello world
   "
   }
   

5. Create file /etc/systemd/system/caddy.service containing

   [Unit]
   Description=Caddy
   Documentation=https://caddyserver.com/docs/
   After=network.target network-online.target
   Requires=network-online.target
   
   [Service]
   Type=notify
   User=caddy
   Group=caddy
   ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
   ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
   TimeoutStopSec=5s
   LimitNOFILE=1048576
   PrivateTmp=true
   ProtectSystem=full
   AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
   
   [Install]
   WantedBy=multi-user.target
   

   It's the same content as
   https://github.com/caddyserver/dist/blob/34f51d87d16c3da70156edffdf7c52a6916f48e6/init/caddy.service#L16-L35
   but /usr/bin/caddy replaced with /usr/local/bin/caddy
6. Create file /etc/systemd/system/caddy.socket

   [Socket]
   ListenStream=0.0.0.0:8077
   ListenStream=%t/caddy.sock
   [Install]
   WantedBy=sockets.target
   

7. Run command

   sudo systemctl daemon-reload
   

8. Run command

   sudo systemctl start caddy.socket`
   

9. Run command

   curl --resolve test.example.com:8077:127.0.0.1 http://test.example.com:8077
   

   The following output is printed

   hello world
   

10. Run command

    sudo systemctl reload caddy.service
    

    The following output is printed

    Job for caddy.service failed.
    See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.
    

11. Run command

     sudo journalctl -xeu caddy.service --no-pager | tail -79
    

    The following output is printed

    Oct 15 17:26:25 fcos-next5 systemd[1]: Starting caddy.service - Caddy...
    ░░ Subject: A start job for unit caddy.service has begun execution
    ░░ Defined-By: systemd
    ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ░░ 
    ░░ A start job for unit caddy.service has begun execution.
    ░░ 
    ░░ The job identifier is 44912.
    Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.HomeDir=/var/lib/caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
    Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.Version=(devel)
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOOS=linux
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOARCH=arm64
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Compiler=gc
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.NumCPU=2
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOMAXPROCS=2
    Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Version=go1.23.2
    Oct 15 17:26:26 fcos-next5 caddy[43845]: os.Getwd=/
    Oct 15 17:26:26 fcos-next5 caddy[43845]: LANG=C.UTF-8
    Oct 15 17:26:26 fcos-next5 caddy[43845]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
    Oct 15 17:26:26 fcos-next5 caddy[43845]: NOTIFY_SOCKET=/run/systemd/notify
    Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_PID=43845
    Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDS=2
    Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDNAMES=caddy.socket:caddy.socket
    Oct 15 17:26:26 fcos-next5 caddy[43845]: USER=caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: LOGNAME=caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: HOME=/var/lib/caddy
    Oct 15 17:26:26 fcos-next5 caddy[43845]: INVOCATION_ID=3b389f56fa5645e19bcc47c90e4df4c0
    Oct 15 17:26:26 fcos-next5 caddy[43845]: JOURNAL_STREAM=9:242885
    Oct 15 17:26:26 fcos-next5 caddy[43845]: SYSTEMD_EXEC_PID=43845
    Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
    Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1250536,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.133084,"msg":"adapted config to JSON","adapter":"caddyfile"}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1350574,"logger":"admin","msg":"admin endpoint started","address":"fd/4","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.135701,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1358926,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400063cb00"}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1370368,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.137156,"msg":"serving initial configuration"}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1375623,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"55170","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"test.example.com:8077","uri":"/","headers":{"User-Agent":["curl/8.9.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.000027332,"size":12,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain; charset=utf-8"]}}
    Oct 15 17:26:26 fcos-next5 systemd[1]: Started caddy.service - Caddy.
    ░░ Subject: A start job for unit caddy.service has finished successfully
    ░░ Defined-By: systemd
    ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ░░ 
    ░░ A start job for unit caddy.service has finished successfully.
    ░░ 
    ░░ The job identifier is 44912.
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1550741,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"896930ed-628f-4cbf-8f97-981466427717","try_again":1729099586.1550682,"try_again_in":86399.999999042}
    Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1552908,"logger":"tls","msg":"finished cleaning storage units"}
    Oct 15 17:26:39 fcos-next5 systemd[1]: Reloading caddy.service - Caddy...
    ░░ Subject: A reload job for unit caddy.service has begun execution
    ░░ Defined-By: systemd
    ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ░░ 
    ░░ A reload job for unit caddy.service has begun execution.
    ░░ 
    ░░ The job identifier is 45045.
    Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.412728,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
    Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.4152887,"msg":"adapted config to JSON","adapter":"caddyfile"}
    Oct 15 17:26:39 fcos-next5 caddy[43855]: Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd
    Oct 15 17:26:39 fcos-next5 systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
    ░░ Subject: Unit process exited
    ░░ Defined-By: systemd
    ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ░░ 
    ░░ An ExecReload= process belonging to unit caddy.service has exited.
    ░░ 
    ░░ The process' exit code is 'exited' and its exit status is 1.
    Oct 15 17:26:39 fcos-next5 systemd[1]: Reload failed for caddy.service - Caddy.
    ░░ Subject: A reload job for unit caddy.service has finished
    ░░ Defined-By: systemd
    ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
    ░░ 
    ░░ A reload job for unit caddy.service has finished.
    ░░ 
    ░░ The job identifier is 45045 and the job result is failed.
    

Describe the results you received

The command in Step 10 failed (sudo systemctl reload caddy.service)

In the log output there is the error message

 Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd

Describe the results you expected

I would have expected the command in Step 10 to succeed.

Additional information

I tried the same reproducer but with /etc/systemd/system/caddy.socket
having the following content

[Socket]
ListenStream=0.0.0.0:8077
ListenStream=%t/caddy.sock
SocketUser=caddy
SocketGroup=caddy
[Install]
WantedBy=sockets.target

The error was the same.

About the system

Fedora CoreOS 41.20241006.1.1

# rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Tue 2024-10-15 17:10:11 UTC)
Deployments:
● fedora:fedora/aarch64/coreos/next
                  Version: 41.20241006.1.1 (2024-10-08T00:00:38Z)
               BaseCommit: 0dae4f8983c2c67147e4e44dcb9e0183ee8d1a301a2047fe30be33937727c7ed
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
      RemovedBasePackages: containerd 1.7.22-2.fc41 moby-engine 27.3.1-1.fc41 runc 2:1.1.12-4.fc41

  fedora:fedora/aarch64/coreos/next
                  Version: 40.20240906.1.0 (2024-09-10T00:50:59Z)
               BaseCommit: 0db203f6e5bdc3ec6ce57e200c84d4506e12e641c681f2212ef77e8019f1381b
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
      RemovedBasePackages: containerd 1.6.23-5.fc40 moby-engine 24.0.5-4.fc40 runc 2:1.1.12-3.fc40

[mholt]

So, this is a good question, how do we connect to "fd 4" from an external process?

[septatrix]

So, this is a good question, how do we connect to "fd 4" from an external process?

Is it possible to connect via /proc/$MAINPID/fd/4?

[septatrix]

Otherwise it is also possible to alternatively implement Type=notify-reload in caddy.