caddy reload command returns successfully but does not work

[knacky34]

Environement

• Podman
• Image: Caddy-2.7.2-alpine

Step to reproduce

1. Create Caddyfile :

http://localhost
respond "Hello World"

1. Create container :

podman run -d \
	-v path/to/Caddyfile:/etc/caddy/Caddyfile:ro \
	-v /srv:/srv:ro \
	-v caddy_data:/data \
	-v caddy_config:/config \
	-p 80:80 \
	--name caddy docker.io/caddy-alpine

1. run curl localhost, output Hello World
2. Modify Caddyfile's respond line to respond "Goodbye World"
3. Run podman exec -w /etc/caddy caddy caddy reload
4. curl localhost, output Hello World instead of Goodbye World

Workaround
Restart the container with podman restart caddy

Log Output after run caddy reload

{"level":"info","ts":1691840974.3116052,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"54268","headers":{"Accept-Encoding":["gzip"],"Content-Length":["230"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":1691840974.3117096,"msg":"config is unchanged"}         # <-- Seems that it does not detect the value of respond changed
{"level":"info","ts":1691840974.311715,"logger":"admin.api","msg":"load complete"}

Also Tried
Seeing the log output, I think maybe it does not detect a change of the value in respond. So I tried adding a new directive, but it still does not work. Furthermore, the log output still say config is unchanged

1. Edit Caddyfile and add header Custom-Header "My value"
2. Run podman exec -w /etc/caddy caddy caddy reload
3. Run curl -v localhost, I should see the newly added header in the response but it's not there

[francislavoie]

And what's in your container logs (i.e. Caddy's stdout/stderr) after you attempt to reload?

[mholt]

Thanks for opening an issue! We'll look into this.

It's not immediately clear to me what is going on, so I'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible using the latest version of Caddy. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍 I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

This should be the latest version of Caddy:

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Instructions -- please heed otherwise we cannot help you (help us help you!)

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials. Please enable debug and access logs.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

[knacky34]

@mholt @francislavoie, I've updated my initial post to precise things.

[emilylange]

Pretty sure, that's due to how podman/docker handle bind mounts.

You should be able to compare the inode numbers with ls -i before and after your change made to the Caddyfile and then see how the inodes are no longer in sync.

[knacky34]

@emilylange, impressive !

1. Restart the container
2. Check with ls -i from both outside and inside container -> the inode numbers are equals
3. make a change to Caddyfile from outside of the container.
4. Check again with ls -i -> the inode numbers are different. And cat from inside of the container does not output latest file

I've googled a bit, I managed to fix the issue using the solution in this Stackoverflow's post

1. Move Caddyfile to a new subdirectory on host
2. Recreate container with a bind mount from path/to/subdir/containing/Caddyfile to /etc/caddy
3. Now you should be able to modify Caddyfile while container is running

As the issue is related to the docker/podman itself, you may close this issue. You may also update the Graceful reloads paragraph on Docker Hub according to the fix I written above.

[Guear]

Hello team,

I just encountered this bug and thanks to @knacky34, I was able to resolve this issue. Could this be updated in the the Graceful reloads paragraph on Docker Hub ?
Thank you

[ver4a]

I know this was closed, but the proper solution/explanation/reasoning(?) is not here yet, i think.

This behavior isn't actually a bug in docker or podman, it's a consequence of backupcopy=auto in vim (default almost everywhere), the "edited" file is actually a new file that's been just renamed to the original file's name, the bind mount is still (correctly) pointing to the old file's inode. If you want to actually change the file in place, you have to use "backupcopy=yes", this can be set in vim with ":set backupcopy=yes" in vim or in .vimrc with "set backupcopy=yes". If you use a different editor and see this behavior, it's the same case, just find out how to reconfigure your editor.

Bind mounting a directory is a workaround, it doesn't address the cause.

TL;DR: use ":set backupcopy=yes" in vim or "set backupcopy" in .vimrc

[purple-emily]

@TheSiman I'm having this issue and use vscode to control my server. I ssh in using the remote ssh plugin for vscode.

So I have confirmed that the inode is different on the host and inside my container. You're saying that what vscode is doing is creating a new file in place of the old one? I don't know which setting will fix this, any ideas?

[BjoernLudwigPTB]

You may also update the Graceful reloads paragraph on Docker Hub according to the fix I written above.

I just opened a PR to finally update the docs.

[ver4a]

@purple-emily Yes, that's what I'm saying. I'm not familiar with the plugin unfortunately.

@BjoernLudwigPTB Some wires got crossed when linking the PR, here's the correct link: docker-library/docs#2483 (same thing with this issue's link in the PR comment)

[francislavoie]

@BjoernLudwigPTB I don't agree with that change to the docs. The problem is with the text editor you're using, not with the Docker setup.

[BjoernLudwigPTB]

@BjoernLudwigPTB I don't agree with that change to the docs. The problem is with the text editor you're using, not with the Docker setup.

Thanks for considering my contribution! As mentioned in the PR I will come up with a better wording to reflect the actual scope of the problem addressed shortly.