tls: certificates auto renewal will become stuck if issuer is changed between config reloads

[WeidiDeng]

When config is reloaded with a changed acme issuer, certmagic will check for the existence of the certificate of created from the new issuer next time certificate should be issued. These certificates don't exist because we want to use them to be created in the first place. certmagic will try in vain for 30 days to renew these certificates.

Detailed explanation:

When caddy is starting, a global tls cache is created if needed

caddy/modules/caddytls/tls.go

Lines 158 to 164 in a1751ad

	certCacheMu.Lock()
	if certCache == nil {
	certCache = certmagic.NewCache(cacheOpts)
	} else {
	certCache.SetOptions(cacheOpts)
	}
	certCacheMu.Unlock()

it will be destroyed if tls is not used anymore

caddy/modules/caddytls/tls.go

Lines 424 to 430 in a1751ad

	} else {
	// no more TLS app running, so delete in-memory cert cache
	certCache.Stop()
	certCacheMu.Lock()
	certCache = nil
	certCacheMu.Unlock()
	}

TLS cache will start renewing certificates in the background

https://github.com/caddyserver/certmagic/blob/3fcd710c0cfc6d80026011c8ef9b0d7e94860b2b/cache.go#L127

Managed domains are updated through caddy configuration.

Eventually, renewal will be done here

https://github.com/caddyserver/certmagic/blob/3fcd710c0cfc6d80026011c8ef9b0d7e94860b2b/maintain.go#L235

TLS cache will try to renew the certificate using the latest issuer url, but first it will check the existence of the old certificate:

https://github.com/caddyserver/certmagic/blob/3fcd710c0cfc6d80026011c8ef9b0d7e94860b2b/config.go#L807-L812

It doesn't exist because the old certificate if from a different issuer and the path checked is from the latest issuer.

This will be retried here

https://github.com/caddyserver/certmagic/blob/3fcd710c0cfc6d80026011c8ef9b0d7e94860b2b/config.go#L982

There are at least two ways to fix this: to restart caddy or remove the active caddy configuration and reload it so that caddy will realize these certificates don't exist and should be created instead.

[mholt]

Thanks for the report. I know you did in Slack, but could you share your logs here too? For the record, so as I go to fix I can ensure that the proper code paths are recreated and I fix the right problem. :)

[WeidiDeng]

This is the screenshort shat shows the stuck job:

The log is as

Nov 12 16:37:54 linux caddy[455]: {"level":"warn","ts":1731400674.037309,"logger":"tls.cache.maintenance","msg":"error while checking if stored certificate is also expiring soon","identifiers":["example.com"],"error":"open /tmp/caddy/certificates/new-acme/example.com/example.com.key: no such file or directory"}
Nov 12 16:37:54 linux  caddy[455]: {"level":"info","ts":1731400674.0373702,"logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["example.com"],"remaining":1956125.962630629}
Nov 12 16:37:54 linux  caddy[455]: {"level":"info","ts":1731400674.0377727,"logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["example.com"],"remaining":1956125.962229608}
Nov 12 16:37:54 linux  caddy[455]: {"level":"info","ts":1731400674.1341906,"logger":"tls.renew","msg":"acquiring lock","identifier":"example.com"}
Nov 12 16:37:54 linux  caddy[455]: {"level":"info","ts":1731400674.1503463,"logger":"tls.renew","msg":"lock acquired","identifier":"example.com"}
Nov 12 16:37:54 linux  caddy[455]: {"level":"error","ts":1731400674.1505845,"logger":"tls.renew","msg":"will retry","error":"open /tmp/caddy/certificates/new-acme/example.com/example.com.key: no such file or directory","attempt":1,"retrying_in":60,"elapsed":0.000176754,"max_duration":2592000}
Nov 12 16:38:54 linux  caddy[455]: {"level":"error","ts":1731400734.152456,"logger":"tls.renew","msg":"will retry","error":"open /tmp/caddy/certificates/new-acme/example.com/example.com.key: no such file or directory","attempt":2,"retrying_in":120,"elapsed":60.002046754,"max_duration":2592000}
Nov 12 16:40:54 linux  caddy[455]: {"level":"error","ts":1731400854.1529422,"logger":"tls.renew","msg":"will retry","error":"open /tmp/caddy/certificates/new-acme/example.com/example.com.key: no such file or directory","attempt":3,"retrying_in":120,"elapsed":180.002532897,"max_duration":2592000}

The log about certificate expires soon; queuing for renewal and no file or directory appears multiple times later and is omitted.

The cached certificate is from /tmp/caddy/certificates/old-acme/example.com/example.com.key, and I reverted acme url and deleted the config and reload to fix it.