Error with renewing / obtaining SSL Certificates when using 3rd party storage

[n3o77]

I'm using https://caddyserver.com/docs/modules/caddy.storage.mysql as storage with caddy v2.8.4 and when renewing or trying to obtain a new certificate because of an previous error i'm getting this error:

ERR ts=1733056373.1711109 logger=tls.obtain msg=will retry error=[#DOMAIN#] Obtain: registering account [mailto:default.json] with server: attempt 1: https://acme-staging-v02.api.letsencrypt.org/acme/new-acct: HTTP 400 urn:ietf:params:acme:error:invalidContact - Error creating new account :: "default.json" is not a valid e-mail address attempt=2 retrying_in=120 elapsed=62.43641957 max_duration=2592000

There's another bug post in the community with the postgres storage which seems to have the same issue and with more details: https://caddy.community/t/invalid-email-default-json-when-retrieving-cert-from-lets-encrypt/17451

When deleting the entries for key acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json and acme/acme-v02.api.letsencrypt.org-directory/users/default/default.key in the database everything works fine again. But as they're always created again this is not a longterm solution.

Any help is very much appreciated, thank you in advance.

[mohammed90]

Thank you for the report! Can you provide more details?

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍 I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

[n3o77]

Thank you @mohammed90

1. Environment

1a. Operating system and version

I have the problem on Debian Bookworm and In the Official Docker Image caddy:builder

Dockerfile:

FROM caddy:builder AS builder

RUN xcaddy build \
    --with github.com/zhangjiayin/caddy-mysql-storage

FROM caddy:builder-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

CMD ["caddy", "docker-proxy"]

1b. Caddy version (run caddy version or paste commit SHA)

v2.8.4

2. Description

2a. What happens (briefly explain what is wrong)

SSL Certificates can't be obtained / renewed when the keys acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json and acme/acme-v02.api.letsencrypt.org-directory/users/default/default.key are already present.

2c. Log output

ERR ts=1733056373.1711109 logger=tls.obtain msg=will retry error=[#DOMAIN#] Obtain: registering account [mailto:default.json] with server: attempt 1: https://acme-staging-v02.api.letsencrypt.org/acme/new-acct: HTTP 400 urn:ietf:params:acme:error:invalidContact - Error creating new account :: "default.json" is not a valid e-mail address attempt=2 retrying_in=120 elapsed=62.43641957 max_duration=2592000

2d. Workaround(s)

Delete entries with key acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json and acme/acme-v02.api.letsencrypt.org-directory/users/default/default.key

3. Tutorial (minimal steps to reproduce the bug)

1. Install Docker Image
2. Configure MySQL Storage in Caddyfile

{
    storage mysql {
        dsn #USER#:#PWD#W@tcp(#MYSQL_UP#:#MYSQL_PORT#)/#DATABASE#?charset=utf8mb4
    }
}

3. Add Host to Caddyfile
4. Retrieve Certificates
5. Make sure the certificates are in the Database
6. Wait for renewal (did not find anything how to force renewal)

Let me know if you need anything else.

[mohammed90]

Please share the full config

[n3o77]

{
    storage mysql {
        dsn #USER#:#PWD#W@tcp(#MYSQL_UP#:#MYSQL_PORT#)/#DATABASE#?charset=utf8mb4
    }
}

#DOMAIN# {
    root * /srv/www/domain/public

    encode zstd gzip
    header -server
    file_server

    php_fastcgi unix//var/run/php/php8.3-fpm.sock {
        resolve_root_symlink
    }

    @phpFile {
        path *.php*
    }

    error @phpFile "Not found" 404

    @dotFiles {
      path */.*
      not path /.well-known/*
    }

    error @dotFiles "Not found" 404
}

[mholt]

Sounds like the storage module is returning the key instead of the value or something like that. Might be an issue to take upstream.

[n3o77]

I did here, just for reference: zhangjiayin/caddy-mysql-storage#3

I'm not really familiar with go, so not sure if i'm getting this right, but looking at the storage implementation there doesn't seem to be anything weird with that. In the community post with the same issue this also seems to be a problem with a postgres implementation.

Also to quote the author of the community post:

The default.json invalid email that Let’s Encrypt complains about is likely coming from the cert key (id # 11 in the postgres screenshot). Based off of certmagic’s code, it seems like this is… correct? It seems like it is using the last part of the key as the email address, and for some reason mine is default.json. What am I missing?

But my go understanding is not good enough to verify this.

If you can give me some instructions on how to debug this further i'm happy to do so.

[mholt]

What I generally do is put some print statements in as an easy way to see where values are coming from and what the variables are. Are you up for adding some print statements in various places and compiling and running? (I appreciate that you filled out the help template above; just very busy rn!)

[n3o77]

No worries, i appreciate the help. If you could give me some rough instructions what's of interest in the logs and best practice in how to compile with the changes etc. would be very helpful.

[mholt]

Hmm, well, I didn't author the mysql storage module, so I'm not super familiar with its code. But in CertMagic you could try putting some logs in this function for example:

https://github.com/caddyserver/certmagic/blob/3fcd710c0cfc6d80026011c8ef9b0d7e94860b2b/account.go#L371

log.Printf("Account list:", accountList) for example.