Replica stuck at startup because database file is locked

[penberg]

We hit the following problem with a replica being stuck because database file is locked:

[info] _ _
[info] ___ __ _| | __| |
[info] / __|/ _` | |/ _` |
[info] \__ \ (_| | | (_| |
[info] |___/\__, |_|\__,_|
[info] |_|
[info] Welcome to sqld!
[info] version: 0.17.1
[info] commit SHA: 15edf5e5c902711c1ef8aa0e0cdb81efb6d18ed4
[info] build date: 2023-07-26
[info] This software is in BETA version.
[info] If you encounter any bug, please open an issue at https://github.com/libsql/sqld/issues
[info] config:
[info] - mode: replica (primary at [REDACTED])
[info] - database path: /data/iku.db
[info] - extensions path: <disabled>
[info] - listening for HTTP requests on: [::]:8080
[info] - grpc_tls: no
[info] 2023-08-20T06:42:52.828441Z INFO sqld::replication::replica::replicator: Attempting to perform handshake with primary.
[info] 2023-08-20T06:42:52.831402Z INFO sqld: Using JWT-based authentication
[info] 2023-08-20T06:42:52.832518Z INFO sqld: Server sending heartbeat to URL [REDACTED]
[info] 2023-08-20T06:42:52.832770Z INFO sqld::http: listening for HTTP requests on [::]:8080
[info] 2023-08-20T06:42:52.834973Z INFO sqld::hrana::ws: Listening for Hrana connections on [::]:2023
[info] 2023-08-20T06:42:52.838643Z WARN sqld: failed to open connection for storager monitor: Error code 5: The database file is locked, trying again in 60s