CKB Process Occasionally Fails to Terminate

[sunchengzhu]

Bug Report

Current Behavior

I restart CKB every two hours and I've noticed that there are occasional issues with missing metrics data.

Terminate command:

killckb() {
    PIDS=$(sudo lsof -ti:8114)
    for i in $PIDS; do
        echo "killed the ckb $i"
        sudo kill $i
    done
}

killckb

### Expected Behavior metrics data is not lost

Environment

• CKB version: ckb 0.118.0-rc1 (63ae338 2024-08-19)
• Chain: mainnet
• Operating system: Ubuntu 20.04.6 LTS
• Arch: x64

Additional context/Screenshots

My ckb.toml configuration:

[metrics.exporter.prometheus]
target = { type = "prometheus", listen_address = "0.0.0.0:8100" }

# # Experimental: Monitor memory changes.
[memory_tracker]
# # Seconds between checking the process, 0 is disable, default is 0.
interval = 5

[15168316096]

This is an intermittent issue.

The situation you described indicates that other nodes running ckb v118 did not experience missing metrics during the same time period. If the data were not collected from the source, it would suggest that the ckb process was terminated. However, based on the ckb logs we reviewed earlier, it shows that the service restarted successfully and continued to sync normally.

Therefore, it is more likely that the issue occurred when Prometheus was scraping data from ckb.

[sunchengzhu]

From the run.log, it can be seen that the CKB process did not exit promptly:

1. Received the Ctrl-C signal at 15:20.
2. At 15:25, an attempt to start CKB resulted in an ERROR.
3. The CKB shutdown log was not seen until 15:26.

[sunchengzhu]

This issue has been verified and resolved in PR 4615.