Error opening metrics server listener error; bind: address already in use"

[Perkinje]

Behaviour

Metrics server starts when the container starts. I can visit the metrics endpoint and see the the data. So when a tunnel is started and the metrics server can't run because there is already a metrics server running on that port. The tunnel fails to start. Ducked the error without finding much for help.

Steps to reproduce this issue

This is my docker file.

version: "3.5"

services:
  cloudflared:
    image: crazymax/cloudflared:latest
    container_name: cloudflared

    volumes:
      - /mnt/dockers/config/cloudflared:/home/cloudflared
    ports:
      - target: 5053
        published: 5053
        protocol: udp
      - target: 49312
        published: 49312
        protocol: tcp
    environment:
      - "TZ=America/Los_Angeles"
      - "TUNNEL_DNS_UPSTREAM=https://u6ko3ti7wp.cloudflare-gateway.com/dns-query"
      - "TUNNEL_ORIGIN_CERT=/home/cloudflared/.cloudflared/cert.pem"
    restart: always 

The docker starts without issues. Going to localhost:49312/metrics shows

# HELP build_info Build and version information
# TYPE build_info gauge
build_info{goversion="go1.17.7",revision="",type="",version="2022.2.1"} 1
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# TYPE cloudflared_tunnel_active_streams gauge
cloudflared_tunnel_active_streams 0
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# TYPE cloudflared_tunnel_concurrent_requests_per_tunnel gauge
cloudflared_tunnel_concurrent_requests_per_tunnel 0
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# TYPE cloudflared_tunnel_ha_connections gauge
cloudflared_tunnel_ha_connections 0
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# TYPE cloudflared_tunnel_request_errors counter 
[Redacted for brevity]

Within the docker shell I start the tunnel using...

cloudflared tunnel --config /home/cloudflared/.cloudflared/config.yml run cloud

and it fails due to the metrics address already in use.

2022-02-21T06:02:26Z INF Starting tunnel tunnelID=[REDACTED]
2022-02-21T06:02:26Z INF Version 2022.2.1
2022-02-21T06:02:26Z INF GOOS: linux, GOVersion: go1.17.7, GoArch: amd64
2022-02-21T06:02:26Z INF Settings: map[config:/home/cloudflared/.cloudflared/config.yml cred-file:/home/cloudflared/.cloudflared/[REDACTED].json credentials-file:/home/cloudflared/.cloudflared/REDACTED.json]
2022-02-21T06:02:26Z INF Environmental variables map[TUNNEL_DNS_ADDRESS:0.0.0.0 TUNNEL_DNS_PORT:5053 TUNNEL_DNS_UPSTREAM:https://u6ko3ti7wp.cloudflare-gateway.com/dns-query TUNNEL_METRICS:0.0.0.0:49312 TUNNEL_ORIGIN_CERT:/home/cloudflared/.cloudflared/cert.pem]
2022-02-21T06:02:26Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/run-as-service
2022-02-21T06:02:26Z INF Generated Connector ID: [REDACTED]
2022-02-21T06:02:26Z INF Initial protocol http2
2022-02-21T06:02:26Z ERR Error opening metrics server listener error="listen tcp 0.0.0.0:49312: bind: address already in use"
Error opening metrics server listener: listen tcp 0.0.0.0:49312: bind: address already in use

Expected behaviour

The metrics server should only need to start once. Either when the docker comes up or when the tunnel is created, but not at both times.

Configuration

• Docker version (type docker --version) : 20.10.11, build dea9396
• Docker compose version if applicable (type docker-compose --version) : 1.25.0, build unknown
• Platform (Debian 9, Ubuntu 18.04, ...) : TrueNAS Scale 5.10.81
• System info (type uname -a) : Linux truenas-scale.local 5.10.81+truenas x86_64 GNU/Linux
• Include all necessary configuration files : docker-compose.yml, .env, ...

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 22
  Running: 22
  Paused: 0
  Stopped: 0
 Images: 26
 Server Version: 20.10.11
 Storage Driver: zfs
  Zpool: dockers
  Zpool Health: ONLINE
  Parent Dataset: dockers
  Space Used By Parent: 77581218624
  Space Available: 3212133043392
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: [REDACTED]
  Is Manager: true
  ClusterID: [REDACTED]
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: [REDACTED]
  Manager Addresses:
   [REDACTED]
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.10.81+truenas
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 64
 Total Memory: 62.84GiB
 Name: truenas-scale.local
 ID: [REDACTED]
 Docker Root Dir: /mnt/dockers/home
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false