Skip to content
This repository has been archived by the owner on Jan 18, 2024. It is now read-only.

Cluster node restarted unexpectedly #626

Open
boskiv opened this issue Oct 20, 2023 · 2 comments
Open

Cluster node restarted unexpectedly #626

boskiv opened this issue Oct 20, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@boskiv
Copy link

boskiv commented Oct 20, 2023 •
edited
Loading

What happened?

timescaledb-0 timescaledb 2023-10-20 06:02:33 UTC [51474]: [653217f9.c912-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-20 06:02:33 UTC [51474]: [653217f9.c912-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-20 06:02:35,714 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=4.999811467001564)")
timescaledb-0 timescaledb 2023-10-20 06:02:38,394 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=2.310818386664323)")
timescaledb-0 timescaledb 2023-10-20 06:02:40,713 ERROR: Request to server https://10.128.0.1:443 failed: ReadTimeoutError("HTTPSConnectionPool(host='10.128.0.1', port=443): Read timed out. (read timeout=1.919020875437127)")
timescaledb-0 timescaledb 2023-10-20 06:02:40,714 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-20 06:02:40,716 WARNING: Loop time exceeded, rescheduling immediately.
timescaledb-0 timescaledb 2023-10-20 06:02:40 UTC [93]: [65318ad1.5d-7] @,app= [00000] LOG:  received fast shutdown request
timescaledb-0 timescaledb 2023-10-20 06:02:40 UTC [93]: [65318ad1.5d-8] @,app= [00000] LOG:  aborting any active transactions

Did you expect to see something different?

How to reproduce it (as minimally and precisely as possible):

Environment

  • Which helm chart and what version are you using?
helm chart : 0.33.1
timescale/timescaledb-ha:pg15.4-ts2.11.2-all
  • What is in your values.yaml ?
  values:
    clusterName: "sf-cluster"
    image:
      tag: pg15.4-ts2.11.2-all
    persistentVolumes:
      data:
        size: 300Gi
      wal:
        size: 100Gi
    pgBouncer:
      enabled: true
      config:
        ignore_startup_parameters: extra_float_digits
        server_reset_query: DISCARD ALL
        max_client_conn: 5000
        default_pool_size: 40
        pool_mode: transaction
    patroni:
      bootstrap:
        dcs:
          postgresql:
            parameters:
              max_wal_size: 1GB
              min_wal_size: 512MB
              shared_preload_libraries: 'timescaledb'
              shared_buffers: 975091kB
              effective_cache_size: 2856MB
              maintenance_work_mem: 487545kB
              work_mem: 9750kB
              timescaledb.max_background_workers: 16
              max_worker_processes: 21
              max_parallel_workers_per_gather: 1
              max_parallel_workers: 2
              wal_buffers: 16MB
              default_statistics_target: 500
              random_page_cost: 1.1
              checkpoint_completion_target: 0.9
              max_connections: 500
              max_locks_per_transaction: 64
              autovacuum_max_workers: 10
              autovacuum_naptime: 10
              effective_io_concurrency: 256
              temp_file_limit: -1
    prometheus:
      enabled: true
    podMonitor:
      enabled: true
    tolerations:
      - key: "node.k8s/role"
        operator: "Equal"
        value: "timescaledb"
        effect: "NoSchedule"
  • Kubernetes version information:
➜  .kube kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.5
  • Kubernetes cluster kind:

linode managed kubernetes 0.27 version

Anything else we need to know?:

Nothing criminal in grafana at this time

Screenshot 2023-10-20 at 09 56 58
Screenshot 2023-10-20 at 09 56 56
Screenshot 2023-10-20 at 09 56 50
@boskiv boskiv added the bug Something isn't working label Oct 20, 2023
@boskiv
Copy link
Author

boskiv commented Oct 21, 2023 

timescaledb-0 timescaledb 2023-10-21 05:08:41 UTC [59036]: [65335cd9.e69c-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-21 05:08:41 UTC [59036]: [65335cd9.e69c-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-2] @,app= [00000] LOG:  replication terminated by primary server
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-3] @,app= [00000] DETAIL:  End of WAL reached on timeline 8 at F/450000A0.
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [52238]: [6532ed1b.cc0e-4] @,app= [08006] FATAL:  could not send end-of-streaming message to primary: SSL connection has been closed unexpectedly
timescaledb-0 timescaledb 	no COPY in progress
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [46941]: [65329ac8.b75d-19] @,app= [00000] LOG:  invalid resource manager ID 109 at F/450000A0
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [59041]: [65335cdd.e6a1-1] @,app= [08006] FATAL:  could not connect to the primary server: connection to server at "10.2.3.24", port 5432 failed: server closed the connection unexpectedly
timescaledb-0 timescaledb 		This probably means the server terminated abnormally
timescaledb-0 timescaledb 		before or while processing the request.
timescaledb-0 timescaledb 2023-10-21 05:08:45 UTC [46941]: [65329ac8.b75d-20] @,app= [00000] LOG:  waiting for WAL to become available at F/450000B8
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=60972
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:08:46 UTC [59044]: [65335cde.e6a4-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=60972
timescaledb-0 timescaledb 2023-10-21 05:08:50 UTC [59047]: [65335ce2.e6a7-1] @,app= [00000] LOG:  started streaming WAL from primary at F/45000000 on timeline 8
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=32880
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:08:56 UTC [59054]: [65335ce8.e6ae-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=32880
timescaledb-0 timescaledb 2023-10-21 05:08:56,481 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:56,483 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:56,586 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:56,588 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:57,242 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:57,245 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:58,199 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:58,202 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:08:59,787 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:08:59,789 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:01,504 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:01,507 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:02,661 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:02,666 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:04,140 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:04,143 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:04,143 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=56338
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:05 UTC [59061]: [65335cf1.e6b5-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.007 user=standby database= host=127.0.0.1 port=56338
timescaledb-0 timescaledb 2023-10-21 05:09:05,955 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:05,958 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:06,801 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:06,804 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:07,327 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:07,330 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:08,324 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:08,327 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:09,450 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:09,453 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:10,557 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:10,560 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=[local]
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-2] postgres@postgres,app=[unknown] [00000] LOG:  connection authenticated: identity="postgres" method=peer (/var/lib/postgresql/data/pg_hba.conf:3)
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-3] postgres@postgres,app=[unknown] [00000] LOG:  connection authorized: user=postgres database=postgres application_name=pg_isready
timescaledb-0 timescaledb 2023-10-21 05:09:11 UTC [59068]: [65335cf7.e6bc-4] postgres@postgres,app=pg_isready [00000] LOG:  disconnection: session time: 0:00:00.002 user=postgres database=postgres host=[local]
timescaledb-0 timescaledb 2023-10-21 05:09:11,885 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:11,887 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:12,982 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:12,984 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:14,699 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:14,700 ERROR: Error communicating with DCS
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=57358
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59070]: [65335cfb.e6be-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=57358
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=57360
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:15 UTC [59077]: [65335cfb.e6c5-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=57360
timescaledb-0 timescaledb 2023-10-21 05:09:15,959 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:15,962 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:16,175 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:16,177 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:17,111 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:17,114 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:18,060 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:18,064 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:18,868 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:18,870 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:20,094 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)'))': /api/v1/namespaces/timescaledb/endpoints/sf-cluster
timescaledb-0 timescaledb 2023-10-21 05:09:20,097 ERROR: Request to server https://10.128.0.1:443 failed: MaxRetryError("HTTPSConnectionPool(host='10.128.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/timescaledb/endpoints/sf-cluster (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))")
timescaledb-0 timescaledb 2023-10-21 05:09:21,412 ERROR: ObjectCache.run ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
timescaledb-0 timescaledb 2023-10-21 05:09:21,413 ERROR: ObjectCache.run ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
timescaledb-0 timescaledb server signaled
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46937]: [65329ac8.b759-8] @,app= [00000] LOG:  received SIGHUP, reloading configuration files
timescaledb-0 timescaledb Sourcing /home/postgres/.pod_environment
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46937]: [65329ac8.b759-9] @,app= [00000] LOG:  parameter "primary_conninfo" changed to "user=standby passfile=/home/postgres/.pgpass.patroni host=10.2.5.23 port=5432 sslmode=prefer application_name=timescaledb-0 gssencmode=prefer channel_binding=prefer"
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46941]: [65329ac8.b75d-21] @,app= [00000] LOG:  WAL receiver process shutdown requested
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59047]: [65335ce2.e6a7-2] @,app= [57P01] FATAL:  terminating walreceiver process due to administrator command
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59146]: [65335d01.e70a-1] @,app= [00000] LOG:  fetching timeline history file for timeline 9 from primary server
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45530
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59146]: [65335d01.e70a-2] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59147]: [65335d01.e70b-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.007 user=standby database= host=127.0.0.1 port=45530
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [46941]: [65329ac8.b75d-22] @,app= [00000] LOG:  new target timeline is 9
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59158]: [65335d01.e716-1] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:21 UTC [59163]: [65335d01.e71b-1] @,app= [08P01] FATAL:  could not start WAL streaming: ERROR:  replication slot "timescaledb_0" does not exist
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [46941]: [65329ac8.b75d-23] @,app= [00000] LOG:  waiting for WAL to become available at F/450000B8
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45534
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59166]: [65335d02.e71e-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.006 user=standby database= host=127.0.0.1 port=45534
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45538
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-2] standby@[unknown],app=[unknown] [00000] LOG:  connection authenticated: identity="standby" method=md5 (/var/lib/postgresql/data/pg_hba.conf:8)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-3] standby@[unknown],app=[unknown] [00000] LOG:  replication connection authorized: user=standby SSL enabled (protocol=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384, bits=256)
timescaledb-0 timescaledb 2023-10-21 05:09:22 UTC [59167]: [65335d02.e71f-4] standby@[unknown],app=[unknown] [00000] LOG:  disconnection: session time: 0:00:00.005 user=standby database= host=127.0.0.1 port=45538
timescaledb-0 timescaledb 2023-10-21 05:09:24 UTC [59168]: [65335d04.e720-1] [unknown]@[unknown],app=[unknown] [00000] LOG:  connection received: host=127.0.0.1 port=45550

@boskiv
Copy link
Author

boskiv commented Oct 21, 2023

no pods restarted actually, but postgres inside

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@boskiv