Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]kbcli cluster connect cluster failed #5890

Closed
ahjing99 opened this issue Nov 22, 2023 · 2 comments · Fixed by apecloud/kbcli#65
Closed

[BUG]kbcli cluster connect cluster failed #5890

ahjing99 opened this issue Nov 22, 2023 · 2 comments · Fixed by apecloud/kbcli#65
Assignees
@xuriwuyun @Y-Rookie @shanshanying
Labels
bug kind/bug Something isn't working severity/critical Blocking or critical issues
Milestone
Release 0.8.0

Comments

@ahjing99
Copy link
Collaborator

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.8.0-alpha.8
kbcli: 0.8.0-alpha.8

➜  ~ kbcli cluster connect  postgresql -n kubechat
Connect to instance postgresql-postgresql-0: out of postgresql-postgresql-0, postgresql-postgresql-1
psql (12.17 (Ubuntu 12.17-1.pgdg22.04+1))
Type "help" for help.

postgres=# create database kubechat;
ERROR:  cannot execute CREATE DATABASE in a read-only transaction
postgres=# exit

➜  ~ kbcli cluster describe postgresql -n kubechat
Name: postgresql	 Created Time: Nov 22,2023 10:26 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION              STATUS    TERMINATION-POLICY
kubechat    postgresql           postgresql-12.14.1   Running   Delete

Endpoints:
COMPONENT    MODE        INTERNAL                                                EXTERNAL
postgresql   ReadWrite   postgresql-postgresql.kubechat.svc.cluster.local:5432   <none>
                         postgresql-postgresql.kubechat.svc.cluster.local:6432

Topology:
COMPONENT    INSTANCE                  ROLE        STATUS    AZ              NODE                                               CREATED-TIME
postgresql   postgresql-postgresql-0   secondary   Running   us-central1-c   gke-yjtest-default-pool-4b662815-k3k3/10.128.0.5   Nov 22,2023 10:26 UTC+0800
postgresql   postgresql-postgresql-1   primary     Running   us-central1-c   gke-yjtest-default-pool-4b662815-k3k3/10.128.0.5   Nov 22,2023 10:26 UTC+0800

Resources Allocation:
COMPONENT    DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
postgresql   false       500m / 500m          512Mi / 512Mi           data:20Gi      kb-default-sc

Images:
COMPONENT    TYPE         IMAGE
postgresql   postgresql   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/spilo:12.14.1-pgvector-v0.5.0

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n kubechat postgresql


➜  ~ k get cluster postgresql -n kubechat -o yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  creationTimestamp: "2023-11-22T02:26:02Z"
  finalizers:
  - cluster.kubeblocks.io/finalizer
  generation: 1
  labels:
    app.kubernetes.io/instance: postgresql
    app.kubernetes.io/version: 14.8.0
    clusterdefinition.kubeblocks.io/name: postgresql
    clusterversion.kubeblocks.io/name: postgresql-12.14.1
    helm.sh/chart: postgresql-cluster-0.9.0-alpha.0
  name: postgresql
  namespace: kubechat
  resourceVersion: "9787"
  uid: 9ec5f0cd-86c7-4b64-b5f2-b92ebd049c60
spec:
  affinity:
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys:
    - kubernetes.io/hostname
  clusterDefinitionRef: postgresql
  clusterVersionRef: postgresql-12.14.1
  componentSpecs:
  - componentDefRef: postgresql
    enabledLogs:
    - running
    monitor: false
    name: postgresql
    noCreatePDB: false
    replicas: 2
    resources:
      limits:
        cpu: "0.5"
        memory: 0.5Gi
      requests:
        cpu: "0.5"
        memory: 0.5Gi
    switchPolicy:
      type: Noop
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
  terminationPolicy: Delete
status:
  clusterDefGeneration: 2
  components:
    postgresql:
      phase: Running
      podsReady: true
      podsReadyTime: "2023-11-22T02:28:13Z"
  conditions:
  - lastTransitionTime: "2023-11-22T02:26:02Z"
    message: 'The operator has started the provisioning of Cluster: postgresql'
    observedGeneration: 1
    reason: PreCheckSucceed
    status: "True"
    type: ProvisioningStarted
  - lastTransitionTime: "2023-11-22T02:26:02Z"
    message: Successfully applied for resources
    observedGeneration: 1
    reason: ApplyResourcesSucceed
    status: "True"
    type: ApplyResources
  - lastTransitionTime: "2023-11-22T02:28:13Z"
    message: all pods of components are ready, waiting for the probe detection successful
    reason: AllReplicasReady
    status: "True"
    type: ReplicasReady
  - lastTransitionTime: "2023-11-22T02:28:13Z"
    message: 'Cluster: postgresql is ready, current phase is Running'
    reason: ClusterReady
    status: "True"
    type: Ready
  observedGeneration: 1
  phase: Running
@ahjing99 ahjing99 added the kind/bug Something isn't working label Nov 22, 2023
@ahjing99 ahjing99 added this to the Release 0.8.0 milestone Nov 22, 2023
@ahjing99
Copy link
Collaborator Author

Mysql also has the same problem

➜  ~ kbcli cluster describe mysql-kqbtpo
Name: mysql-kqbtpo	 Created Time: Nov 22,2023 11:56 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                            EXTERNAL
mysql       ReadWrite   mysql-kqbtpo-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE               ROLE       STATUS    AZ              NODE                                               CREATED-TIME
mysql       mysql-kqbtpo-mysql-0   follower   Running   us-central1-c   gke-yjtest-default-pool-4b662815-k3k3/10.128.0.5   Nov 22,2023 11:56 UTC+0800
mysql       mysql-kqbtpo-mysql-1   leader     Running   us-central1-c   gke-yjtest-default-pool-4b662815-k3k3/10.128.0.5   Nov 22,2023 11:56 UTC+0800
mysql       mysql-kqbtpo-mysql-2   follower   Running   us-central1-c   gke-yjtest-default-pool-4b662815-k3k3/10.128.0.5   Nov 22,2023 11:56 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:1Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default mysql-kqbtpo

➜  ~ kbcli cluster connect mysql-kqbtpo
Connect to instance mysql-kqbtpo-mysql-0: out of mysql-kqbtpo-mysql-0, mysql-kqbtpo-mysql-1, mysql-kqbtpo-mysql-2
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 8.0.30 WeSQL Server - GPL, Release 5, Revision 4ca1eb8

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database test;
ERROR 7504 (HY000): The consensus follower is not allowed to to do current operation.

@ahjing99 ahjing99 changed the title [BUG]kbcli cluster connect postgresql secondary pod [BUG]kbcli cluster connect postgresql/mysql secondary pod Nov 22, 2023
@ahjing99 ahjing99 changed the title [BUG]kbcli cluster connect postgresql/mysql secondary pod [BUG]kbcli cluster connect cluster failed Nov 22, 2023
@ahjing99
Copy link
Collaborator Author

Seems many clusters have the same problem

➜  ~ kbcli cluster connect mongo-kmcrcs
Connect to instance mongo-kmcrcs-mongodb-0: out of mongo-kmcrcs-mongodb-0, mongo-kmcrcs-mongodb-1, mongo-kmcrcs-mongodb-2
MongoDB shell version v4.4.24
connecting to: mongodb://%24KB_POD_FQDN:27017/admin?compressors=disabled&gssapiServiceName=mongodb&replicaSet=%24KB_CLUSTER_COMP_NAME
{"t":{"$date":"2023-11-22T04:12:24.202Z"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"$KB_CLUSTER_COMP_NAME","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set $KB_CLUSTER_COMP_NAME"}}
Error: connect failed to replica set $KB_CLUSTER_COMP_NAME/$KB_POD_FQDN:27017 :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
command terminated with exit code 1

@ahjing99 ahjing99 added the severity/critical Blocking or critical issues label Nov 22, 2023
@xuriwuyun xuriwuyun linked a pull request Nov 22, 2023 that will close this issue
chore: cluster connect to replica whose pod role label is set as primary apecloud/kbcli#65
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

@Y-Rookie Y-Rookie

@shanshanying shanshanying

Labels
bug kind/bug Something isn't working severity/critical Blocking or critical issues
Projects
None yet
Milestone
Release 0.8.0
Development

Successfully merging a pull request may close this issue.

chore: cluster connect to replica whose pod role label is set as primary apecloud/kbcli
5 participants
@xuriwuyun @Y-Rookie @ahjing99 @shanshanying @apecloud-bot