From 9e560154861ca349a201798fe37bdfbda7b50edf Mon Sep 17 00:00:00 2001 From: yuanyuan zhang <111744220+michelle-0808@users.noreply.github.com> Date: Thu, 30 Mar 2023 15:59:14 +0800 Subject: [PATCH 01/80] docs: fix docs bugs (#2325) --- README.md | 4 +- docs/user_docs/introduction/introduction.md | 27 +- .../cluster-management/expand-volume.md | 22 +- .../migration/migration.md | 8 +- .../cluster-management/expand-volume.md | 26 +- .../restart-postgresql-cluster.md | 5 +- .../scale-for-postgresql.md | 6 +- .../start-stop-a-cluster.md | 12 +- .../configuration/configuration.md | 284 ------------------ 9 files changed, 52 insertions(+), 342 deletions(-) delete mode 100644 docs/user_docs/kubeblocks-for-postgresql/configuration/configuration.md diff --git a/README.md b/README.md index 998cce997..a1302bb0a 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,9 @@ ## What is KubeBlocks -KubeBlocks is an open-source tool designed to help developers and platform engineers build and manage stateful workloads, such as databases and analytics, on Kubernetes. It is cloud-neutral and supports multiple public cloud providers. KubeBlocks provides a unified and declarative approach to increace productivity in DevOps practices. +KubeBlocks is an open-source tool designed to help developers and platform engineers build and manage stateful workloads, such as databases and analytics, on Kubernetes. It is cloud-neutral and supports multiple public cloud providers, providing a unified and declarative approach to increase productivity in DevOps practices. -The name KubeBlocks is derived from Kubernetes and building blocks, which indicates that standardizing databases and analytics on Kubernetes can be both enjoyable and productive, like playing with construction toys. KubeBlocks combines the large-scale production experiences of top public cloud providers with enhanced usability and stability. +The name KubeBlocks is derived from Kubernetes and building blocks, which indicates that standardizing databases and analytics on Kubernetes can be both productive and enjoyable, like playing with construction toys. KubeBlocks combines the large-scale production experiences of top public cloud providers with enhanced usability and stability. ### Why you need KubeBlocks diff --git a/docs/user_docs/introduction/introduction.md b/docs/user_docs/introduction/introduction.md index a77929e00..fea3e2934 100644 --- a/docs/user_docs/introduction/introduction.md +++ b/docs/user_docs/introduction/introduction.md @@ -7,9 +7,9 @@ sidebar_position: 1 # KubeBlocks overview ## Introduction -KubeBlocks is an open-source tool designed to help developers and platform engineers build and manage stateful workloads, such as databases and analytics, on Kubernetes. It is cloud-neutral and supports multiple public cloud providers. KubeBlocks provides a unified and declarative approach to increace productivity in DevOps practices. +KubeBlocks is an open-source tool designed to help developers and platform engineers build and manage stateful workloads, such as databases and analytics, on Kubernetes. It is cloud-neutral and supports multiple public cloud providers, providing a unified and declarative approach to increase productivity in DevOps practices. -The name KubeBlocks is derived from Kubernetes and building blocks, which indicates that standardizing databases and analytics on Kubernetes can be both enjoyable and productive, like playing with construction toys. KubeBlocks combines the large-scale production experiences of top public cloud providers with enhanced usability and stability. +The name KubeBlocks is derived from Kubernetes and building blocks, which indicates that standardizing databases and analytics on Kubernetes can be both productive and enjoyable, like playing with construction toys. KubeBlocks combines the large-scale production experiences of top public cloud providers with enhanced usability and stability. ## Why You Need KubeBlocks @@ -21,18 +21,11 @@ To address these challenges, and solve the problem of complexity, KubeBlocks int - Latency-based election weight reduces the possibility of related workloads or components being located in different available zones. - Maintains the status of data replication and automatically repairs replication errors or delays. -## Feature list - -* Kubernetes native and multi-cloud supported -* Provisioning a database cluster within several minutes without the knowledge of Kubernetes -* Supports multiple database engines including MySQL, PostgreSQL, Redis and Apache Kafka -* Provides high-availability database clusters with single-availability zone deployment, double-availability zone deployment, or three-availability zone deployment. -* Automatic operation and maintenance, including vertical scaling, horizontal scaling, volume expansion and restarting clusters -* Snapshot backup at minute-level through EBS -* Scheduled backup -* Point in time restore -* Built-in Prometheus, Grafana, and AlertManager -* Resource overcommitment to allocate more database instances on one EC2 to decrease costs efficiently -* Role-based access control (RBAC) -* `kbcli`, an easy-to-use CLI, supports common operations such as database cluster creating, connecting, backup, restore, monitoring and trouble shooting. -* Supports custom robot alerts for Slack, Feishu, Wechat and DingTalks. \ No newline at end of file +## KubeBlocks Key Features + +- Kubernetes-native and multi-cloud supported. +- Supports multiple database engines, including MySQL, PostgreSQL, Redis, MongoDB, and more. +- Provides production-level performance, resilience, scalability, and observability. +- Simplifies day-2 operations, such as upgrading, scaling, monitoring, backup, and restore. +- Declarative configuration is made simple, and imperative commands are made powerful. +- The learning curve is flat, and you are welcome to submit new issues on GitHub. \ No newline at end of file diff --git a/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md b/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md index 13c0392ba..337461593 100644 --- a/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md +++ b/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md @@ -1,6 +1,6 @@ --- title: Expand volume -description: How to expand the volume of a PostgreSQL cluster +description: How to expand the volume of a MySQL cluster sidebar_position: 3 sidebar_label: Expand volume --- @@ -26,15 +26,15 @@ kbcli cluster list ```bash kbcli cluster list mysql-cluster > -NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME -mysql-cluster default postgresql postgresql-14.7.0 Delete Running Mar 3,2023 10:29 UTC+0800 +NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME +mysql-cluster default apecloud-mysql ac-mysql-8.0.30 Delete Running Jan 29,2023 14:29 UTC+0800 ``` ## Option 1. Use kbcli Configure the values of `--component-names`, `--volume-claim-template-names`, and `--storage`, and run the command below to expand the volume. ```bash -kbcli cluster volume-expand pg-cluster --component-names="postgresql" \ +kbcli cluster volume-expand mysql-cluster --component-names="mysql" \ --volume-claim-template-names="data" --storage="2Gi" ``` @@ -52,10 +52,10 @@ kind: OpsRequest metadata: name: ops-volume-expansion spec: - clusterRef: pg-cluster + clusterRef: mysql-cluster type: VolumeExpansion volumeExpansion: - - componentName: postgresql + - componentName: mysql volumeClaimTemplates: - name: data storage: "2Gi" @@ -70,14 +70,14 @@ Change the value of `spec.components.volumeClaimTemplates.spec.resources` in the apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: - name: pg-cluster + name: mysql-cluster namespace: default spec: - clusterDefinitionRef: postgresql - clusterVersionRef: postgresql-14.7.0 + clusterDefinitionRef: apecloud-mysql + clusterVersionRef: ac-mysql-8.0.30 components: - - name: postgresql - type: postgresql + - name: mysql + type: mysql replicas: 1 volumeClaimTemplates: - name: data diff --git a/docs/user_docs/kubeblocks-for-mysql/migration/migration.md b/docs/user_docs/kubeblocks-for-mysql/migration/migration.md index c5021a61f..54d39201d 100644 --- a/docs/user_docs/kubeblocks-for-mysql/migration/migration.md +++ b/docs/user_docs/kubeblocks-for-mysql/migration/migration.md @@ -71,18 +71,18 @@ The Kubernetes ClusterIP of ApeCloud MySQL is exposed by default in the EKS envi alb.ingress.kubernetes.io/scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-subnets: ${subnet name1},${subnet name2} labels: - app.kubernetes.io/component-name: mysql + apps.kubeblocks.io/component-name: mysql app.kubernetes.io/instance: ${mysql clustername} app.kubernetes.io/managed-by: kubeblocks - app.kubernetes.io/name: state.mysql-apecloud-mysql + app.kubernetes.io/name: apecloud-mysql spec: externalTrafficPolicy: Cluster type: LoadBalancer selector: - app.kubernetes.io/component-name: mysql + apps.kubeblocks.io/component-name: mysql app.kubernetes.io/instance: ${mysql clustername} app.kubernetes.io/managed-by: kubeblocks - cs.dbaas.kubeblocks.io/role: leader + kubeblocks.io/role: leader ports: - name: http protocol: TCP diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md index 7f603621c..4948d4b66 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md @@ -1,6 +1,6 @@ --- title: Expand volume -description: How to expand the volume of a MySQL cluster +description: How to expand the volume of a PostgreSQL cluster sidebar_position: 3 sidebar_label: Expand volume --- @@ -24,17 +24,17 @@ kbcli cluster list ***Example*** ```bash -kbcli cluster list mysql-cluster +kbcli cluster list pg-cluster > -NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME -mysql-cluster default apecloud-mysql ac-mysql-8.0.30 Delete Running Jan 29,2023 14:29 UTC+0800 +NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME +pg-cluster default postgresql postgresql-14.7.0 Delete Running Mar 3,2023 10:29 UTC+0800 ``` ## Option 1. Use kbcli Configure the values of `--component-names`, `--volume-claim-template-names`, and `--storage`, and run the command below to expand the volume. ```bash -kbcli cluster volume-expand mysql-cluster --component-names="mysql" \ +kbcli cluster volume-expand pg-cluster --component-names="pg-replication" \ --volume-claim-template-names="data" --storage="2Gi" ``` @@ -52,10 +52,10 @@ kind: OpsRequest metadata: name: ops-volume-expansion spec: - clusterRef: mysql-cluster + clusterRef: pg-cluster type: VolumeExpansion volumeExpansion: - - componentName: mysql + - componentName: pg-replication volumeClaimTemplates: - name: data storage: "2Gi" @@ -70,14 +70,14 @@ Change the value of `spec.components.volumeClaimTemplates.spec.resources` in the apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: - name: mysql-cluster + name: pg-cluster namespace: default spec: - clusterDefinitionRef: apecloud-mysql - clusterVersionRef: ac-mysql-8.0.30 + clusterDefinitionRef: postgresql + clusterVersionRef: postgresql-14.7.0 components: - - name: mysql - type: mysql + - name: pg-replication + type: postgresql replicas: 1 volumeClaimTemplates: - name: data @@ -88,4 +88,4 @@ spec: requests: storage: 1Gi # Change the volume storage size. terminationPolicy: Halt -``` +``` \ No newline at end of file diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/restart-postgresql-cluster.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/restart-postgresql-cluster.md index 4e9cd1527..c39c26644 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/restart-postgresql-cluster.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/restart-postgresql-cluster.md @@ -5,6 +5,7 @@ sidebar_position: 4 sidebar_label: Restart --- + # Restart PostgreSQL cluster You can restart all pods of the cluster. When an exception occurs in a database, you can try to restart it. @@ -23,7 +24,7 @@ All pods restart in the order of learner -> follower -> leader and the leader ma Configure the values of `component-names` and `ttlSecondsAfterSucceed` and run the command below to restart a specified cluster. ```bash - kbcli cluster restart NAME --component-names="postgresql" \ + kbcli cluster restart NAME --component-names="pg-replication" \ --ttlSecondsAfterSucceed=30 ``` - `component-names` describes the component name that needs to be restarted. @@ -42,7 +43,7 @@ All pods restart in the order of learner -> follower -> leader and the leader ma clusterRef: pg-cluster type: Restart restart: - - componentName: postgresql + - componentName: pg-replication EOF ``` 2. Validate the restarting. diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md index 0ea0f144f..20f59e62e 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md @@ -45,7 +45,7 @@ pg-cluster default postgresql-cluster postgresql-14.7.0 Delete ```bash kbcli cluster vscale pg-cluster \ - --component-names="postgresql" \ + --component-names="pg-replication" \ --requests.memory="2Gi" --requests.cpu="1" \ --limits.memory="4Gi" --limits.cpu="2" ``` @@ -66,7 +66,7 @@ pg-cluster default postgresql-cluster postgresql-14.7.0 Delete clusterRef: pg-cluster type: VerticalScaling verticalScaling: - - componentName: postgresql + - componentName: pg-replication requests: memory: "2Gi" cpu: "1000m" @@ -92,7 +92,7 @@ pg-cluster default postgresql-cluster postgresql-14.7.0 Delete clusterDefinitionRef: postgresql-cluster clusterVersionRef: postgre-14.7.0 components: - - name: postgresql + - name: pg-replication type: postgresql replicas: 1 resources: # Change the values of resources. diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md index a98e69dcb..ed231a092 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md @@ -1,11 +1,11 @@ --- -title: Stop/Start a MySQL cluster -description: How to start/stop a MySQL cluster +title: Stop/Start a PostgreSQL cluster +description: How to start/stop a PostgreSQL cluster sidebar_position: 5 sidebar_label: Stop/Start --- -# Stop/Start MySQL Cluster +# Stop/Start PostgreSQL Cluster You can stop/start a cluster to save computing resources. When a cluster is stopped, the computing resources of this cluster are released, which means the pods of Kubernetes are released, but the storage resources are reserved. Start this cluster again if you want to restore the cluster resources from the original storage by snapshots. @@ -20,7 +20,7 @@ kbcli cluster stop ***Example*** ```bash -kbcli cluster stop mysql-cluster +kbcli cluster stop pg-cluster ``` ### Option 2. Create an OpsRequest @@ -52,7 +52,7 @@ spec: clusterVersionRef: postgresql-14.7.0 terminationPolicy: WipeOut components: - - name: postgresql + - name: pg-replication type: postgresql monitor: false replicas: 0 @@ -114,7 +114,7 @@ spec: clusterVersionRef: postgresql-14.7.0 terminationPolicy: WipeOut components: - - name: postgresql + - name: pg-replication type: postgresql monitor: false replicas: 1 diff --git a/docs/user_docs/kubeblocks-for-postgresql/configuration/configuration.md b/docs/user_docs/kubeblocks-for-postgresql/configuration/configuration.md deleted file mode 100644 index 70ea81629..000000000 --- a/docs/user_docs/kubeblocks-for-postgresql/configuration/configuration.md +++ /dev/null @@ -1,284 +0,0 @@ ---- -title: Configure cluster parameters -description: Configure cluster parameters -sidebar_position: 1 ---- - -# Configure cluster parameters - -The KubeBlocks configuration function provides a set of consistent default configuration generation strategies for all the databases running on KubeBlocks and also provides a unified parameter change interface to facilitate managing parameter reconfiguration, searching the parameter user guide, and validating parameter effectiveness. - -## Before you start - -1. Install KubeBlocks. For details, refer to [Install KubeBlocks](./../../installation/install-and-uninstall-kbcli-and-kubeblocks.md). -2. Create a MySQL standalone and wait until the cluster status is Running. - -## View the parameter information - -1. Run the command below to search for parameter information. - - ```bash - kbcli cluster explain-config mysql-cluster |head -n 20 - > - template meta: - ConfigSpec: mysql-consensusset-config ComponentName: mysql ClusterName: mysql-cluster - - Parameter Explain: - +----------------------------------------------------------+--------------------------------------------+--------+---------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | PARAMETER NAME | ALLOWED VALUES | SCOPE | DYNAMIC | TYPE | DESCRIPTION | - +---------------------------------------------------------- +--------------------------------------------+--------+---------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | activate_all_roles_on_login | "0","1","OFF", "ON" | Global | false | string | Automatically set all granted roles as active after the user has authenticated successfully. | - | allow-suspicious-udfs | "0","1","OFF","ON" | Global | false | string | Controls whether user-defined functions that have only an xxx symbol for the main function can be loaded | - | auto_generate_certs | "0","1","OFF","ON" | Global | false | string | Controls whether the server autogenerates SSL key and certificate files in the data directory, if they do not already exist. | - | auto_increment_increment | [1-65535] | Global | false | integer | Intended for use with master-to-master replication, and can be used to control the operation of AUTO_INCREMENT columns | - | auto_increment_offset | [1-65535] | Global | false | integer | Determines the starting point for the AUTO_INCREMENT column value | - | autocommit | "0","1","OFF","ON" | Global | false | string | Sets the autocommit mode | - | automatic_sp_privileges | "0","1","OFF","ON" | Global | false | string | When this variable has a value of 1 (the default), the server automatically grants the EXECUTE and ALTER ROUTINE privileges to the creator of a stored routine, if the user cannot already execute and alter or drop the routine. | - | avoid_temporal_upgrade | "0","1","OFF","ON" | Global | false | string | This variable controls whether ALTER TABLE implicitly upgrades temporal columns found to be in pre-5.6.4 format. | - | back_log | [1-65535] | Global | false | integer | The number of outstanding connection requests MySQL can have | - | basedir | | Global | false | string | The MySQL installation base directory. | - | big_tables | "0","1","OFF","ON" | Global | false | string | | - | bind_address | | Global | false | string | | - | binlog_cache_size | [4096-18446744073709548000] | Global | false | integer | The size of the cache to hold the SQL statements for the binary log during a transaction. - ``` - -2. View the user guide of a parameter. - ```bash - kbcli cluster explain-config mysql-cluster --param=innodb_buffer_pool_size - template meta: - ConfigSpec: mysql-consensusset-config ComponentName: mysql ClusterName: mysql-cluster - - Configure Constraint: - Parameter Name: innodb_buffer_pool_size - Allowed Values: [5242880-18446744073709552000] - Scope: Global - Dynamic: false - Type: integer - Description: The size in bytes of the memory buffer innodb uses to cache data and indexes of its tables - ``` - * Allowed Values: It defines the valid value of this parameter. - * Dynamic: The value of `Dynamic` in `Configure Constraint` defines how the parameter reconfiguration takes effect. As mentioned in [How KubeBlocks configuration works](#how-kubeblocks-configuration-works), there are two different reconfiguration strategies based on the effectiveness type of changed parameters, i.e. **dynamic** and **static**. - * When `Dynamic` is `true`, it means the effectiveness type of parameters is **dynamic** and you can follow the instructions in [Reconfigure dynamic parameters](#reconfigure-dynamic-parameters). - * When `Dynamic` is `false`, it means the effectiveness type of parameters is **static** and you can follow the instructions in [Reconfigure static parameters](#reconfigure-static-parameters). - * Description: It describes the parameter definition. - - -## Reconfigure dynamic parameters - -Here we take reconfiguring `max_connection` and `beb_buffer_pool_size` as an example. - -1. Run the command to view the current values of `max_connection` and `beb_buffer_pool_size`. - ```bash - kbcli cluster connect mysql-cluster - ``` - - ```bash - mysql> show variables like '%max_connections%'; - > - +-----------------+-------+ - | Variable_name | Value | - +-----------------+-------+ - | max_connections | 167 | - +-----------------+-------+ - 1 row in set (0.04 sec) - ``` - - ```bash - mysql> show variables like '%innodb_buffer_pool_size%'; - > - +-------------------------+-----------+ - | Variable_name | Value | - +-------------------------+-----------+ - | innodb_buffer_pool_size | 134217728 | - +-------------------------+-----------+ - 1 row in set (0.00 sec) - ``` - -2. Adjust the values of `max_connections` and `innodb_buffer_pool_size`. - ```bash - kbcli cluster configure rose15 --set=max_connections=600,innodb_buffer_pool_size=512M - ``` - - :::note - - Make sure the value you set is within the Allowed Values of this parameters. If you set a value that does not meet the value range, the system prompts an error. For example, - - ```bash - kbcli cluster configure mysql-cluster --set=max_connections=200,innodb_buffer_pool_size=2097152 - error: failed to validate updated config: [failed to cue template render configure: [mysqld.innodb_buffer_pool_size: invalid value 2097152 (out of bound >=5242880): - 343:34 - ] - ] - ``` - - ::: - -3. Search the status of the parameter reconfiguration. - `Status.Progress` shows the overall status of the parameter change and `Conditions` show the details. - - ```bash - kbcli cluster describe-ops mysql-cluster-reconfiguring-z2wvn - > - Spec: - Name: mysql-cluster-reconfiguring-z2wvn NameSpace: default Cluster: mysql-cluster Type: Reconfiguring - - Command: - kbcli cluster configure mysql-cluster --component-names=mysql --template-name=mysql-consensusset-config --config-file=my.cnf --set innodb_buffer_pool_size=512M --set max_connections=600 - - Status: - Start Time: Mar 13,2023 02:55 UTC+0800 - Completion Time: Mar 13,2023 02:55 UTC+0800 - Duration: 1s - Status: Succeed - Progress: -/- - - Conditions: - LAST-TRANSITION-TIME TYPE REASON STATUS MESSAGE - Mar 13,2023 02:55 UTC+0800 Progressing OpsRequestProgressingStarted True Start to process the OpsRequest: mysql-cluster-reconfiguring-z2wvn in Cluster: mysql-cluster - Mar 13,2023 02:55 UTC+0800 Validated ValidateOpsRequestPassed True OpsRequest: mysql-cluster-reconfiguring-z2wvn is validated - Mar 13,2023 02:55 UTC+0800 Reconfigure ReconfigureStarted True Start to reconfigure in Cluster: mysql-cluster, Component: mysql - Mar 13,2023 02:55 UTC+0800 ReconfigureMerged ReconfigureMerged True Reconfiguring in Cluster: mysql-cluster, Component: mysql, ConfigTpl: mysql-consensusset-config, info: updated: map[my.cnf:{"mysqld":{"innodb_buffer_pool_size":"512M","max_connections":"600"}}], added: map[], deleted:map[] - Mar 13,2023 02:55 UTC+0800 ReconfigureSucceed ReconfigureSucceed True Reconfiguring in Cluster: mysql-cluster, Component: mysql, ConfigTpl: mysql-consensusset-config, info: updated policy: , updated: map[my.cnf:{"mysqld":{"innodb_buffer_pool_size":"512M","max_connections":"600"}}], added: map[], deleted:map[] - Mar 13,2023 02:55 UTC+0800 Succeed OpsRequestProcessedSuccessfully True Successfully processed the OpsRequest: mysql-cluster-reconfiguring-z2wvn in Cluster: mysql-cluster - ``` - -4. Verify whether the parameters are modified. - The whole searching process has a 30-second delay since it takes some time for kubelete to synchronize changes to the volume of the pod. - - ```bash - kbcli cluster connect mysql-cluster - ``` - - ```bash - mysql> show variables like '%max_connections%'; - > - +-----------------+-------+ - | Variable_name | Value | - +-----------------+-------+ - | max_connections | 600 | - +-----------------+-------+ - 1 row in set (0.04 sec) - ``` - - ```bash - mysql> show variables like '%innodb_buffer_pool_size%'; - > - +-------------------------+-----------+ - | Variable_name | Value | - +-------------------------+-----------+ - | innodb_buffer_pool_size | 536870912 | - +-------------------------+-----------+ - 1 row in set (0.00 sec) - ``` - -## Reconfigure static parameters - -Static parameter reconfiguring requires restarting the pod. Here we take reconfiguring `ngram_token_size` as an example. - -1. Search the current value of `ngram_token_size` and the default value is 2. - ```bash - kbcli cluster explain-config mysql-cluster --param=ngram_token_size - > - template meta: - ConfigSpec: mysql-consensusset-config ComponentName: mysql ClusterName: mysql-cluster - - Configure Constraint: - Parameter Name: ngram_token_size - Allowed Values: [1-10] - Scope: Global - Dynamic: false - Type: integer - Description: Defines the n-gram token size for the n-gram full-text parser. - ``` - - ```bash - kbcli cluster connect mysql-cluster - ``` - - ```bash - mysql> show variables like '%ngram_token_size%'; - > - +------------------+-------+ - | Variable_name | Value | - +------------------+-------+ - | ngram_token_size | 2 | - +------------------+-------+ - 1 row in set (0.01 sec) - ``` - -2. Adjust the value of `ngram_token_size`. - ```bash - kbcli cluster configure mysql-cluster --set=ngram_token_size=6 - > - Will updated configure file meta: - TemplateName: mysql-consensusset-config ConfigureFile: my.cnf ComponentName: mysql ClusterName: mysql-cluster - OpsRequest mysql-cluster-reconfiguring-nrnpf created - ``` - - :::note - - Make sure the value you set is within the Allowed Values of this parameters. Otherwise, the configuration may fail. - - ::: - -3. Watch the progress of searching parameter reconfiguration and pay attention to the output of `Status.Progress` and `Status.Status`. - ```bash - # In progress - kbcli cluster describe-ops mysql-cluster-reconfiguring-nrnpf - > - Spec: - Name: mysql-cluster-reconfiguring-nrnpf NameSpace: default Cluster: mysql-cluster Type: Reconfiguring - - Command: - kbcli cluster configure mysql-cluster --component-names=mysql --template-name=mysql-consensusset-config --config-file=my.cnf --set ngram_token_size=6 - - Status: - Start Time: Mar 13,2023 03:37 UTC+0800 - Duration: 22s - Status: Running - Progress: 0/1 - OBJECT-KEY STATUS DURATION MESSAGE - ``` - - ```bash - # Parameter change is completed - kbcli cluster describe-ops mysql-cluster-reconfiguring-nrnpf - > - Spec: - Name: mysql-cluster-reconfiguring-nrnpf NameSpace: default Cluster: mysql-cluster Type: Reconfiguring - - Command: - kbcli cluster configure mysql-cluster --component-names=mysql --template-name=mysql-consensusset-config --config-file=my.cnf --set ngram_token_size=6 - - Status: - Start Time: Mar 13,2023 03:37 UTC+0800 - Completion Time: Mar 13,2023 03:37 UTC+0800 - Duration: 26s - Status: Succeed - Progress: 1/1 - OBJECT-KEY STATUS DURATION MESSAGE - ``` - -4. After the reconfiguration is completed, connect to the database and verify the changes. - ```bash - kbcli cluster connect mysql-cluster - - Copyright (c) 2000, 2022, Oracle and/or its affiliates. - - Oracle is a registered trademark of Oracle Corporation and/or its - affiliates. Other names may be trademarks of their respective - owners. - - Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. - ``` - - ```bash - mysql> show variables like '%ngram_token_size%'; - > - +------------------+-------+ - | Variable_name | Value | - +------------------+-------+ - | ngram_token_size | 6 | - +------------------+-------+ - 1 row in set (0.09 sec) - ``` \ No newline at end of file From 198f917cace8ec659b361290045a739bc516548a Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Thu, 30 Mar 2023 16:00:04 +0800 Subject: [PATCH 02/80] fix: fixed bump-chart-version Makefile job (#2326) --- Makefile | 15 ++++++++++--- deploy/apecloud-mysql-cluster/Chart.yaml | 17 +-------------- .../apecloud-mysql-scale-cluster/Chart.yaml | 6 +++++- deploy/apecloud-mysql-scale/Chart.yaml | 2 +- deploy/apecloud-mysql/Chart.yaml | 17 +-------------- deploy/clickhouse-cluster/Chart.yaml | 17 +-------------- deploy/clickhouse/Chart.yaml | 2 +- deploy/kafka-cluster/Chart.yaml | 17 +-------------- deploy/kafka/Chart.yaml | 7 +++++-- deploy/mongodb-cluster/Chart.yaml | 17 +-------------- deploy/mongodb/Chart.yaml | 17 +-------------- deploy/nyancat/Chart.yaml | 15 ------------- deploy/postgresql-cluster/Chart.yaml | 19 ++--------------- .../postgresql-patroni-ha-cluster/Chart.yaml | 19 ++--------------- deploy/postgresql-patroni-ha/Chart.yaml | 21 +++---------------- deploy/postgresql/Chart.yaml | 21 +++---------------- deploy/redis-cluster/Chart.yaml | 17 +-------------- deploy/redis/Chart.yaml | 19 ++--------------- 18 files changed, 43 insertions(+), 222 deletions(-) diff --git a/Makefile b/Makefile index c76a40138..96df96004 100644 --- a/Makefile +++ b/Makefile @@ -357,18 +357,26 @@ fix-license-header: ## Run license header fix. ##@ Helm Chart Tasks +bump-single-chart-appver.%: chart=$(word 2,$(subst ., ,$@)) +bump-single-chart-appver.%: +ifeq ($(GOOS), darwin) + sed -i '' "s/^appVersion:.*/appVersion: $(VERSION)/" deploy/$(chart)/Chart.yaml +else + sed -i "s/^appVersion:.*/appVersion: $(VERSION)/" deploy/$(chart)/Chart.yaml +endif + bump-single-chart-ver.%: chart=$(word 2,$(subst ., ,$@)) bump-single-chart-ver.%: ifeq ($(GOOS), darwin) sed -i '' "s/^version:.*/version: $(VERSION)/" deploy/$(chart)/Chart.yaml - sed -i '' "s/^appVersion:.*/appVersion: $(VERSION)/" deploy/$(chart)/Chart.yaml else sed -i "s/^version:.*/version: $(VERSION)/" deploy/$(chart)/Chart.yaml - sed -i "s/^appVersion:.*/appVersion: $(VERSION)/" deploy/$(chart)/Chart.yaml endif .PHONY: bump-chart-ver -bump-chart-ver: bump-single-chart-ver.helm \ +bump-chart-ver: \ + bump-single-chart-ver.helm \ + bump-single-chart-appver.helm \ bump-single-chart-ver.apecloud-mysql \ bump-single-chart-ver.apecloud-mysql-cluster \ bump-single-chart-ver.apecloud-mysql-scale \ @@ -380,6 +388,7 @@ bump-chart-ver: bump-single-chart-ver.helm \ bump-single-chart-ver.mongodb \ bump-single-chart-ver.mongodb-cluster \ bump-single-chart-ver.nyancat \ + bump-single-chart-appver.nyancat \ bump-single-chart-ver.postgresql \ bump-single-chart-ver.postgresql-cluster \ bump-single-chart-ver.postgresql-patroni-ha \ diff --git a/deploy/apecloud-mysql-cluster/Chart.yaml b/deploy/apecloud-mysql-cluster/Chart.yaml index 29b218044..561977003 100644 --- a/deploy/apecloud-mysql-cluster/Chart.yaml +++ b/deploy/apecloud-mysql-cluster/Chart.yaml @@ -2,23 +2,8 @@ apiVersion: v2 name: apecloud-mysql-cluster description: An ApeCloud MySQL Cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "8.0.30" diff --git a/deploy/apecloud-mysql-scale-cluster/Chart.yaml b/deploy/apecloud-mysql-scale-cluster/Chart.yaml index 19ec949d7..9e6fda3d4 100644 --- a/deploy/apecloud-mysql-scale-cluster/Chart.yaml +++ b/deploy/apecloud-mysql-scale-cluster/Chart.yaml @@ -6,4 +6,8 @@ type: application version: 0.5.0-alpha.0 -appVersion: 0.5.0-alpha.0 +# This is the version number of the ApeCloud MySQL being deployed, +# rather than the version number of ApeCloud MySQL-Scale itself. +# From a user's perspective, the version number of the frontend +# proxy of the MySQL being used is not relevant. +appVersion: "8.0.30" diff --git a/deploy/apecloud-mysql-scale/Chart.yaml b/deploy/apecloud-mysql-scale/Chart.yaml index c5f96b3f8..12cee23dd 100644 --- a/deploy/apecloud-mysql-scale/Chart.yaml +++ b/deploy/apecloud-mysql-scale/Chart.yaml @@ -11,7 +11,7 @@ version: 0.5.0-alpha.0 # rather than the version number of ApeCloud MySQL-Scale itself. # From a user's perspective, the version number of the frontend # proxy of the MySQL being used is not relevant. -appVersion: 0.5.0-alpha.0 +appVersion: "8.0.30" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/apecloud-mysql/Chart.yaml b/deploy/apecloud-mysql/Chart.yaml index b5bcf81b1..473f64748 100644 --- a/deploy/apecloud-mysql/Chart.yaml +++ b/deploy/apecloud-mysql/Chart.yaml @@ -7,26 +7,11 @@ description: ApeCloud MySQL is fully compatible with MySQL syntax and supports s environment since it can automatically perform a high-availability switch to maintain business continuity when container exceptions, server exceptions, or availability zone exceptions occur. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "8.0.30" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/clickhouse-cluster/Chart.yaml b/deploy/clickhouse-cluster/Chart.yaml index 084537def..a8abf1f6f 100644 --- a/deploy/clickhouse-cluster/Chart.yaml +++ b/deploy/clickhouse-cluster/Chart.yaml @@ -2,26 +2,11 @@ apiVersion: v2 name: clickhouse-cluster description: A ClickHouse cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: 22.9.4 home: https://clickhouse.com/ diff --git a/deploy/clickhouse/Chart.yaml b/deploy/clickhouse/Chart.yaml index 69214da81..f360e8b59 100644 --- a/deploy/clickhouse/Chart.yaml +++ b/deploy/clickhouse/Chart.yaml @@ -11,7 +11,7 @@ type: application version: 0.5.0-alpha.0 -appVersion: 0.5.0-alpha.0 +appVersion: 22.9.4 home: https://clickhouse.com/ icon: https://bitnami.com/assets/stacks/clickhouse/img/clickhouse-stack-220x234.png diff --git a/deploy/kafka-cluster/Chart.yaml b/deploy/kafka-cluster/Chart.yaml index 20985923f..b6cb8f75a 100644 --- a/deploy/kafka-cluster/Chart.yaml +++ b/deploy/kafka-cluster/Chart.yaml @@ -2,26 +2,11 @@ apiVersion: v2 name: kafka-cluster description: A Kafka server cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: 3.4.0 home: https://kubeblocks.io/ diff --git a/deploy/kafka/Chart.yaml b/deploy/kafka/Chart.yaml index 6e8f87cb1..636f75a4c 100644 --- a/deploy/kafka/Chart.yaml +++ b/deploy/kafka/Chart.yaml @@ -1,6 +1,9 @@ apiVersion: v2 name: kafka -description: Apache Kafka is a distributed streaming platform designed to build real-time pipelines and can be used as a message broker or as a replacement for a log aggregation solution for big data applications. +description: |- + Apache Kafka is a distributed streaming platform designed to build real-time pipelines and can be used as a message + broker or as a replacement for a log aggregation solution for big data applications. This chart provides KubeBlocks' + ClusterDefinition API manifests. annotations: category: Infrastructure @@ -10,7 +13,7 @@ type: application version: 0.5.0-alpha.0 -appVersion: 0.5.0-alpha.0 +appVersion: 3.4.0 home: https://kubeblocks.io/ icon: https://bitnami.com/assets/stacks/kafka/img/kafka-stack-220x234.png diff --git a/deploy/mongodb-cluster/Chart.yaml b/deploy/mongodb-cluster/Chart.yaml index dcf72a9a1..e8f9140ae 100644 --- a/deploy/mongodb-cluster/Chart.yaml +++ b/deploy/mongodb-cluster/Chart.yaml @@ -2,26 +2,11 @@ apiVersion: v2 name: mongodb-cluster description: A MongoDB cluster Helm chart for KubeBlocks -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "6.0.3" home: https://www.mongodb.com icon: https://bitnami.com/assets/stacks/mongodb/img/mongodb-stack-220x234.png diff --git a/deploy/mongodb/Chart.yaml b/deploy/mongodb/Chart.yaml index 62ed7e25c..f54899ef2 100644 --- a/deploy/mongodb/Chart.yaml +++ b/deploy/mongodb/Chart.yaml @@ -2,26 +2,11 @@ apiVersion: v2 name: mongodb description: MongoDB is a document database designed for ease of application development and scaling. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "6.0.3" home: https://www.mongodb.com icon: https://bitnami.com/assets/stacks/mongodb/img/mongodb-stack-220x234.png diff --git a/deploy/nyancat/Chart.yaml b/deploy/nyancat/Chart.yaml index 9c231ee29..040676076 100644 --- a/deploy/nyancat/Chart.yaml +++ b/deploy/nyancat/Chart.yaml @@ -2,25 +2,10 @@ apiVersion: v2 name: nyancat description: A demo application for showing database cluster availability. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. appVersion: 0.5.0-alpha.0 kubeVersion: '>=1.22.0-0' diff --git a/deploy/postgresql-cluster/Chart.yaml b/deploy/postgresql-cluster/Chart.yaml index a257c2fcb..59221a94e 100644 --- a/deploy/postgresql-cluster/Chart.yaml +++ b/deploy/postgresql-cluster/Chart.yaml @@ -1,24 +1,9 @@ apiVersion: v2 name: pgcluster -description: A PostgreSQL Helm chart for KubeBlocks. +description: A PostgreSQL cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "14.7.0" diff --git a/deploy/postgresql-patroni-ha-cluster/Chart.yaml b/deploy/postgresql-patroni-ha-cluster/Chart.yaml index a257c2fcb..f73123fae 100644 --- a/deploy/postgresql-patroni-ha-cluster/Chart.yaml +++ b/deploy/postgresql-patroni-ha-cluster/Chart.yaml @@ -1,24 +1,9 @@ apiVersion: v2 name: pgcluster -description: A PostgreSQL Helm chart for KubeBlocks. +description: A PostgreSQL (with Patroni HA) cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "14.7.0" diff --git a/deploy/postgresql-patroni-ha/Chart.yaml b/deploy/postgresql-patroni-ha/Chart.yaml index 868370105..b82b5a880 100644 --- a/deploy/postgresql-patroni-ha/Chart.yaml +++ b/deploy/postgresql-patroni-ha/Chart.yaml @@ -1,27 +1,12 @@ apiVersion: v2 name: postgresql-ha -description: A postgresql support ha helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. +description: A PostgreSQL (with Patroni HA) cluster definition Helm chart for Kubernetes + type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "14.5.0" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/postgresql/Chart.yaml b/deploy/postgresql/Chart.yaml index 7658083b3..202b4b80b 100644 --- a/deploy/postgresql/Chart.yaml +++ b/deploy/postgresql/Chart.yaml @@ -1,27 +1,12 @@ apiVersion: v2 name: postgresql -description: A postgresql helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. +description: A PostgreSQL cluster definition Helm chart for Kubernetes + type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "14.7.0" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/redis-cluster/Chart.yaml b/deploy/redis-cluster/Chart.yaml index e356cf5a6..a8f597daf 100644 --- a/deploy/redis-cluster/Chart.yaml +++ b/deploy/redis-cluster/Chart.yaml @@ -2,26 +2,11 @@ apiVersion: v2 name: redis-cluster description: An Redis Replication Cluster Helm chart for KubeBlocks. -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "7.0.5" home: https://redis.io/ icon: https://bitnami.com/assets/stacks/redis/img/redis-stack-220x234.png diff --git a/deploy/redis/Chart.yaml b/deploy/redis/Chart.yaml index c5fd30695..20c6562c7 100644 --- a/deploy/redis/Chart.yaml +++ b/deploy/redis/Chart.yaml @@ -1,27 +1,12 @@ apiVersion: v2 name: redis -description: A Redis Helm chart for Kubernetes +description: A Redis cluster definition Helm chart for Kubernetes -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. type: application -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) version: 0.5.0-alpha.0 -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: "7.0.5" home: https://redis.io/ icon: https://bitnami.com/assets/stacks/redis/img/redis-stack-220x234.png From 31f50b371793bf106eeea10e67fa306185785a35 Mon Sep 17 00:00:00 2001 From: wangyelei Date: Thu, 30 Mar 2023 18:46:41 +0800 Subject: [PATCH 03/80] fix: opsquest error message is not clear when component name is wrong (#2328) --- apis/apps/v1alpha1/opsrequest_webhook.go | 235 ++++++++---------- apis/apps/v1alpha1/opsrequest_webhook_test.go | 60 ++--- .../operations/horizontal_scaling_test.go | 2 +- 3 files changed, 121 insertions(+), 176 deletions(-) diff --git a/apis/apps/v1alpha1/opsrequest_webhook.go b/apis/apps/v1alpha1/opsrequest_webhook.go index ff40ab887..f45dc347a 100644 --- a/apis/apps/v1alpha1/opsrequest_webhook.go +++ b/apis/apps/v1alpha1/opsrequest_webhook.go @@ -23,19 +23,14 @@ import ( "reflect" "strings" - "golang.org/x/exp/maps" - storagev1 "k8s.io/api/storage/v1" - "k8s.io/kubectl/pkg/util/storage" - "github.com/pkg/errors" "golang.org/x/exp/slices" corev1 "k8s.io/api/core/v1" + storagev1 "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" - "k8s.io/apimachinery/pkg/util/validation/field" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" logf "sigs.k8s.io/controller-runtime/pkg/log" @@ -73,7 +68,7 @@ func (r *OpsRequest) ValidateUpdate(old runtime.Object) error { opsrequestlog.Info("validate update", "name", r.Name) lastOpsRequest := old.(*OpsRequest) if r.isForbiddenUpdate() && !reflect.DeepEqual(lastOpsRequest.Spec, r.Spec) { - return newInvalidError(OpsRequestKind, r.Name, "spec", fmt.Sprintf("update OpsRequest is forbidden when status.Phase is %s", r.Status.Phase)) + return fmt.Errorf("update OpsRequest: %s is forbidden when status.Phase is %s", r.Name, r.Status.Phase) } // if no spec updated, we should skip validation. // if not, we can not delete the OpsRequest when cluster has been deleted. @@ -120,14 +115,14 @@ func (r *OpsRequest) validateClusterPhase(cluster *Cluster) error { // 1. the opsRequest is Reentrant. // 2. the opsRequest supports concurrent execution of the same kind. if v.Name != r.Name && !slices.Contains(opsBehaviour.FromClusterPhases, v.ToClusterPhase) { - return newInvalidError(OpsRequestKind, r.Name, "spec.type", fmt.Sprintf("Existing OpsRequest: %s is running in Cluster: %s, handle this OpsRequest first", v.Name, cluster.Name)) + return fmt.Errorf("existing OpsRequest: %s is running in Cluster: %s, handle this OpsRequest first", v.Name, cluster.Name) } opsNamesInQueue[i] = v.Name } // check if the opsRequest can be executed in the current cluster phase unless this opsRequest is reentrant. if !slices.Contains(opsBehaviour.FromClusterPhases, cluster.Status.Phase) && !slices.Contains(opsNamesInQueue, r.Name) { - return newInvalidError(OpsRequestKind, r.Name, "spec.type", fmt.Sprintf("%s is forbidden when Cluster.status.Phase is %s", r.Spec.Type, cluster.Status.Phase)) + return fmt.Errorf("opsRequest kind: %s is forbidden when Cluster.status.Phase is %s", r.Spec.Type, cluster.Status.Phase) } return nil } @@ -140,35 +135,22 @@ func (r *OpsRequest) getCluster(ctx context.Context, k8sClient client.Client) (* cluster := &Cluster{} // get cluster resource if err := k8sClient.Get(ctx, types.NamespacedName{Namespace: r.Namespace, Name: r.Spec.ClusterRef}, cluster); err != nil { - return nil, newInvalidError(OpsRequestKind, r.Name, "spec.clusterRef", err.Error()) + return nil, fmt.Errorf("get cluster: %s failed, err: %s", r.Spec.ClusterRef, err.Error()) } return cluster, nil } -func (r *OpsRequest) getClusterDefinition(ctx context.Context, cli client.Client, cluster *Cluster) (*ClusterDefinition, error) { - cd := &ClusterDefinition{} - if err := cli.Get(ctx, types.NamespacedName{Name: cluster.Spec.ClusterDefRef}, cd); err != nil { - return nil, err - } - return cd, nil -} - // Validate validates OpsRequest func (r *OpsRequest) Validate(ctx context.Context, k8sClient client.Client, cluster *Cluster, isCreate bool) error { - var allErrs field.ErrorList if isCreate { if err := r.validateClusterPhase(cluster); err != nil { return err } } - r.validateOps(ctx, k8sClient, cluster, &allErrs) - if len(allErrs) > 0 { - return apierrors.NewInvalid(schema.GroupKind{Group: APIVersion, Kind: OpsRequestKind}, r.Name, allErrs) - } - return nil + return r.validateOps(ctx, k8sClient, cluster) } // ValidateEntry OpsRequest webhook validate entry @@ -188,63 +170,59 @@ func (r *OpsRequest) validateEntry(isCreate bool) error { // validateOps validates ops attributes func (r *OpsRequest) validateOps(ctx context.Context, k8sClient client.Client, - cluster *Cluster, - allErrs *field.ErrorList) { + cluster *Cluster) error { // Check whether the corresponding attribute is legal according to the operation type switch r.Spec.Type { case UpgradeType: - r.validateUpgrade(ctx, k8sClient, cluster, allErrs) + return r.validateUpgrade(ctx, k8sClient) case VerticalScalingType: - r.validateVerticalScaling(cluster, allErrs) + return r.validateVerticalScaling(cluster) case HorizontalScalingType: - r.validateHorizontalScaling(ctx, k8sClient, cluster, allErrs) + return r.validateHorizontalScaling(ctx, k8sClient, cluster) case VolumeExpansionType: - r.validateVolumeExpansion(ctx, k8sClient, cluster, allErrs) + return r.validateVolumeExpansion(ctx, k8sClient, cluster) case RestartType: - r.validateRestart(cluster, allErrs) + return r.validateRestart(cluster) case ReconfiguringType: - r.validateReconfigure(cluster, allErrs) + return r.validateReconfigure(cluster) } + return nil } // validateUpgrade validates spec.restart -func (r *OpsRequest) validateRestart(cluster *Cluster, allErrs *field.ErrorList) { +func (r *OpsRequest) validateRestart(cluster *Cluster) error { restartList := r.Spec.RestartList if len(restartList) == 0 { - addInvalidError(allErrs, "spec.restart", restartList, "can not be empty") - return + return notEmptyError("spec.restart") } compNames := make([]string, len(restartList)) for i, v := range restartList { compNames[i] = v.ComponentName } - r.checkComponentExistence(nil, cluster, compNames, allErrs) + return r.checkComponentExistence(cluster, compNames) } // validateUpgrade validates spec.clusterOps.upgrade func (r *OpsRequest) validateUpgrade(ctx context.Context, - k8sClient client.Client, - cluster *Cluster, - allErrs *field.ErrorList) { + k8sClient client.Client) error { if r.Spec.Upgrade == nil { - addNotFoundError(allErrs, "spec.upgrade", "") - return + return notEmptyError("spec.upgrade") } clusterVersion := &ClusterVersion{} clusterVersionRef := r.Spec.Upgrade.ClusterVersionRef if err := k8sClient.Get(ctx, types.NamespacedName{Name: clusterVersionRef}, clusterVersion); err != nil { - addInvalidError(allErrs, "spec.upgrade.clusterVersionRef", clusterVersionRef, err.Error()) + return fmt.Errorf("get clusterVersion: %s failed, err: %s", clusterVersionRef, err.Error()) } + return nil } // validateVerticalScaling validates api when spec.type is VerticalScaling -func (r *OpsRequest) validateVerticalScaling(cluster *Cluster, allErrs *field.ErrorList) { +func (r *OpsRequest) validateVerticalScaling(cluster *Cluster) error { verticalScalingList := r.Spec.VerticalScalingList if len(verticalScalingList) == 0 { - addInvalidError(allErrs, "spec.verticalScaling", verticalScalingList, "can not be empty") - return + return notEmptyError("spec.verticalScaling") } // validate resources is legal and get component name slice @@ -252,29 +230,25 @@ func (r *OpsRequest) validateVerticalScaling(cluster *Cluster, allErrs *field.Er for i, v := range verticalScalingList { componentNames[i] = v.ComponentName if invalidValue, err := validateVerticalResourceList(v.Requests); err != nil { - addInvalidError(allErrs, fmt.Sprintf("spec.verticalScaling[%d].requests", i), invalidValue, err.Error()) - continue + return invalidValueError(invalidValue, err.Error()) } if invalidValue, err := validateVerticalResourceList(v.Limits); err != nil { - addInvalidError(allErrs, fmt.Sprintf("spec.verticalScaling[%d].limits", i), invalidValue, err.Error()) - continue + return invalidValueError(invalidValue, err.Error()) } if invalidValue, err := compareRequestsAndLimits(v.ResourceRequirements); err != nil { - addInvalidError(allErrs, fmt.Sprintf("spec.verticalScaling[%d].requests", i), invalidValue, err.Error()) + return invalidValueError(invalidValue, err.Error()) } } - - r.checkComponentExistence(nil, cluster, componentNames, allErrs) + return r.checkComponentExistence(cluster, componentNames) } // validateVerticalScaling validate api is legal when spec.type is VerticalScaling -func (r *OpsRequest) validateReconfigure(cluster *Cluster, allErrs *field.ErrorList) { +func (r *OpsRequest) validateReconfigure(cluster *Cluster) error { reconfigure := r.Spec.Reconfigure if reconfigure == nil { - addInvalidError(allErrs, "spec.reconfigure", reconfigure, "can not be empty") - return + return notEmptyError("spec.reconfigure") } - + return nil // TODO validate updated params } @@ -301,89 +275,59 @@ func compareQuantity(requestQuantity, limitQuantity *resource.Quantity) bool { } // validateHorizontalScaling validates api when spec.type is HorizontalScaling -func (r *OpsRequest) validateHorizontalScaling(ctx context.Context, cli client.Client, cluster *Cluster, allErrs *field.ErrorList) { +func (r *OpsRequest) validateHorizontalScaling(ctx context.Context, cli client.Client, cluster *Cluster) error { horizontalScalingList := r.Spec.HorizontalScalingList if len(horizontalScalingList) == 0 { - addInvalidError(allErrs, "spec.horizontalScaling", horizontalScalingList, "can not be empty") - return + return notEmptyError("spec.horizontalScaling") } componentNames := make([]string, len(horizontalScalingList)) for i, v := range horizontalScalingList { componentNames[i] = v.ComponentName } - - clusterDef, err := r.getClusterDefinition(ctx, cli, cluster) - if err != nil { - addInvalidError(allErrs, "spec.horizontalScaling", horizontalScalingList, "get cluster definition error: "+err.Error()) - return - } - r.checkComponentExistence(clusterDef, cluster, componentNames, allErrs) + return r.checkComponentExistence(cluster, componentNames) } // validateVolumeExpansion validates volumeExpansion api when spec.type is VolumeExpansion -func (r *OpsRequest) validateVolumeExpansion(ctx context.Context, cli client.Client, cluster *Cluster, allErrs *field.ErrorList) { +func (r *OpsRequest) validateVolumeExpansion(ctx context.Context, cli client.Client, cluster *Cluster) error { volumeExpansionList := r.Spec.VolumeExpansionList if len(volumeExpansionList) == 0 { - addInvalidError(allErrs, "spec.volumeExpansion", volumeExpansionList, "can not be empty") - return + return notEmptyError("spec.volumeExpansion") } componentNames := make([]string, len(volumeExpansionList)) for i, v := range volumeExpansionList { componentNames[i] = v.ComponentName } - r.checkComponentExistence(nil, cluster, componentNames, allErrs) + if err := r.checkComponentExistence(cluster, componentNames); err != nil { + return err + } - r.checkVolumesAllowExpansion(ctx, cli, cluster, allErrs) + return r.checkVolumesAllowExpansion(ctx, cli, cluster) } // checkComponentExistence checks whether components to be operated exist in cluster spec. -func (r *OpsRequest) checkComponentExistence(clusterDef *ClusterDefinition, cluster *Cluster, compNames []string, errs *field.ErrorList) { +func (r *OpsRequest) checkComponentExistence(cluster *Cluster, compNames []string) error { compSpecNameMap := make(map[string]bool) for _, compSpec := range cluster.Spec.ComponentSpecs { compSpecNameMap[compSpec.Name] = true } - // To keep the compatibility, do a cross validation with ClusterDefinition's components to meet the topology constraint, - // but we should still carefully consider the necessity for the validation here. - validCompNameMap := make(map[string]bool) - if clusterDef == nil { - validCompNameMap = compSpecNameMap - } else { - for _, compSpec := range cluster.Spec.ComponentSpecs { - for _, compDef := range clusterDef.Spec.ComponentDefs { - if compSpec.ComponentDefRef == compDef.Name { - validCompNameMap[compSpec.Name] = true - break - } - } - } - } - var notFoundCompNames []string - var notSupportCompNames []string for _, compName := range compNames { if _, ok := compSpecNameMap[compName]; !ok { notFoundCompNames = append(notFoundCompNames, compName) - continue - } - if _, ok := validCompNameMap[compName]; !ok { - notSupportCompNames = append(notSupportCompNames, compName) } } if len(notFoundCompNames) > 0 { - addInvalidError(errs, fmt.Sprintf("spec.%s[*].componentName", lowercaseInitial(r.Spec.Type)), - notFoundCompNames, "not found in Cluster.spec.components[*].name") - } - if len(notSupportCompNames) > 0 { - addInvalidError(errs, fmt.Sprintf("spec.%s[*].componentName", lowercaseInitial(r.Spec.Type)), - notSupportCompNames, fmt.Sprintf("not supported the %s operation", r.Spec.Type)) + return fmt.Errorf("components: %v not found, you can view the components by command: "+ + "kbcli cluster describe %s -n %s", notFoundCompNames, cluster.Name, r.Namespace) } + return nil } -func (r *OpsRequest) checkVolumesAllowExpansion(ctx context.Context, cli client.Client, cluster *Cluster, errs *field.ErrorList) { +func (r *OpsRequest) checkVolumesAllowExpansion(ctx context.Context, cli client.Client, cluster *Cluster) error { type Entity struct { existInSpec bool storageClassName *string @@ -421,35 +365,57 @@ func (r *OpsRequest) checkVolumesAllowExpansion(ctx context.Context, cli client. if !e.existInSpec { continue } - if allowExpansion, err := checkStorageClassAllowExpansion(ctx, cli, e.storageClassName); err != nil { + if e.storageClassName == nil { + e.storageClassName = r.getSCNameByPvc(ctx, cli, cname, vname) + } + allowExpansion, err := r.checkStorageClassAllowExpansion(ctx, cli, e.storageClassName) + if err != nil { continue // ignore the error and take it as not-supported - } else { - vols[cname][vname] = Entity{e.existInSpec, e.storageClassName, allowExpansion} } + vols[cname][vname] = Entity{e.existInSpec, e.storageClassName, allowExpansion} } } - for i, v := range maps.Values(vols) { - invalid := make([]string, 0) - for vct, e := range v { - if !e.existInSpec || !e.allowExpansion { - invalid = append(invalid, vct) + for cname, compVols := range vols { + var ( + notFound []string + notSupport []string + notSupportSc []string + ) + for vct, e := range compVols { + if !e.existInSpec { + notFound = append(notFound, vct) + } + if !e.allowExpansion { + notSupport = append(notSupport, vct) + if e.storageClassName != nil { + notSupportSc = append(notSupportSc, *e.storageClassName) + } } } - if len(invalid) > 0 { - message := "not support volume expansion, check the StorageClass whether allow volume expansion." - addInvalidError(errs, fmt.Sprintf("spec.volumeExpansion[%d].volumeClaimTemplates[*].name", i), invalid, message) + if len(notFound) > 0 { + return fmt.Errorf("volumeClaimTemplates: %v not found in component: %s, you can view infos by command: "+ + "kbcli cluster describe %s -n %s", notFound, cname, cluster.Name, r.Namespace) + } + if len(notSupport) > 0 { + var notSupportScString string + if len(notSupportSc) > 0 { + notSupportScString = fmt.Sprintf("storageClass: %v of ", notSupportSc) + } + return fmt.Errorf(notSupportScString+"volumeClaimTemplate: %s not support volume expansion in component: %s, you can view infos by command: "+ + "kubectl get sc", notSupport, cname) } } + return nil } // checkStorageClassAllowExpansion checks whether the specified storage class supports volume expansion. -func checkStorageClassAllowExpansion(ctx context.Context, cli client.Client, storageClassName *string) (bool, error) { +func (r *OpsRequest) checkStorageClassAllowExpansion(ctx context.Context, + cli client.Client, + storageClassName *string) (bool, error) { if storageClassName == nil { - // TODO: check the real storage class by pvc. - return checkDefaultStorageClassAllowExpansion(ctx, cli) + return false, nil } - storageClass := &storagev1.StorageClass{} // take not found error as unsupported if err := cli.Get(ctx, types.NamespacedName{Name: *storageClassName}, storageClass); err != nil && !apierrors.IsNotFound(err) { @@ -461,24 +427,23 @@ func checkStorageClassAllowExpansion(ctx context.Context, cli client.Client, sto return *storageClass.AllowVolumeExpansion, nil } -// checkDefaultStorageClassAllowExpansion checks whether the default storage class supports volume expansion. -func checkDefaultStorageClassAllowExpansion(ctx context.Context, cli client.Client) (bool, error) { - storageClassList := &storagev1.StorageClassList{} - if err := cli.List(ctx, storageClassList); err != nil { - return false, err +// getSCNameByPvc gets the storageClassName by pvc. +func (r *OpsRequest) getSCNameByPvc(ctx context.Context, + cli client.Client, + compName, + vctName string) *string { + pvcList := &corev1.PersistentVolumeClaimList{} + if err := cli.List(ctx, pvcList, client.InNamespace(r.Namespace), client.MatchingLabels{ + "app.kubernetes.io/instance": r.Spec.ClusterRef, + "apps.kubeblocks.io/component-name": compName, + "vct.kubeblocks.io/name": vctName, + }, client.Limit(1)); err != nil { + return nil } - for _, sc := range storageClassList.Items { - if sc.Annotations == nil || sc.Annotations[storage.IsDefaultStorageClassAnnotation] != "true" { - continue - } - return sc.AllowVolumeExpansion != nil && *sc.AllowVolumeExpansion, nil + if len(pvcList.Items) == 0 { + return nil } - return false, nil -} - -func lowercaseInitial(opsType OpsType) string { - str := string(opsType) - return strings.ToLower(str[:1]) + str[1:] + return pvcList.Items[0].Spec.StorageClassName } // validateVerticalResourceList checks if k8s resourceList is legal @@ -491,10 +456,10 @@ func validateVerticalResourceList(resourceList map[corev1.ResourceName]resource. return "", nil } -func addInvalidError(allErrs *field.ErrorList, fieldPath string, value interface{}, msg string) { - *allErrs = append(*allErrs, field.Invalid(field.NewPath(fieldPath), value, msg)) +func notEmptyError(target string) error { + return fmt.Errorf(`"%s" can not be empty`, target) } -func addNotFoundError(allErrs *field.ErrorList, fieldPath string, value interface{}) { - *allErrs = append(*allErrs, field.NotFound(field.NewPath(fieldPath), value)) +func invalidValueError(target string, value string) error { + return fmt.Errorf(`invalid value for "%s": %s`, target, value) } diff --git a/apis/apps/v1alpha1/opsrequest_webhook_test.go b/apis/apps/v1alpha1/opsrequest_webhook_test.go index 5334c2177..d5ef49961 100644 --- a/apis/apps/v1alpha1/opsrequest_webhook_test.go +++ b/apis/apps/v1alpha1/opsrequest_webhook_test.go @@ -95,6 +95,10 @@ var _ = Describe("OpsRequest webhook", func() { return storageClass } + notFoundComponentsString := func(notFoundComponents string) string { + return fmt.Sprintf("components: [%s] not found", notFoundComponents) + } + testUpgrade := func(cluster *Cluster) { opsRequest := createTestOpsRequest(clusterName, opsRequestName+"-upgrade", UpgradeType) @@ -133,7 +137,7 @@ var _ = Describe("OpsRequest webhook", func() { return "" } return err.Error() - }).Should(ContainSubstring("Existing OpsRequest: testOpsName")) + }).Should(ContainSubstring("existing OpsRequest: testOpsName")) // test opsRequest reentry addClusterRequestAnnotation(cluster, opsRequest.Name, SpecReconcilingClusterPhase) By("By creating a upgrade opsRequest, it should be succeed") @@ -164,7 +168,7 @@ var _ = Describe("OpsRequest webhook", func() { patch = client.MergeFrom(opsRequest.DeepCopy()) opsRequest.Spec.ClusterRef = newClusterName - return Expect(k8sClient.Patch(ctx, opsRequest, patch).Error()).To(ContainSubstring("update OpsRequest is forbidden when status.Phase is Succeed")) + return Expect(k8sClient.Patch(ctx, opsRequest, patch).Error()).To(ContainSubstring("is forbidden when status.Phase is Succeed")) }).Should(BeTrue()) } @@ -201,13 +205,13 @@ var _ = Describe("OpsRequest webhook", func() { By("By testing verticalScaling opsRequest components is not exist") opsRequest := createTestOpsRequest(clusterName, opsRequestName, VerticalScalingType) opsRequest.Spec.VerticalScalingList = []VerticalScaling{verticalScalingList[0]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notFoundComponentsString("vs-not-exist"))) By("By testing verticalScaling opsRequest components is not consistent") opsRequest = createTestOpsRequest(clusterName, opsRequestName, VerticalScalingType) // [0] is not exist, and [1] is valid. opsRequest.Spec.VerticalScalingList = []VerticalScaling{verticalScalingList[0], verticalScalingList[1]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found")) By("By testing verticalScaling opsRequest components partly") opsRequest = createTestOpsRequest(clusterName, opsRequestName, VerticalScalingType) @@ -276,36 +280,29 @@ var _ = Describe("OpsRequest webhook", func() { By("By testing volumeExpansion - target component not exist") opsRequest := createTestOpsRequest(clusterName, opsRequestName, VolumeExpansionType) opsRequest.Spec.VolumeExpansionList = []VolumeExpansion{volumeExpansionList[0]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notFoundComponentsString("ve-not-exist"))) By("By testing volumeExpansion - target volume not exist") opsRequest.Spec.VolumeExpansionList = []VolumeExpansion{volumeExpansionList[1]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not support volume expansion, check the StorageClass whether allow volume expansion")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("volumeClaimTemplates: [log] not found in component: replicasets")) By("By testing volumeExpansion - create a new storage class") storageClassName := "sc-test-volume-expansion" storageClass := createStorageClass(testCtx.Ctx, storageClassName, "false", true) Expect(storageClass != nil).Should(BeTrue()) - By("By testing volumeExpansion - has no default storage class") + By("By testing volumeExpansion - has no pvc") for _, compSpec := range cluster.Spec.ComponentSpecs { for _, vct := range compSpec.VolumeClaimTemplates { Expect(vct.Spec.StorageClassName == nil).Should(BeTrue()) } } - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not support volume expansion, check the StorageClass whether allow volume expansion.")) - - By("By testing volumeExpansion - patch it as default and use it to re-create") - if storageClass.Annotations == nil { - storageClass.Annotations = make(map[string]string) - } - storageClass.Annotations[storage.IsDefaultStorageClassAnnotation] = "true" - Expect(testCtx.Cli.Update(testCtx.Ctx, storageClass)).Should(BeNil()) - Eventually(func() bool { - opsRequest.Spec.VolumeExpansionList = []VolumeExpansion{volumeExpansionList[2]} - return testCtx.CheckedCreateObj(ctx, opsRequest) == nil - }).Should(BeTrue()) - + opsRequest.Spec.VolumeExpansionList = []VolumeExpansion{volumeExpansionList[2]} + notSupportMsg := "volumeClaimTemplate: [data] not support volume expansion in component: replicasets, you can view infos by command: kubectl get sc" + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notSupportMsg)) + // TODO + By("testing volumeExpansion - pvc exists") + // TODO By("By testing volumeExpansion - (TODO)use specified storage class") // Eventually(func() bool { // opsRequest.Spec.VolumeExpansionList = []VolumeExpansion{volumeExpansionList[3]} @@ -347,29 +344,12 @@ var _ = Describe("OpsRequest webhook", func() { By("By testing horizontalScaling - target component not exist") opsRequest := createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) opsRequest.Spec.HorizontalScalingList = []HorizontalScaling{hScalingList[0]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notFoundComponentsString("hs-not-exist"))) By("By testing horizontalScaling - target component not exist partly") opsRequest = createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) opsRequest.Spec.HorizontalScalingList = []HorizontalScaling{hScalingList[0], hScalingList[2]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) - - By("By testing horizontalScaling - target component not supported") - opsRequest = createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) - opsRequest.Spec.HorizontalScalingList = []HorizontalScaling{hScalingList[1]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not supported the HorizontalScaling operation")) - - By("By testing horizontalScaling - target component not supported partly") - opsRequest = createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) - opsRequest.Spec.HorizontalScalingList = []HorizontalScaling{hScalingList[1], hScalingList[2]} - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not supported the HorizontalScaling operation")) - - By("By testing horizontalScaling - target component not exist and not supported partly") - opsRequest = createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) - opsRequest.Spec.HorizontalScalingList = []HorizontalScaling{hScalingList[0], hScalingList[1], hScalingList[2]} - err := testCtx.CreateObj(ctx, opsRequest) - Expect(err.Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) - Expect(err.Error()).To(ContainSubstring("not supported the HorizontalScaling operation")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notFoundComponentsString("hs-not-exist"))) By("By testing horizontalScaling. if api is legal, it will create successfully") opsRequest = createTestOpsRequest(clusterName, opsRequestName, HorizontalScalingType) @@ -410,7 +390,7 @@ var _ = Describe("OpsRequest webhook", func() { opsRequest.Spec.RestartList = []ComponentOps{ {ComponentName: "replicasets1"}, } - Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring("not found in Cluster.spec.components[*].name")) + Expect(testCtx.CreateObj(ctx, opsRequest).Error()).To(ContainSubstring(notFoundComponentsString("replicasets1"))) By("By testing restart. if api is legal, it will create successfully") Eventually(func() bool { diff --git a/controllers/apps/operations/horizontal_scaling_test.go b/controllers/apps/operations/horizontal_scaling_test.go index 14f74daf9..11ec55172 100644 --- a/controllers/apps/operations/horizontal_scaling_test.go +++ b/controllers/apps/operations/horizontal_scaling_test.go @@ -101,7 +101,7 @@ var _ = Describe("HorizontalScaling OpsRequest", func() { } })).ShouldNot(HaveOccurred()) _, err = GetOpsManager().Do(reqCtx, k8sClient, opsRes) - Expect(err.Error()).Should(ContainSubstring("Existing OpsRequest:")) + Expect(err.Error()).Should(ContainSubstring("existing OpsRequest:")) // reset cluster annotation Expect(testapps.ChangeObj(&testCtx, opsRes.Cluster, func() { From 6b92ae0ce43f41764d23e62bb1861cf66501fe79 Mon Sep 17 00:00:00 2001 From: wangyelei Date: Thu, 30 Mar 2023 19:50:31 +0800 Subject: [PATCH 04/80] chore: add LatestOpsRequestProcessed condition in cluster.status.conditions and improve kbcli list cluster cmd (#2275) --- apis/apps/v1alpha1/cluster_types.go | 18 +- apis/apps/v1alpha1/opsrequest_webhook.go | 5 +- apis/apps/v1alpha1/type.go | 35 +- .../bases/apps.kubeblocks.io_clusters.yaml | 34 +- .../bases/apps.kubeblocks.io_opsrequests.yaml | 3 +- controllers/apps/cluster_controller.go | 3 + controllers/apps/cluster_controller_test.go | 24 +- controllers/apps/cluster_status_utils.go | 45 +- controllers/apps/cluster_status_utils_test.go | 2 +- .../apps/components/component_status.go | 4 +- .../stateful_set_controller_test.go | 4 +- .../apps/components/util/component_utils.go | 15 - .../components/util/component_utils_test.go | 14 - .../apps/operations/horizontal_scaling.go | 10 +- controllers/apps/operations/ops_manager.go | 7 +- .../apps/operations/ops_progress_util.go | 3 +- controllers/apps/operations/ops_util.go | 62 +-- controllers/apps/operations/ops_util_test.go | 9 - controllers/apps/operations/reconfigure.go | 7 +- controllers/apps/operations/restart.go | 12 +- controllers/apps/operations/start.go | 7 +- controllers/apps/operations/stop.go | 8 +- controllers/apps/operations/type.go | 24 + controllers/apps/operations/upgrade.go | 9 +- .../apps/operations/util/common_util.go | 5 +- .../apps/operations/util/common_util_test.go | 8 +- .../apps/operations/vertical_scaling.go | 9 +- .../apps/operations/volume_expansion.go | 10 +- .../apps/operations/volume_expansion_test.go | 19 +- .../operations/volume_expansion_updater.go | 14 +- .../apps/opsrequest_controller_test.go | 29 +- controllers/apps/systemaccount_controller.go | 4 +- controllers/apps/tls_utils_test.go | 2 +- .../crds/apps.kubeblocks.io_clusters.yaml | 34 +- .../crds/apps.kubeblocks.io_opsrequests.yaml | 3 +- internal/cli/cluster/cluster.go | 7 + internal/cli/cmd/cluster/list_ops_test.go | 34 +- internal/cli/cmd/cluster/list_test.go | 36 +- internal/cli/testing/fake.go | 3 +- .../lifecycle/cluster_plan_builder.go | 176 +++++-- .../lifecycle/cluster_plan_utils.go | 9 +- .../lifecycle/cluster_status_conditions.go | 215 ++------- .../controller/lifecycle/transform_types.go | 10 +- .../lifecycle/transformer_cluster_status.go | 438 ++++++++---------- .../controller/lifecycle/transformer_init.go | 6 +- .../lifecycle/transformer_object_action.go | 4 + .../transformer_sts_horizontal_scaling.go | 19 +- 47 files changed, 699 insertions(+), 759 deletions(-) diff --git a/apis/apps/v1alpha1/cluster_types.go b/apis/apps/v1alpha1/cluster_types.go index 777ee73bc..49aae1b3b 100644 --- a/apis/apps/v1alpha1/cluster_types.go +++ b/apis/apps/v1alpha1/cluster_types.go @@ -77,10 +77,8 @@ type ClusterStatus struct { // Stopped: cluster has stopped, all its components are stopped. [terminal state] // Failed: cluster is unavailable. [terminal state] // Abnormal: Cluster is still running, but part of its components are Abnormal/Failed. [terminal state] - // Starting: Cluster has entered starting process. + // Creating: Cluster has entered creating process. // Updating: Cluster has entered updating process, triggered by Spec. updated. - // Stopping: Cluster has entered a stopping process. - // if the component workload type is Consensus/Replication, the Leader/Primary pod must be ready in Abnormal phase. // +optional Phase ClusterPhase `json:"phase,omitempty"` @@ -183,16 +181,14 @@ type ComponentMessageMap map[string]string // ClusterComponentStatus record components status information type ClusterComponentStatus struct { // phase describes the phase of the component, the detail information of the phases are as following: - // Failed: component is unavailable, i.e, all pods are not ready for Stateless/Stateful component; - // Leader/Primary pod is not ready for Consensus/Replication component. // Running: component is running. [terminal state] // Stopped: component is stopped, as no running pod. [terminal state] - // Failed: component has failed to start running. [terminal state] - // Abnormal: component is running but part of its pods are not ready. [terminal state] - // Starting: component has entered starting process. + // Failed: component is unavailable. i.e, all pods are not ready for Stateless/Stateful component, + // Leader/Primary pod is not ready for Consensus/Replication component. [terminal state] + // Abnormal: component is running but part of its pods are not ready. + // Leader/Primary pod is ready for Consensus/Replication component. [terminal state] + // Creating: component has entered creating process. // Updating: component has entered updating process, triggered by Spec. updated. - // Stopping: component has entered a stopping process. - // If the component workload type is Consensus/Replication, the Leader/Primary pod must be ready in Abnormal phase. Phase ClusterComponentPhase `json:"phase,omitempty"` // message records the component details message in current phase. @@ -557,7 +553,7 @@ func GetClusterUpRunningPhases() []ClusterPhase { return []ClusterPhase{ RunningClusterPhase, AbnormalClusterPhase, - // FailedClusterPhase, // REVIEW/TODO: single component with single pod component are handled as FailedClusterPhase, ought to remove this. + FailedClusterPhase, // REVIEW/TODO: single component with single pod component are handled as FailedClusterPhase, ought to remove this. } } diff --git a/apis/apps/v1alpha1/opsrequest_webhook.go b/apis/apps/v1alpha1/opsrequest_webhook.go index f45dc347a..db701388b 100644 --- a/apis/apps/v1alpha1/opsrequest_webhook.go +++ b/apis/apps/v1alpha1/opsrequest_webhook.go @@ -41,8 +41,7 @@ import ( var ( opsrequestlog = logf.Log.WithName("opsrequest-resource") opsRequestAnnotationKey = "kubeblocks.io/ops-request" - - // OpsRequestBehaviourMapper records in which cluster phases OpsRequest can run + // OpsRequestBehaviourMapper records the opsRequest behaviour according to the OpsType. OpsRequestBehaviourMapper = map[OpsType]OpsRequestBehaviour{} ) @@ -114,7 +113,7 @@ func (r *OpsRequest) validateClusterPhase(cluster *Cluster) error { // judge whether the opsRequest meets the following conditions: // 1. the opsRequest is Reentrant. // 2. the opsRequest supports concurrent execution of the same kind. - if v.Name != r.Name && !slices.Contains(opsBehaviour.FromClusterPhases, v.ToClusterPhase) { + if v.Name != r.Name { return fmt.Errorf("existing OpsRequest: %s is running in Cluster: %s, handle this OpsRequest first", v.Name, cluster.Name) } opsNamesInQueue[i] = v.Name diff --git a/apis/apps/v1alpha1/type.go b/apis/apps/v1alpha1/type.go index 6254b052c..081534160 100644 --- a/apis/apps/v1alpha1/type.go +++ b/apis/apps/v1alpha1/type.go @@ -32,7 +32,7 @@ const ( // ClusterPhase defines the Cluster CR .status.phase // +enum -// +kubebuilder:validation:Enum={Running,Stopped,Failed,Abnormal,Starting,Updating,Stopping} +// +kubebuilder:validation:Enum={Running,Stopped,Failed,Abnormal,Creating,Updating} type ClusterPhase string const ( @@ -41,15 +41,14 @@ const ( StoppedClusterPhase ClusterPhase = "Stopped" FailedClusterPhase ClusterPhase = "Failed" AbnormalClusterPhase ClusterPhase = "Abnormal" // Abnormal is a sub-state of failed, where one of the cluster components has "Failed" or "Abnormal" status phase. - StartingClusterPhase ClusterPhase = "Starting" + CreatingClusterPhase ClusterPhase = "Creating" SpecReconcilingClusterPhase ClusterPhase = "Updating" - StoppingClusterPhase ClusterPhase = "Stopping" // DeletingClusterPhase ClusterPhase = "Deleting" // DO REVIEW: may merged with Stopping ) // ClusterComponentPhase defines the Cluster CR .status.components.phase // +enum -// +kubebuilder:validation:Enum={Running,Stopped,Failed,Abnormal,Updating,Starting,Stopping} +// +kubebuilder:validation:Enum={Running,Stopped,Failed,Abnormal,Creating,Updating} type ClusterComponentPhase string const ( @@ -58,8 +57,7 @@ const ( FailedClusterCompPhase ClusterComponentPhase = "Failed" AbnormalClusterCompPhase ClusterComponentPhase = "Abnormal" // Abnormal is a sub-state of failed, where one or more workload pods is not in "Running" phase. SpecReconcilingClusterCompPhase ClusterComponentPhase = "Updating" - StartingClusterCompPhase ClusterComponentPhase = "Starting" - StoppingClusterCompPhase ClusterComponentPhase = "Stopping" + CreatingClusterCompPhase ClusterComponentPhase = "Creating" // DeletingClusterCompPhase ClusterComponentPhase = "Deleting" // DO REVIEW: may merged with Stopping // REVIEW: following are variant of "Updating", why not have "Updating" phase with detail Status.Conditions @@ -72,6 +70,16 @@ const ( // RollingClusterCompPhase ClusterComponentPhase = "Rolling" // REVIEW: original value is Rebooting, and why not having stopping -> stopped -> starting -> running ) +const ( + // define the cluster condition type + ConditionTypeLatestOpsRequestProcessed = "LatestOpsRequestProcessed" // ConditionTypeLatestOpsRequestProcessed describes whether the latest OpsRequest that affect the cluster lifecycle has been processed. + ConditionTypeProvisioningStarted = "ProvisioningStarted" // ConditionTypeProvisioningStarted the operator starts resource provisioning to create or change the cluster + ConditionTypeApplyResources = "ApplyResources" // ConditionTypeApplyResources the operator start to apply resources to create or change the cluster + ConditionTypeReplicasReady = "ReplicasReady" // ConditionTypeReplicasReady all pods of components are ready + ConditionTypeReady = "Ready" // ConditionTypeReady all components are running + +) + // Phase defines the ClusterDefinition and ClusterVersion CR .status.phase // +enum // +kubebuilder:validation:Enum={Available,Unavailable} @@ -106,9 +114,9 @@ const ( VolumeExpansionType OpsType = "VolumeExpansion" UpgradeType OpsType = "Upgrade" ReconfiguringType OpsType = "Reconfiguring" - RestartType OpsType = "Restart" - StopType OpsType = "Stop" - StartType OpsType = "Start" + RestartType OpsType = "Restart" // RestartType the restart operation is a special case of the rolling update operation. + StopType OpsType = "Stop" // StopType the stop operation will delete all pods in a cluster concurrently. + StartType OpsType = "Start" // StartType the start operation will start the pods which is deleted in stop operation. ExposeType OpsType = "Expose" ) @@ -203,7 +211,7 @@ const ( DedicatedNode TenancyType = "DedicatedNode" ) -// ProgressStatus defined +// ProgressStatus defines the status of the opsRequest progress. // +enum // +kubebuilder:validation:Enum={Processing,Pending,Failed,Succeed} type ProgressStatus string @@ -216,15 +224,16 @@ const ( ) type OpsRequestBehaviour struct { - FromClusterPhases []ClusterPhase - ToClusterPhase ClusterPhase + FromClusterPhases []ClusterPhase + ToClusterPhase ClusterPhase + ProcessingReasonInClusterCondition string } type OpsRecorder struct { // name OpsRequest name Name string `json:"name"` // clusterPhase the cluster phase when the OpsRequest is running - ToClusterPhase ClusterPhase `json:"clusterPhase"` + Type OpsType `json:"type"` } // ProvisionPolicyType defines the policy for creating accounts. diff --git a/config/crd/bases/apps.kubeblocks.io_clusters.yaml b/config/crd/bases/apps.kubeblocks.io_clusters.yaml index fa070bd75..a6b1eafec 100644 --- a/config/crd/bases/apps.kubeblocks.io_clusters.yaml +++ b/config/crd/bases/apps.kubeblocks.io_clusters.yaml @@ -789,27 +789,24 @@ spec: type: object phase: description: 'phase describes the phase of the component, the - detail information of the phases are as following: Failed: - component is unavailable, i.e, all pods are not ready for - Stateless/Stateful component; Leader/Primary pod is not ready - for Consensus/Replication component. Running: component is - running. [terminal state] Stopped: component is stopped, as - no running pod. [terminal state] Failed: component has failed - to start running. [terminal state] Abnormal: component is - running but part of its pods are not ready. [terminal state] - Starting: component has entered starting process. Updating: + detail information of the phases are as following: Running: + component is running. [terminal state] Stopped: component + is stopped, as no running pod. [terminal state] Failed: component + is unavailable. i.e, all pods are not ready for Stateless/Stateful + component, Leader/Primary pod is not ready for Consensus/Replication + component. [terminal state] Abnormal: component is running + but part of its pods are not ready. Leader/Primary pod is + ready for Consensus/Replication component. [terminal state] + Creating: component has entered creating process. Updating: component has entered updating process, triggered by Spec. - updated. Stopping: component has entered a stopping process. - If the component workload type is Consensus/Replication, the - Leader/Primary pod must be ready in Abnormal phase.' + updated.' enum: - Running - Stopped - Failed - Abnormal + - Creating - Updating - - Starting - - Stopping type: string podsReady: description: podsReady checks if all pods of the component are @@ -940,19 +937,16 @@ spec: cluster has stopped, all its components are stopped. [terminal state] Failed: cluster is unavailable. [terminal state] Abnormal: Cluster is still running, but part of its components are Abnormal/Failed. - [terminal state] Starting: Cluster has entered starting process. + [terminal state] Creating: Cluster has entered creating process. Updating: Cluster has entered updating process, triggered by Spec. - updated. Stopping: Cluster has entered a stopping process. if the - component workload type is Consensus/Replication, the Leader/Primary - pod must be ready in Abnormal phase.' + updated.' enum: - Running - Stopped - Failed - Abnormal - - Starting + - Creating - Updating - - Stopping type: string type: object type: object diff --git a/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml b/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml index 507dae163..7101384a0 100644 --- a/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml +++ b/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml @@ -380,9 +380,8 @@ spec: - Stopped - Failed - Abnormal + - Creating - Updating - - Starting - - Stopping type: string progressDetails: description: progressDetails describes the progress details diff --git a/controllers/apps/cluster_controller.go b/controllers/apps/cluster_controller.go index af644dd42..840c35009 100644 --- a/controllers/apps/cluster_controller.go +++ b/controllers/apps/cluster_controller.go @@ -32,6 +32,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/log" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + dataprotectionv1alpha1 "github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1" "github.com/apecloud/kubeblocks/controllers/k8score" "github.com/apecloud/kubeblocks/internal/constant" "github.com/apecloud/kubeblocks/internal/controller/lifecycle" @@ -154,6 +155,8 @@ func (r *ClusterReconciler) SetupWithManager(mgr ctrl.Manager) error { Owns(&corev1.PersistentVolumeClaim{}). Owns(&policyv1.PodDisruptionBudget{}). Owns(&snapshotv1.VolumeSnapshot{}). + Owns(&dataprotectionv1alpha1.BackupPolicy{}). + Owns(&dataprotectionv1alpha1.Backup{}). Complete(r) } diff --git a/controllers/apps/cluster_controller_test.go b/controllers/apps/cluster_controller_test.go index 78712bc63..4efad311d 100644 --- a/controllers/apps/cluster_controller_test.go +++ b/controllers/apps/cluster_controller_test.go @@ -118,7 +118,7 @@ var _ = Describe("Cluster Controller", func() { waitForCreatingResourceCompletely := func(clusterKey client.ObjectKey, compNames ...string) { Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) for _, compName := range compNames { - Eventually(testapps.GetClusterComponentPhase(testCtx, clusterKey.Name, compName)).Should(Equal(appsv1alpha1.StartingClusterCompPhase)) + Eventually(testapps.GetClusterComponentPhase(testCtx, clusterKey.Name, compName)).Should(Equal(appsv1alpha1.CreatingClusterCompPhase)) } } @@ -365,7 +365,7 @@ var _ = Describe("Cluster Controller", func() { By("Waiting for the cluster enter running phase") Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) - Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) // REVIEW: this test flow @@ -384,7 +384,7 @@ var _ = Describe("Cluster Controller", func() { By("Waiting for the cluster enter running phase") Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) - Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) // REVIEW: this test flow @@ -440,7 +440,7 @@ var _ = Describe("Cluster Controller", func() { By("Waiting for the cluster enter running phase") Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) - Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) replicasSeq := []int32{5, 3, 1, 0, 2, 4} expectedOG := int64(1) @@ -452,7 +452,7 @@ var _ = Describe("Cluster Controller", func() { By("Checking cluster status and the number of replicas changed") Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, fetched *appsv1alpha1.Cluster) { g.Expect(fetched.Status.ObservedGeneration).To(BeEquivalentTo(expectedOG)) - g.Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + g.Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) })).Should(Succeed()) Eventually(func(g Gomega) { stsList := testk8s.ListAndCheckStatefulSet(&testCtx, clusterKey) @@ -1615,9 +1615,9 @@ var _ = Describe("Cluster Controller", func() { By("test when clusterDefinition not found") Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, tmpCluster *appsv1alpha1.Cluster) { - condition := meta.FindStatusCondition(tmpCluster.Status.Conditions, lifecycle.ConditionTypeProvisioningStarted) + condition := meta.FindStatusCondition(tmpCluster.Status.Conditions, appsv1alpha1.ConditionTypeProvisioningStarted) g.Expect(condition).ShouldNot(BeNil()) - g.Expect(condition.Reason).Should(BeEquivalentTo(constant.ReasonNotFoundCR)) + g.Expect(condition.Reason).Should(BeEquivalentTo(lifecycle.ReasonPreCheckFailed)) })).Should(Succeed()) // TODO: removed conditionsError phase need to review correct-ness of following commented off block: @@ -1653,9 +1653,9 @@ var _ = Describe("Cluster Controller", func() { Eventually(func(g Gomega) { updateClusterAnnotation(cluster) g.Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, cluster *appsv1alpha1.Cluster) { - condition := meta.FindStatusCondition(cluster.Status.Conditions, lifecycle.ConditionTypeProvisioningStarted) + condition := meta.FindStatusCondition(cluster.Status.Conditions, appsv1alpha1.ConditionTypeProvisioningStarted) g.Expect(condition).ShouldNot(BeNil()) - g.Expect(condition.Reason).Should(BeEquivalentTo(constant.ReasonRefCRUnavailable)) + g.Expect(condition.Reason).Should(BeEquivalentTo(lifecycle.ReasonPreCheckFailed)) })).Should(Succeed()) }).Should(Succeed()) @@ -1672,7 +1672,7 @@ var _ = Describe("Cluster Controller", func() { updateClusterAnnotation(cluster) By("test preCheckFailed") Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, cluster *appsv1alpha1.Cluster) { - condition := meta.FindStatusCondition(cluster.Status.Conditions, lifecycle.ConditionTypeProvisioningStarted) + condition := meta.FindStatusCondition(cluster.Status.Conditions, appsv1alpha1.ConditionTypeProvisioningStarted) g.Expect(condition != nil && condition.Reason == lifecycle.ReasonPreCheckFailed).Should(BeTrue()) })).Should(Succeed()) @@ -1682,7 +1682,7 @@ var _ = Describe("Cluster Controller", func() { })()).ShouldNot(HaveOccurred()) Eventually(testapps.CheckObj(&testCtx, client.ObjectKeyFromObject(cluster), func(g Gomega, tmpCluster *appsv1alpha1.Cluster) { - g.Expect(tmpCluster.Status.Phase).Should(Equal(appsv1alpha1.StartingClusterPhase)) + g.Expect(tmpCluster.Status.Phase).Should(Equal(appsv1alpha1.CreatingClusterPhase)) g.Expect(tmpCluster.Status.ObservedGeneration).ShouldNot(BeZero()) })).Should(Succeed()) @@ -1693,7 +1693,7 @@ var _ = Describe("Cluster Controller", func() { Eventually(testapps.CheckObj(&testCtx, client.ObjectKeyFromObject(cluster), func(g Gomega, tmpCluster *appsv1alpha1.Cluster) { - condition := meta.FindStatusCondition(tmpCluster.Status.Conditions, lifecycle.ConditionTypeApplyResources) + condition := meta.FindStatusCondition(tmpCluster.Status.Conditions, appsv1alpha1.ConditionTypeApplyResources) g.Expect(condition != nil && condition.Reason == lifecycle.ReasonApplyResourcesFailed).Should(BeTrue()) })).Should(Succeed()) }) diff --git a/controllers/apps/cluster_status_utils.go b/controllers/apps/cluster_status_utils.go index 27682b326..3b11d1268 100644 --- a/controllers/apps/cluster_status_utils.go +++ b/controllers/apps/cluster_status_utils.go @@ -40,10 +40,8 @@ import ( intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" ) -const ( - // EventTimeOut timeout of the event - EventTimeOut = 30 * time.Second -) +// EventTimeOut timeout of the event +const EventTimeOut = 30 * time.Second // isTargetKindForEvent checks the event involve object is the target resources func isTargetKindForEvent(event *corev1.Event) bool { @@ -73,7 +71,10 @@ func isExistsEventMsg(compStatusMessage map[string]string, event *corev1.Event) } // updateComponentStatusMessage updates component status message map -func updateComponentStatusMessage(compStatus *appsv1alpha1.ClusterComponentStatus, event *corev1.Event) { +func updateComponentStatusMessage(cluster *appsv1alpha1.Cluster, + compName string, + compStatus *appsv1alpha1.ClusterComponentStatus, + event *corev1.Event) { var ( kind = event.InvolvedObject.Kind name = event.InvolvedObject.Name @@ -84,34 +85,28 @@ func updateComponentStatusMessage(compStatus *appsv1alpha1.ClusterComponentStatu message += event.Message + ";" } compStatus.SetObjectMessage(kind, name, message) + cluster.Status.SetComponentStatus(compName, *compStatus) } // needSyncComponentStatusForEvent checks whether the component status needs to be synchronized the cluster status by event func needSyncComponentStatusForEvent(cluster *appsv1alpha1.Cluster, componentName string, phase appsv1alpha1.ClusterComponentPhase, event *corev1.Event) bool { - var ( - status = &cluster.Status - compStatus appsv1alpha1.ClusterComponentStatus - ok bool - ) if phase == "" { return false } - if compStatus, ok = cluster.Status.Components[componentName]; !ok { + compStatus, ok := cluster.Status.Components[componentName] + if !ok { compStatus = appsv1alpha1.ClusterComponentStatus{Phase: phase} - updateComponentStatusMessage(&compStatus, event) - status.SetComponentStatus(componentName, compStatus) + updateComponentStatusMessage(cluster, componentName, &compStatus, event) return true } if compStatus.Phase != phase { compStatus.Phase = phase - updateComponentStatusMessage(&compStatus, event) - status.SetComponentStatus(componentName, compStatus) + updateComponentStatusMessage(cluster, componentName, &compStatus, event) return true } // check whether it is a new warning event and the component phase is running if !isExistsEventMsg(compStatus.Message, event) && phase != appsv1alpha1.RunningClusterCompPhase { - updateComponentStatusMessage(&compStatus, event) - status.SetComponentStatus(componentName, compStatus) + updateComponentStatusMessage(cluster, componentName, &compStatus, event) return true } return false @@ -144,21 +139,24 @@ func getEventInvolvedObject(ctx context.Context, cli client.Client, event *corev } // handleClusterPhaseWhenCompsNotReady handles the Cluster.status.phase when some components are Abnormal or Failed. -// REVIEW: seem duplicated handling -// Deprecated: +// TODO: Clear definitions need to be added to determine whether components will affect cluster availability in ClusterDefinition. func handleClusterPhaseWhenCompsNotReady(cluster *appsv1alpha1.Cluster, componentMap map[string]string, clusterAvailabilityEffectMap map[string]bool) { var ( - clusterIsFailed bool - failedCompCount int + clusterIsFailed bool + failedCompCount int + isVolumeExpanding bool ) + opsRecords, _ := opsutil.GetOpsRequestSliceFromCluster(cluster) + if len(opsRecords) != 0 && opsRecords[0].Type == appsv1alpha1.VolumeExpansionType { + isVolumeExpanding = true + } for k, v := range cluster.Status.Components { // determine whether other components are still doing operation, i.e., create/restart/scaling. // waiting for operation to complete except for volumeExpansion operation. // because this operation will not affect cluster availability. - // TODO: for appsv1alpha1.VolumeExpandingPhas requires extra handling - if !util.IsCompleted(v.Phase) { + if !slices.Contains(appsv1alpha1.GetComponentTerminalPhases(), v.Phase) && !isVolumeExpanding { return } if v.Phase == appsv1alpha1.FailedClusterCompPhase { @@ -267,6 +265,7 @@ func handleClusterStatusByEvent(ctx context.Context, cli client.Client, recorder return opsutil.MarkRunningOpsRequestAnnotation(ctx, cli, cluster) } +// TODO: Unified cluster event processing // handleEventForClusterStatus handles event for cluster Warning and Failed phase func handleEventForClusterStatus(ctx context.Context, cli client.Client, recorder record.EventRecorder, event *corev1.Event) error { diff --git a/controllers/apps/cluster_status_utils_test.go b/controllers/apps/cluster_status_utils_test.go index c858fb9a9..374c24ae7 100644 --- a/controllers/apps/cluster_status_utils_test.go +++ b/controllers/apps/cluster_status_utils_test.go @@ -167,7 +167,7 @@ var _ = Describe("test cluster Failed/Abnormal phase", func() { func(g Gomega, fetched *appsv1alpha1.Cluster) { g.Expect(fetched.Generation).To(BeEquivalentTo(1)) g.Expect(fetched.Status.ObservedGeneration).To(BeEquivalentTo(1)) - g.Expect(fetched.Status.Phase).To(Equal(appsv1alpha1.StartingClusterPhase)) + g.Expect(fetched.Status.Phase).To(Equal(appsv1alpha1.CreatingClusterPhase)) })).Should(Succeed()) By("watch normal event") diff --git a/controllers/apps/components/component_status.go b/controllers/apps/components/component_status.go index d545a7559..8a90270e6 100644 --- a/controllers/apps/components/component_status.go +++ b/controllers/apps/components/component_status.go @@ -43,7 +43,6 @@ import ( // and send the BackOff (Normal) event. If it has already consumed the 25 burst quota to send event, event can only be // sent in the rate of once per 300s, in this way, the subsequent warning events of ImagePullError would be dropped. type ComponentStatusSynchronizer struct { - ctx context.Context cli client.Client cluster *appsv1alpha1.Cluster component types.Component @@ -64,7 +63,6 @@ func newClusterStatusSynchronizer(ctx context.Context, return nil, err } return &ComponentStatusSynchronizer{ - ctx: ctx, cli: cli, cluster: cluster, component: component, @@ -137,7 +135,7 @@ func (cs *ComponentStatusSynchronizer) Update(ctx context.Context, obj client.Ob logger.Info("component status changed", "componentName", componentName, "phase", componentStatus.Phase, "componentIsRunning", isRunning, "podsAreReady", podsReady) patch := client.MergeFrom(clusterDeepCopy) - if err = cs.cli.Status().Patch(cs.ctx, cluster, patch); err != nil { + if err = cs.cli.Status().Patch(ctx, cluster, patch); err != nil { return false, err } } diff --git a/controllers/apps/components/stateful_set_controller_test.go b/controllers/apps/components/stateful_set_controller_test.go index 691d7495e..e03ae6343 100644 --- a/controllers/apps/components/stateful_set_controller_test.go +++ b/controllers/apps/components/stateful_set_controller_test.go @@ -173,9 +173,9 @@ var _ = Describe("StatefulSet Controller", func() { By("mock component of cluster is stopping") Expect(testapps.GetAndChangeObjStatus(&testCtx, client.ObjectKeyFromObject(cluster), func(tmpCluster *appsv1alpha1.Cluster) { - tmpCluster.Status.Phase = appsv1alpha1.StoppingClusterPhase + tmpCluster.Status.Phase = appsv1alpha1.SpecReconcilingClusterPhase tmpCluster.Status.SetComponentStatus(consensusCompName, appsv1alpha1.ClusterComponentStatus{ - Phase: appsv1alpha1.StoppingClusterCompPhase, + Phase: appsv1alpha1.SpecReconcilingClusterCompPhase, }) })()).Should(Succeed()) diff --git a/controllers/apps/components/util/component_utils.go b/controllers/apps/components/util/component_utils.go index fab7ae356..952e376fb 100644 --- a/controllers/apps/components/util/component_utils.go +++ b/controllers/apps/components/util/component_utils.go @@ -47,16 +47,8 @@ var ( ErrReqClusterObj = errors.New("required arg *appsv1alpha1.Cluster is nil") ErrReqClusterComponentDefObj = errors.New("required arg *appsv1alpha1.ClusterComponentDefinition is nil") ErrReqClusterComponentSpecObj = errors.New("required arg *appsv1alpha1.ClusterComponentSpec is nil") - ErrNoOps = errors.New("no operation required") ) -func IgnoreNoOps(err error) error { - if err != nil && err != ErrNoOps { - return err - } - return nil -} - func ComponentRuntimeReqArgsCheck(cli client.Client, cluster *appsv1alpha1.Cluster, component *appsv1alpha1.ClusterComponentSpec) error { @@ -90,13 +82,6 @@ func GetClusterByObject(ctx context.Context, return cluster, nil } -// IsCompleted checks whether the component has completed the operation -// -// Deprecated: should use appsv1alpha1.ClusterStatus.GetTerminalPhase() -func IsCompleted(phase appsv1alpha1.ClusterComponentPhase) bool { - return slices.Index(appsv1alpha1.GetComponentTerminalPhases(), phase) != -1 -} - func IsFailedOrAbnormal(phase appsv1alpha1.ClusterComponentPhase) bool { return slices.Index([]appsv1alpha1.ClusterComponentPhase{ appsv1alpha1.FailedClusterCompPhase, diff --git a/controllers/apps/components/util/component_utils_test.go b/controllers/apps/components/util/component_utils_test.go index 095e11a8f..75139da23 100644 --- a/controllers/apps/components/util/component_utils_test.go +++ b/controllers/apps/components/util/component_utils_test.go @@ -36,19 +36,6 @@ import ( testk8s "github.com/apecloud/kubeblocks/internal/testutil/k8s" ) -func checkCompletedPhase(t *testing.T, phase appsv1alpha1.ClusterComponentPhase) { - isComplete := IsCompleted(phase) - if !isComplete { - t.Errorf("%s status is the completed status", phase) - } -} - -func TestIsCompleted(t *testing.T) { - checkCompletedPhase(t, appsv1alpha1.FailedClusterCompPhase) - checkCompletedPhase(t, appsv1alpha1.RunningClusterCompPhase) - checkCompletedPhase(t, appsv1alpha1.AbnormalClusterCompPhase) -} - func TestIsFailedOrAbnormal(t *testing.T) { if !IsFailedOrAbnormal(appsv1alpha1.AbnormalClusterCompPhase) { t.Error("isAbnormal should be true") @@ -217,7 +204,6 @@ var _ = Describe("Consensus Component", func() { By("test ComponentRuntimeReqArgsCheck function") err = ComponentRuntimeReqArgsCheck(k8sClient, cluster, clusterComp) Expect(err).Should(Succeed()) - Expect(IgnoreNoOps(err)).Should(Succeed()) By("test ComponentRuntimeReqArgsCheck function when cluster nil") err = ComponentRuntimeReqArgsCheck(k8sClient, nil, clusterComp) Expect(err).ShouldNot(Succeed()) diff --git a/controllers/apps/operations/horizontal_scaling.go b/controllers/apps/operations/horizontal_scaling.go index abb38130f..3f56d20d5 100644 --- a/controllers/apps/operations/horizontal_scaling.go +++ b/controllers/apps/operations/horizontal_scaling.go @@ -33,10 +33,12 @@ var _ OpsHandler = horizontalScalingOpsHandler{} func init() { horizontalScalingBehaviour := OpsBehaviour{ - // REVIEW: can do opsrequest if not running? - FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.HorizontalScalingPhase, - OpsHandler: horizontalScalingOpsHandler{}, + // if cluster is Abnormal or Failed, new opsRequest may can repair it. + // TODO: we should add "force" flag for these opsRequest. + FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + OpsHandler: horizontalScalingOpsHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonHorizontalScaling, } opsMgr := GetOpsManager() opsMgr.RegisterOps(appsv1alpha1.HorizontalScalingType, horizontalScalingBehaviour) diff --git a/controllers/apps/operations/ops_manager.go b/controllers/apps/operations/ops_manager.go index 3424c909f..9142b85f7 100644 --- a/controllers/apps/operations/ops_manager.go +++ b/controllers/apps/operations/ops_manager.go @@ -36,8 +36,9 @@ var ( func (opsMgr *OpsManager) RegisterOps(opsType appsv1alpha1.OpsType, opsBehaviour OpsBehaviour) { opsManager.OpsMap[opsType] = opsBehaviour appsv1alpha1.OpsRequestBehaviourMapper[opsType] = appsv1alpha1.OpsRequestBehaviour{ - FromClusterPhases: opsBehaviour.FromClusterPhases, - ToClusterPhase: opsBehaviour.ToClusterPhase, + FromClusterPhases: opsBehaviour.FromClusterPhases, + ToClusterPhase: opsBehaviour.ToClusterPhase, + ProcessingReasonInClusterCondition: opsBehaviour.ProcessingReasonInClusterCondition, } } @@ -81,7 +82,7 @@ func (opsMgr *OpsManager) Do(reqCtx intctrlutil.RequestCtx, cli client.Client, o // patch cluster.status after updating cluster.spec. // because cluster controller probably reconciles status.phase to Running if cluster is not updated. - return nil, patchClusterStatus(reqCtx, cli, opsRes, opsBehaviour) + return nil, patchClusterStatusAndRecordEvent(reqCtx, cli, opsRes, opsBehaviour) } // Reconcile entry function when OpsRequest.status.phase is Running. diff --git a/controllers/apps/operations/ops_progress_util.go b/controllers/apps/operations/ops_progress_util.go index e32181d9e..70b8c83fa 100644 --- a/controllers/apps/operations/ops_progress_util.go +++ b/controllers/apps/operations/ops_progress_util.go @@ -328,7 +328,8 @@ func handleFailedOrProcessingProgressDetail(opsRes *OpsResource, // podIsPendingDuringOperation checks if pod is pending during the component is doing operation. func podIsPendingDuringOperation(opsStartTime metav1.Time, pod *corev1.Pod, componentPhase appsv1alpha1.ClusterComponentPhase) bool { - return pod.CreationTimestamp.Before(&opsStartTime) && !util.IsCompleted(componentPhase) && pod.DeletionTimestamp.IsZero() + return pod.CreationTimestamp.Before(&opsStartTime) && pod.DeletionTimestamp.IsZero() && + !slices.Contains(appsv1alpha1.GetComponentTerminalPhases(), componentPhase) } // podIsFailedDuringOperation checks if pod is failed during operation. diff --git a/controllers/apps/operations/ops_util.go b/controllers/apps/operations/ops_util.go index 3f66092e9..a5879b1e0 100644 --- a/controllers/apps/operations/ops_util.go +++ b/controllers/apps/operations/ops_util.go @@ -216,21 +216,14 @@ func PatchValidateErrorCondition(ctx context.Context, cli client.Client, opsRes } // getOpsRequestNameFromAnnotation gets OpsRequest.name from cluster.annotations -func getOpsRequestNameFromAnnotation(cluster *appsv1alpha1.Cluster, toClusterPhase appsv1alpha1.ClusterPhase) string { +func getOpsRequestNameFromAnnotation(cluster *appsv1alpha1.Cluster, opsType appsv1alpha1.OpsType) *string { opsRequestSlice, _ := opsutil.GetOpsRequestSliceFromCluster(cluster) - opsRecorder := getOpsRecorderWithClusterPhase(opsRequestSlice, toClusterPhase) - return opsRecorder.Name -} - -// getOpsRecorderWithClusterPhase gets OpsRequest recorder from slice by target cluster phase -func getOpsRecorderWithClusterPhase(opsRequestSlice []appsv1alpha1.OpsRecorder, - toClusterPhase appsv1alpha1.ClusterPhase) appsv1alpha1.OpsRecorder { for _, v := range opsRequestSlice { - if v.ToClusterPhase == toClusterPhase { - return v + if v.Type == opsType { + return &v.Name } } - return appsv1alpha1.OpsRecorder{} + return nil } // GetOpsRecorderFromSlice gets OpsRequest recorder from slice by target cluster phase @@ -257,18 +250,23 @@ func patchOpsRequestToCreating(ctx context.Context, return PatchOpsStatusWithOpsDeepCopy(ctx, cli, opsRes, opsDeepCoy, appsv1alpha1.OpsCreatingPhase, validatePassCondition, condition) } -// patchClusterStatus updates Cluster.status to record cluster and components information if the ops need to maintain cluster status by self. -func patchClusterStatus(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource, opsBehaviour OpsBehaviour) error { - toClusterPhase := opsBehaviour.ToClusterPhase - if toClusterPhase == "" { - return nil - } - // if ops does not maintain the cluster phase by self, return - if !opsBehaviour.MaintainClusterPhaseBySelf { +// patchClusterStatusAndRecordEvent records the ops event in the cluster and +// will update the cluster status if the ops need to maintain cluster status by self. +func patchClusterStatusAndRecordEvent(reqCtx intctrlutil.RequestCtx, + cli client.Client, + opsRes *OpsResource, + opsBehaviour OpsBehaviour) error { + sendStartOpsRequestEvent := func() { + opsRes.Recorder.Eventf(opsRes.Cluster, corev1.EventTypeNormal, string(opsRes.OpsRequest.Spec.Type), + `Start to process the %s opsRequest "%s" in Cluster: %s`, opsRes.OpsRequest.Spec.Type, + opsRes.OpsRequest.Name, opsRes.Cluster.Name) + } + if opsBehaviour.ToClusterPhase == "" || !opsBehaviour.MaintainClusterPhaseBySelf { + sendStartOpsRequestEvent() return nil } patch := client.MergeFrom(opsRes.Cluster.DeepCopy()) - opsRes.Cluster.Status.Phase = toClusterPhase + opsRes.Cluster.Status.Phase = opsBehaviour.ToClusterPhase // update cluster.status.components phase realChangeCompMap := opsBehaviour.OpsHandler.GetRealAffectedComponentMap(opsRes.OpsRequest) for k := range realChangeCompMap { @@ -280,8 +278,7 @@ func patchClusterStatus(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes if err := cli.Status().Patch(reqCtx.Ctx, opsRes.Cluster, patch); err != nil { return err } - opsRes.Recorder.Eventf(opsRes.Cluster, corev1.EventTypeNormal, string(opsRes.OpsRequest.Spec.Type), - "Start %s in Cluster: %s", opsRes.OpsRequest.Spec.Type, opsRes.Cluster.Name) + sendStartOpsRequestEvent() return nil } @@ -301,9 +298,6 @@ func DeleteOpsRequestAnnotationInCluster(ctx context.Context, cli client.Client, } // delete the opsRequest information in Cluster.annotations opsRequestSlice = slices.Delete(opsRequestSlice, index, index+1) - if err = patchClusterPhaseWhenExistsOtherOps(ctx, cli, opsRes, opsRequestSlice); err != nil { - return err - } return opsutil.PatchClusterOpsAnnotations(ctx, cli, opsRes.Cluster, opsRequestSlice) } @@ -331,26 +325,12 @@ func addOpsRequestAnnotationToCluster(ctx context.Context, cli client.Client, op opsRequestSlice = make([]appsv1alpha1.OpsRecorder, 0) } opsRequestSlice = append(opsRequestSlice, appsv1alpha1.OpsRecorder{ - Name: opsRes.OpsRequest.Name, - ToClusterPhase: opsBehaviour.ToClusterPhase, + Name: opsRes.OpsRequest.Name, + Type: opsRes.OpsRequest.Spec.Type, }) return opsutil.UpdateClusterOpsAnnotations(ctx, cli, opsRes.Cluster, opsRequestSlice) } -// patchClusterPhaseWhenExistsOtherOps -func patchClusterPhaseWhenExistsOtherOps(ctx context.Context, cli client.Client, opsRes *OpsResource, opsRequestSlice []appsv1alpha1.OpsRecorder) error { - // If there are other OpsRequests running, modify the cluster.status.phase with other opsRequest's ToClusterPhase - if len(opsRequestSlice) == 0 { - return nil - } - patch := client.MergeFrom(opsRes.Cluster.DeepCopy()) - opsRes.Cluster.Status.Phase = opsRequestSlice[0].ToClusterPhase - if err := cli.Status().Patch(ctx, opsRes.Cluster, patch); err != nil { - return err - } - return nil -} - // isOpsRequestFailedPhase checks the OpsRequest phase is Failed func isOpsRequestFailedPhase(opsRequestPhase appsv1alpha1.OpsPhase) bool { return opsRequestPhase == appsv1alpha1.OpsFailedPhase diff --git a/controllers/apps/operations/ops_util_test.go b/controllers/apps/operations/ops_util_test.go index 4cb70afca..eb2352d3a 100644 --- a/controllers/apps/operations/ops_util_test.go +++ b/controllers/apps/operations/ops_util_test.go @@ -68,15 +68,6 @@ var _ = Describe("OpsUtil functions", func() { Expect(patchOpsHandlerNotSupported(ctx, k8sClient, opsRes)).Should(Succeed()) Expect(isOpsRequestFailedPhase(appsv1alpha1.OpsFailedPhase)).Should(BeTrue()) Expect(PatchClusterNotFound(ctx, k8sClient, opsRes)).Should(Succeed()) - opsRecorder := []appsv1alpha1.OpsRecorder{ - { - Name: "mysql-restart", - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, - }, - } - Expect(patchClusterPhaseWhenExistsOtherOps(ctx, k8sClient, opsRes, opsRecorder)).Should(Succeed()) - index, opsRecord := GetOpsRecorderFromSlice(opsRecorder, "mysql-restart") - Expect(index == 0 && opsRecord.Name == "mysql-restart").Should(BeTrue()) }) }) diff --git a/controllers/apps/operations/reconfigure.go b/controllers/apps/operations/reconfigure.go index 1cb633b26..78859c5ee 100644 --- a/controllers/apps/operations/reconfigure.go +++ b/controllers/apps/operations/reconfigure.go @@ -37,9 +37,10 @@ func init() { // REVIEW: can do opsrequest if not running? FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), // TODO: add cluster reconcile Reconfiguring phase. - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.ReconfiguringPhase, - MaintainClusterPhaseBySelf: true, - OpsHandler: &reAction, + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + MaintainClusterPhaseBySelf: true, + OpsHandler: &reAction, + ProcessingReasonInClusterCondition: ProcessingReasonReconfiguring, } cfgcore.ConfigEventHandlerMap["ops_status_reconfigure"] = &reAction opsManager.RegisterOps(appsv1alpha1.ReconfiguringType, reconfigureBehaviour) diff --git a/controllers/apps/operations/restart.go b/controllers/apps/operations/restart.go index daddd982a..d7a49f463 100644 --- a/controllers/apps/operations/restart.go +++ b/controllers/apps/operations/restart.go @@ -36,11 +36,13 @@ var _ OpsHandler = restartOpsHandler{} func init() { restartBehaviour := OpsBehaviour{ - // REVIEW: can do opsrequest if not running? - FromClusterPhases: appsv1alpha1.GetClusterTerminalPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.RebootingPhase, - OpsHandler: restartOpsHandler{}, - MaintainClusterPhaseBySelf: true, + // if cluster is Abnormal or Failed, new opsRequest may can repair it. + // TODO: we should add "force" flag for these opsRequest. + FromClusterPhases: appsv1alpha1.GetClusterTerminalPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.RebootingPhase, + OpsHandler: restartOpsHandler{}, + MaintainClusterPhaseBySelf: true, + ProcessingReasonInClusterCondition: ProcessingReasonRestarting, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/start.go b/controllers/apps/operations/start.go index 60fb1a228..36b0fdb2a 100644 --- a/controllers/apps/operations/start.go +++ b/controllers/apps/operations/start.go @@ -34,9 +34,10 @@ var _ OpsHandler = StartOpsHandler{} func init() { stopBehaviour := OpsBehaviour{ - FromClusterPhases: []appsv1alpha1.ClusterPhase{appsv1alpha1.StoppedClusterPhase}, - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.StartingPhase, - OpsHandler: StartOpsHandler{}, + FromClusterPhases: []appsv1alpha1.ClusterPhase{appsv1alpha1.StoppedClusterPhase}, + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.StartingPhase, + OpsHandler: StartOpsHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonStarting, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/stop.go b/controllers/apps/operations/stop.go index de166229d..00d027923 100644 --- a/controllers/apps/operations/stop.go +++ b/controllers/apps/operations/stop.go @@ -34,10 +34,10 @@ var _ OpsHandler = StopOpsHandler{} func init() { stopBehaviour := OpsBehaviour{ - // REVIEW: can do opsrequest if not running? - FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.StoppingPhase, - OpsHandler: StopOpsHandler{}, + FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + OpsHandler: StopOpsHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonStopping, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/type.go b/controllers/apps/operations/type.go index 2458390c4..2e5dab449 100644 --- a/controllers/apps/operations/type.go +++ b/controllers/apps/operations/type.go @@ -65,6 +65,11 @@ type OpsBehaviour struct { // StatefulSet/Deployment by Cluster controller and make pod to rebuilt, you need to maintain the cluster/component phase yourself. MaintainClusterPhaseBySelf bool + // ProcessingReasonInClusterCondition indicates the reason of the condition that type is "OpsRequestProcessed" in Cluster.Status.Conditions and + // is only valid when ToClusterPhase is not empty. it will indicate what operation the cluster is doing and + // will be displayed of "kblic cluster list". + ProcessingReasonInClusterCondition string + OpsHandler OpsHandler } @@ -87,3 +92,22 @@ type progressResource struct { clusterComponentDef *appsv1alpha1.ClusterComponentDefinition opsIsCompleted bool } + +const ( + // ProcessingReasonHorizontalScaling is the reason of the "OpsRequestProcessed" condition for the horizontal scaling opsRequest processing in cluster. + ProcessingReasonHorizontalScaling = "HorizontalScaling" + // ProcessingReasonVerticalScaling is the reason of the "OpsRequestProcessed" condition for the vertical scaling opsRequest processing in cluster. + ProcessingReasonVerticalScaling = "VerticalScaling" + // ProcessingReasonVolumeExpanding is the reason of the "OpsRequestProcessed" condition for the volume expansion opsRequest processing in cluster. + ProcessingReasonVolumeExpanding = "VolumeExpanding" + // ProcessingReasonStarting is the reason of the "OpsRequestProcessed" condition for the start opsRequest processing in cluster. + ProcessingReasonStarting = "Starting" + // ProcessingReasonStopping is the reason of the "OpsRequestProcessed" condition for the stop opsRequest processing in cluster. + ProcessingReasonStopping = "Stopping" + // ProcessingReasonRestarting is the reason of the "OpsRequestProcessed" condition for the restart opsRequest processing in cluster. + ProcessingReasonRestarting = "Restarting" + // ProcessingReasonReconfiguring is the reason of the "OpsRequestProcessed" condition for the reconfiguration opsRequest processing in cluster. + ProcessingReasonReconfiguring = "Reconfiguring" + // ProcessingReasonVersionUpgrading is the reason of the "OpsRequestProcessed" condition for the version upgrade opsRequest processing in cluster. + ProcessingReasonVersionUpgrading = "VersionUpgrading" +) diff --git a/controllers/apps/operations/upgrade.go b/controllers/apps/operations/upgrade.go index 0a7957416..3625df5e6 100644 --- a/controllers/apps/operations/upgrade.go +++ b/controllers/apps/operations/upgrade.go @@ -34,9 +34,12 @@ var _ OpsHandler = upgradeOpsHandler{} func init() { upgradeBehaviour := OpsBehaviour{ - FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.VersionUpgradingPhase, - OpsHandler: upgradeOpsHandler{}, + // if cluster is Abnormal or Failed, new opsRequest may can repair it. + // TODO: we should add "force" flag for these opsRequest. + FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + OpsHandler: upgradeOpsHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonVersionUpgrading, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/util/common_util.go b/controllers/apps/operations/util/common_util.go index a3d2db400..0db958277 100644 --- a/controllers/apps/operations/util/common_util.go +++ b/controllers/apps/operations/util/common_util.go @@ -25,7 +25,6 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" - componentutil "github.com/apecloud/kubeblocks/controllers/apps/components/util" intctrlutil "github.com/apecloud/kubeblocks/internal/constant" ) @@ -71,7 +70,7 @@ func PatchOpsRequestReconcileAnnotation(ctx context.Context, cli client.Client, opsRequest.Annotations = map[string]string{} } // because many changes may be triggered within one second, if the accuracy is only seconds, the event may be lost. - // so we used RFC3339Nano format. + // so use nanoseconds to record the time. opsRequest.Annotations[intctrlutil.ReconcileAnnotationKey] = time.Now().Format(time.RFC3339Nano) return cli.Patch(ctx, opsRequest, patch) } @@ -123,7 +122,7 @@ func MarkRunningOpsRequestAnnotation(ctx context.Context, cli client.Client, clu if len(notExistOps) != 0 { return RemoveClusterInvalidOpsRequestAnnotation(ctx, cli, cluster, opsRequestSlice, notExistOps) } - return componentutil.ErrNoOps + return nil } // RemoveClusterInvalidOpsRequestAnnotation deletes the OpsRequest annotation in cluster when the OpsRequest not existing. diff --git a/controllers/apps/operations/util/common_util_test.go b/controllers/apps/operations/util/common_util_test.go index fe4be8b45..93333ef87 100644 --- a/controllers/apps/operations/util/common_util_test.go +++ b/controllers/apps/operations/util/common_util_test.go @@ -70,12 +70,12 @@ var _ = Describe("OpsRequest Controller", func() { Expect(PatchOpsRequestReconcileAnnotation(ctx, k8sClient, cluster.Namespace, testOpsName)).Should(Succeed()) opsRecordSlice := []appsv1alpha1.OpsRecorder{ { - Name: testOpsName, - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + Name: testOpsName, + Type: appsv1alpha1.RestartType, }, { - Name: "not-exists-ops", - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + Name: "not-exists-ops", + Type: appsv1alpha1.RestartType, }, } Expect(PatchClusterOpsAnnotations(ctx, k8sClient, cluster, opsRecordSlice)).Should(Succeed()) diff --git a/controllers/apps/operations/vertical_scaling.go b/controllers/apps/operations/vertical_scaling.go index 42871e22f..dc286bcf3 100644 --- a/controllers/apps/operations/vertical_scaling.go +++ b/controllers/apps/operations/vertical_scaling.go @@ -33,9 +33,12 @@ var _ OpsHandler = verticalScalingHandler{} func init() { verticalScalingBehaviour := OpsBehaviour{ - FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.VerticalScalingPhase, - OpsHandler: verticalScalingHandler{}, + // if cluster is Abnormal or Failed, new opsRequest may can repair it. + // TODO: we should add "force" flag for these opsRequest. + FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + OpsHandler: verticalScalingHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonVerticalScaling, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/volume_expansion.go b/controllers/apps/operations/volume_expansion.go index a35a886a6..42ca55517 100644 --- a/controllers/apps/operations/volume_expansion.go +++ b/controllers/apps/operations/volume_expansion.go @@ -45,11 +45,11 @@ const ( func init() { // the volume expansion operation only support online expanding now, so this operation not affect the cluster availability. volumeExpansionBehaviour := OpsBehaviour{ - FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), - ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, // appsv1alpha1.VolumeExpandingPhase, - // TODO: add cluster reconcile VolumeExpanding phase. - MaintainClusterPhaseBySelf: true, - OpsHandler: volumeExpansionOpsHandler{}, + FromClusterPhases: appsv1alpha1.GetClusterUpRunningPhases(), + ToClusterPhase: appsv1alpha1.SpecReconcilingClusterPhase, + MaintainClusterPhaseBySelf: true, + OpsHandler: volumeExpansionOpsHandler{}, + ProcessingReasonInClusterCondition: ProcessingReasonVolumeExpanding, } opsMgr := GetOpsManager() diff --git a/controllers/apps/operations/volume_expansion_test.go b/controllers/apps/operations/volume_expansion_test.go index f29fcc540..4ed9f598e 100644 --- a/controllers/apps/operations/volume_expansion_test.go +++ b/controllers/apps/operations/volume_expansion_test.go @@ -82,17 +82,16 @@ var _ = Describe("OpsRequest Controller Volume Expansion Handler", func() { consensusCompName, "data").SetStorage("2Gi").SetStorageClass(storageClassName).Create(&testCtx) } - mockDoOperationOnCluster := func(cluster *appsv1alpha1.Cluster, opsRequestName string, toClusterPhase appsv1alpha1.ClusterPhase) { + mockDoOperationOnCluster := func(cluster *appsv1alpha1.Cluster, opsRequestName string, opsType appsv1alpha1.OpsType) { Expect(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(cluster), func(tmpCluster *appsv1alpha1.Cluster) { if tmpCluster.Annotations == nil { tmpCluster.Annotations = map[string]string{} } - tmpCluster.Annotations[constant.OpsRequestAnnotationKey] = fmt.Sprintf(`[{"clusterPhase": "%s", "name":"%s"}]`, toClusterPhase, opsRequestName) + tmpCluster.Annotations[constant.OpsRequestAnnotationKey] = fmt.Sprintf(`[{"type": "%s", "name":"%s"}]`, opsType, opsRequestName) })()).ShouldNot(HaveOccurred()) Eventually(testapps.CheckObj(&testCtx, client.ObjectKeyFromObject(cluster), func(g Gomega, myCluster *appsv1alpha1.Cluster) { - g.Expect(getOpsRequestNameFromAnnotation(myCluster, appsv1alpha1.SpecReconcilingClusterPhase)).ShouldNot(BeEmpty()) // appsv1alpha1.VolumeExpandingPhase - // TODO: add status condition expect for appsv1alpha1.VolumeExpandingPhase + g.Expect(getOpsRequestNameFromAnnotation(myCluster, appsv1alpha1.VolumeExpansionType)).ShouldNot(BeNil()) })).Should(Succeed()) } @@ -117,8 +116,7 @@ var _ = Describe("OpsRequest Controller Volume Expansion Handler", func() { ops = testapps.CreateOpsRequest(ctx, testCtx, ops) By("mock do operation on cluster") - mockDoOperationOnCluster(clusterObject, ops.Name, appsv1alpha1.SpecReconcilingClusterPhase) // appsv1alpha1.VolumeExpandingPhase - // TODO: add status condition expect for appsv1alpha1.VolumeExpandingPhase + mockDoOperationOnCluster(clusterObject, ops.Name, appsv1alpha1.VolumeExpansionType) // create-pvc pvcName := fmt.Sprintf("%s-%s-%s-%d", vctName, clusterObject.Name, consensusCompName, index) @@ -143,8 +141,7 @@ var _ = Describe("OpsRequest Controller Volume Expansion Handler", func() { opsRes.OpsRequest = newOps _, err := GetOpsManager().Reconcile(reqCtx, k8sClient, opsRes) Expect(err == nil).Should(BeTrue()) - Eventually(testapps.GetOpsRequestCompPhase(ctx, testCtx, newOps.Name, consensusCompName)).Should(Equal(appsv1alpha1.SpecReconcilingClusterCompPhase)) // VolumeExpandingPhase - // TODO: add status condition expect for VolumeExpandingPhase + Eventually(testapps.GetOpsRequestCompPhase(ctx, testCtx, newOps.Name, consensusCompName)).Should(Equal(appsv1alpha1.SpecReconcilingClusterCompPhase)) } testWarningEventOnPVC := func(reqCtx intctrlutil.RequestCtx, clusterObject *appsv1alpha1.Cluster, opsRes *OpsResource) { @@ -170,8 +167,7 @@ var _ = Describe("OpsRequest Controller Volume Expansion Handler", func() { event.InvolvedObject = stsInvolvedObject pvcEventHandler := PersistentVolumeClaimEventHandler{} Expect(pvcEventHandler.Handle(k8sClient, reqCtx, eventRecorder, event)).Should(Succeed()) - Eventually(testapps.GetOpsRequestCompPhase(ctx, testCtx, newOps.Name, consensusCompName)).Should(Equal(appsv1alpha1.SpecReconcilingClusterCompPhase)) // VolumeExpandingPhase - // TODO: add status condition expect for VolumeExpandingPhase + Eventually(testapps.GetOpsRequestCompPhase(ctx, testCtx, newOps.Name, consensusCompName)).Should(Equal(appsv1alpha1.SpecReconcilingClusterCompPhase)) // test when the event reach the conditions event.Count = 5 @@ -244,8 +240,7 @@ var _ = Describe("OpsRequest Controller Volume Expansion Handler", func() { // init resources for volume expansion newOps, pvcName := initResourcesForVolumeExpansion(clusterObject, opsRes, 2) Expect(testapps.ChangeObjStatus(&testCtx, clusterObject, func() { - clusterObject.Status.Phase = appsv1alpha1.SpecReconcilingClusterPhase // appsv1alpha1.VolumeExpandingPhase - // TODO: add status condition for VolumeExpandingPhase + clusterObject.Status.Phase = appsv1alpha1.SpecReconcilingClusterPhase })).ShouldNot(HaveOccurred()) Expect(k8sClient.Delete(ctx, newOps)).Should(Succeed()) Eventually(func() error { diff --git a/controllers/apps/operations/volume_expansion_updater.go b/controllers/apps/operations/volume_expansion_updater.go index e65087353..7b873afd2 100644 --- a/controllers/apps/operations/volume_expansion_updater.go +++ b/controllers/apps/operations/volume_expansion_updater.go @@ -60,18 +60,18 @@ func handleVolumeExpansionWithPVC(reqCtx intctrlutil.RequestCtx, cli client.Clie return err } // check whether the cluster is expanding volume - opsRequestName := getOpsRequestNameFromAnnotation(cluster, appsv1alpha1.SpecReconcilingClusterPhase) - if opsRequestName == "" { + opsRequestName := getOpsRequestNameFromAnnotation(cluster, appsv1alpha1.VolumeExpansionType) + if opsRequestName == nil { return nil } // notice the OpsRequest to reconcile - err := opsutil.PatchOpsRequestReconcileAnnotation(reqCtx.Ctx, cli, cluster.Namespace, opsRequestName) + err := opsutil.PatchOpsRequestReconcileAnnotation(reqCtx.Ctx, cli, cluster.Namespace, *opsRequestName) // if the OpsRequest is not found, means it is deleted by user. // we should delete the invalid OpsRequest annotation in the cluster and reconcile the cluster phase. if apierrors.IsNotFound(err) { opsRequestSlice, _ := opsutil.GetOpsRequestSliceFromCluster(cluster) notExistOps := map[string]struct{}{ - opsRequestName: {}, + *opsRequestName: {}, } if err = opsutil.RemoveClusterInvalidOpsRequestAnnotation(reqCtx.Ctx, cli, cluster, opsRequestSlice, notExistOps); err != nil { @@ -158,12 +158,12 @@ func (pvcEventHandler PersistentVolumeClaimEventHandler) handlePVCFailedStatusOn return err } // get the volume expansion ops which is running on cluster. - opsRequestName := getOpsRequestNameFromAnnotation(cluster, appsv1alpha1.SpecReconcilingClusterPhase) - if opsRequestName == "" { + opsRequestName := getOpsRequestNameFromAnnotation(cluster, appsv1alpha1.VolumeExpansionType) + if opsRequestName == nil { return nil } opsRequest := &appsv1alpha1.OpsRequest{} - if err = cli.Get(reqCtx.Ctx, client.ObjectKey{Name: opsRequestName, Namespace: pvc.Namespace}, opsRequest); err != nil { + if err = cli.Get(reqCtx.Ctx, client.ObjectKey{Name: *opsRequestName, Namespace: pvc.Namespace}, opsRequest); err != nil { return err } compsStatus := opsRequest.Status.Components diff --git a/controllers/apps/opsrequest_controller_test.go b/controllers/apps/opsrequest_controller_test.go index f520fa819..e4751d322 100644 --- a/controllers/apps/opsrequest_controller_test.go +++ b/controllers/apps/opsrequest_controller_test.go @@ -25,6 +25,7 @@ import ( "github.com/spf13/viper" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" @@ -33,6 +34,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" opsutil "github.com/apecloud/kubeblocks/controllers/apps/operations/util" "github.com/apecloud/kubeblocks/internal/constant" + "github.com/apecloud/kubeblocks/internal/controller/lifecycle" intctrlutil "github.com/apecloud/kubeblocks/internal/generics" testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" testk8s "github.com/apecloud/kubeblocks/internal/testutil/k8s" @@ -81,6 +83,24 @@ var _ = Describe("OpsRequest Controller", func() { // Testcases + checkLatestOpsIsProcessing := func(clusterKey client.ObjectKey, opsType appsv1alpha1.OpsType) { + Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, fetched *appsv1alpha1.Cluster) { + con := meta.FindStatusCondition(fetched.Status.Conditions, appsv1alpha1.ConditionTypeLatestOpsRequestProcessed) + g.Expect(con).ShouldNot(BeNil()) + g.Expect(con.Status).Should(Equal(metav1.ConditionFalse)) + g.Expect(con.Reason).Should(Equal(appsv1alpha1.OpsRequestBehaviourMapper[opsType].ProcessingReasonInClusterCondition)) + })).Should(Succeed()) + } + + checkLatestOpsHasProcessed := func(clusterKey client.ObjectKey) { + Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, fetched *appsv1alpha1.Cluster) { + con := meta.FindStatusCondition(fetched.Status.Conditions, appsv1alpha1.ConditionTypeLatestOpsRequestProcessed) + g.Expect(con).ShouldNot(BeNil()) + g.Expect(con.Status).Should(Equal(metav1.ConditionTrue)) + g.Expect(con.Reason).Should(Equal(lifecycle.ReasonOpsRequestProcessed)) + })).Should(Succeed()) + } + mockSetClusterStatusPhaseToRunning := func(namespacedName types.NamespacedName) { Expect(testapps.GetAndChangeObjStatus(&testCtx, namespacedName, func(fetched *appsv1alpha1.Cluster) { @@ -127,7 +147,7 @@ var _ = Describe("OpsRequest Controller", func() { By("Waiting for the cluster enter running phase") Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) - Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) By("mock pod/sts are available and wait for cluster enter running phase") podName := fmt.Sprintf("%s-%s-0", clusterObj.Name, mysqlCompName) @@ -167,6 +187,7 @@ var _ = Describe("OpsRequest Controller", func() { Eventually(testapps.GetOpsRequestPhase(&testCtx, opsKey)).Should(Equal(appsv1alpha1.OpsRunningPhase)) Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.SpecReconcilingClusterPhase)) Eventually(testapps.GetClusterComponentPhase(testCtx, clusterObj.Name, mysqlCompName)).Should(Equal(appsv1alpha1.SpecReconcilingClusterCompPhase)) + checkLatestOpsIsProcessing(clusterKey, verticalScalingOpsRequest.Spec.Type) By("check Cluster and changed component phase is VerticalScaling") Eventually(testapps.CheckObj(&testCtx, clusterKey, func(g Gomega, cluster *appsv1alpha1.Cluster) { @@ -180,6 +201,7 @@ var _ = Describe("OpsRequest Controller", func() { })()).ShouldNot(HaveOccurred()) Eventually(testapps.GetClusterComponentPhase(testCtx, clusterObj.Name, mysqlCompName)).Should(Equal(appsv1alpha1.RunningClusterCompPhase)) Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.RunningClusterPhase)) + checkLatestOpsHasProcessed(clusterKey) By("patch opsrequest controller to run") Expect(testapps.ChangeObj(&testCtx, verticalScalingOpsRequest, func() { @@ -304,9 +326,9 @@ var _ = Describe("OpsRequest Controller", func() { Expect(testCtx.CreateObj(testCtx.Ctx, ops)).Should(Succeed()) By("expect component is Running if don't support volume snapshot during doing h-scale ops") + Eventually(testapps.GetOpsRequestPhase(&testCtx, opsKey)).Should(Equal(appsv1alpha1.OpsRunningPhase)) // cluster phase changes to HorizontalScalingPhase first. then, it will be ConditionsError because it does not support snapshot backup after a period of time. Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.SpecReconcilingClusterPhase)) // HorizontalScalingPhase - Eventually(testapps.GetOpsRequestPhase(&testCtx, opsKey)).Should(Equal(appsv1alpha1.OpsRunningPhase)) Eventually(testapps.GetClusterComponentPhase(testCtx, clusterKey.Name, mysqlCompName)).Should(Equal(appsv1alpha1.RunningClusterCompPhase)) By("delete h-scale ops") @@ -395,7 +417,7 @@ var _ = Describe("OpsRequest Controller", func() { By("should be Running before pods are not deleted successfully") Eventually(testapps.GetOpsRequestPhase(&testCtx, opsKey)).Should(Equal(appsv1alpha1.OpsRunningPhase)) - + checkLatestOpsIsProcessing(clusterKey, stopOps.Spec.Type) // mock pod deleted successfully for _, pod := range podList { Expect(testapps.ChangeObj(&testCtx, pod, func() { @@ -410,6 +432,7 @@ var _ = Describe("OpsRequest Controller", func() { } })).ShouldNot(HaveOccurred()) Eventually(testapps.GetOpsRequestPhase(&testCtx, opsKey)).Should(Equal(appsv1alpha1.OpsSucceedPhase)) + checkLatestOpsHasProcessed(clusterKey) By("test start ops") startOpsName := "start-ops" + testCtx.GetRandomStr() diff --git a/controllers/apps/systemaccount_controller.go b/controllers/apps/systemaccount_controller.go index c77f9bae1..21d002727 100644 --- a/controllers/apps/systemaccount_controller.go +++ b/controllers/apps/systemaccount_controller.go @@ -157,8 +157,8 @@ func (r *SystemAccountReconciler) Reconcile(ctx context.Context, req ctrl.Reques } // wait till the cluster is running - if cluster.Status.Phase != appsv1alpha1.RunningClusterPhase && cluster.Status.Phase != appsv1alpha1.StartingClusterPhase { - reqCtx.Log.Info("Cluster is not ready yet", "cluster", req.NamespacedName) + if cluster.Status.Phase != appsv1alpha1.RunningClusterPhase { + reqCtx.Log.V(1).Info("Cluster is not ready yet", "cluster", req.NamespacedName) return intctrlutil.Reconciled() } diff --git a/controllers/apps/tls_utils_test.go b/controllers/apps/tls_utils_test.go index cd57663fc..6c02e6b30 100644 --- a/controllers/apps/tls_utils_test.go +++ b/controllers/apps/tls_utils_test.go @@ -135,7 +135,7 @@ var _ = Describe("TLS self-signed cert function", func() { // // By("Waiting for the cluster enter creating phase") // Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) - // Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.StartingClusterPhase)) + // Eventually(testapps.GetClusterPhase(&testCtx, clusterKey)).Should(Equal(appsv1alpha1.CreatingClusterPhase)) // // By("By inspect that TLS cert. secret") // ns := clusterObj.Namespace diff --git a/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml b/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml index fa070bd75..a6b1eafec 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml @@ -789,27 +789,24 @@ spec: type: object phase: description: 'phase describes the phase of the component, the - detail information of the phases are as following: Failed: - component is unavailable, i.e, all pods are not ready for - Stateless/Stateful component; Leader/Primary pod is not ready - for Consensus/Replication component. Running: component is - running. [terminal state] Stopped: component is stopped, as - no running pod. [terminal state] Failed: component has failed - to start running. [terminal state] Abnormal: component is - running but part of its pods are not ready. [terminal state] - Starting: component has entered starting process. Updating: + detail information of the phases are as following: Running: + component is running. [terminal state] Stopped: component + is stopped, as no running pod. [terminal state] Failed: component + is unavailable. i.e, all pods are not ready for Stateless/Stateful + component, Leader/Primary pod is not ready for Consensus/Replication + component. [terminal state] Abnormal: component is running + but part of its pods are not ready. Leader/Primary pod is + ready for Consensus/Replication component. [terminal state] + Creating: component has entered creating process. Updating: component has entered updating process, triggered by Spec. - updated. Stopping: component has entered a stopping process. - If the component workload type is Consensus/Replication, the - Leader/Primary pod must be ready in Abnormal phase.' + updated.' enum: - Running - Stopped - Failed - Abnormal + - Creating - Updating - - Starting - - Stopping type: string podsReady: description: podsReady checks if all pods of the component are @@ -940,19 +937,16 @@ spec: cluster has stopped, all its components are stopped. [terminal state] Failed: cluster is unavailable. [terminal state] Abnormal: Cluster is still running, but part of its components are Abnormal/Failed. - [terminal state] Starting: Cluster has entered starting process. + [terminal state] Creating: Cluster has entered creating process. Updating: Cluster has entered updating process, triggered by Spec. - updated. Stopping: Cluster has entered a stopping process. if the - component workload type is Consensus/Replication, the Leader/Primary - pod must be ready in Abnormal phase.' + updated.' enum: - Running - Stopped - Failed - Abnormal - - Starting + - Creating - Updating - - Stopping type: string type: object type: object diff --git a/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml b/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml index 507dae163..7101384a0 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml @@ -380,9 +380,8 @@ spec: - Stopped - Failed - Abnormal + - Creating - Updating - - Starting - - Stopping type: string progressDetails: description: progressDetails describes the progress details diff --git a/internal/cli/cluster/cluster.go b/internal/cli/cluster/cluster.go index 6709478c3..49fd9cab1 100644 --- a/internal/cli/cluster/cluster.go +++ b/internal/cli/cluster/cluster.go @@ -22,6 +22,7 @@ import ( "strings" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" @@ -89,6 +90,12 @@ func (o *ObjectsGetter) Get() (*ClusterObjects, error) { return nil, err } + // wrap the cluster phase if the latest ops request is processing + latestOpsProcessedCondition := meta.FindStatusCondition(objs.Cluster.Status.Conditions, appsv1alpha1.ConditionTypeLatestOpsRequestProcessed) + if latestOpsProcessedCondition != nil && latestOpsProcessedCondition.Status == metav1.ConditionFalse { + objs.Cluster.Status.Phase = appsv1alpha1.ClusterPhase(latestOpsProcessedCondition.Reason) + } + // get cluster definition if o.WithClusterDef { cd := &appsv1alpha1.ClusterDefinition{} diff --git a/internal/cli/cmd/cluster/list_ops_test.go b/internal/cli/cmd/cluster/list_ops_test.go index 282528ff8..97918f6bd 100644 --- a/internal/cli/cmd/cluster/list_ops_test.go +++ b/internal/cli/cmd/cluster/list_ops_test.go @@ -82,27 +82,21 @@ var _ = Describe("Expose", func() { } initOpsRequests := func() { - opsTypes := []appsv1alpha1.OpsType{ - appsv1alpha1.UpgradeType, - appsv1alpha1.HorizontalScalingType, - appsv1alpha1.HorizontalScalingType, - appsv1alpha1.RestartType, - appsv1alpha1.VerticalScalingType, - appsv1alpha1.VerticalScalingType, - appsv1alpha1.VerticalScalingType, + opsKeys := []struct { + opsType appsv1alpha1.OpsType + phase appsv1alpha1.OpsPhase + }{ + {appsv1alpha1.UpgradeType, appsv1alpha1.OpsPendingPhase}, + {appsv1alpha1.HorizontalScalingType, appsv1alpha1.OpsFailedPhase}, + {appsv1alpha1.HorizontalScalingType, appsv1alpha1.OpsSucceedPhase}, + {appsv1alpha1.RestartType, appsv1alpha1.OpsSucceedPhase}, + {appsv1alpha1.VerticalScalingType, appsv1alpha1.OpsRunningPhase}, + {appsv1alpha1.VerticalScalingType, appsv1alpha1.OpsFailedPhase}, + {appsv1alpha1.VerticalScalingType, appsv1alpha1.OpsRunningPhase}, } - phases := []appsv1alpha1.OpsPhase{ - appsv1alpha1.OpsPendingPhase, - appsv1alpha1.OpsFailedPhase, - appsv1alpha1.OpsSucceedPhase, - appsv1alpha1.OpsSucceedPhase, - appsv1alpha1.OpsRunningPhase, - appsv1alpha1.OpsFailedPhase, - appsv1alpha1.OpsRunningPhase, - } - opsList := make([]runtime.Object, len(opsTypes)) - for i := range opsTypes { - opsList[i] = generateOpsObject(opsTypes[i], phases[i]) + opsList := make([]runtime.Object, len(opsKeys)) + for i := range opsKeys { + opsList[i] = generateOpsObject(opsKeys[i].opsType, opsKeys[i].phase) } opsName = opsList[0].(*appsv1alpha1.OpsRequest).Name tf.FakeDynamicClient = clitesting.FakeDynamicClient(opsList...) diff --git a/internal/cli/cmd/cluster/list_test.go b/internal/cli/cmd/cluster/list_test.go index f00d88386..fc8679273 100644 --- a/internal/cli/cmd/cluster/list_test.go +++ b/internal/cli/cmd/cluster/list_test.go @@ -24,6 +24,7 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/cli-runtime/pkg/genericclioptions" @@ -45,8 +46,10 @@ var _ = Describe("list", func() { ) const ( - namespace = "test" - clusterName = "test" + namespace = "test" + clusterName = "test" + clusterName1 = "test1" + verticalScalingReason = "VerticalScaling" ) BeforeEach(func() { @@ -56,6 +59,11 @@ var _ = Describe("list", func() { _ = appsv1alpha1.AddToScheme(scheme.Scheme) codec := scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...) cluster := testing.FakeCluster(clusterName, namespace) + clusterWithCondition := testing.FakeCluster(clusterName1, namespace, metav1.Condition{ + Type: appsv1alpha1.ConditionTypeLatestOpsRequestProcessed, + Status: metav1.ConditionFalse, + Reason: verticalScalingReason, + }) pods := testing.FakePods(3, namespace, clusterName) httpResp := func(obj runtime.Object) *http.Response { return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, obj)} @@ -67,20 +75,21 @@ var _ = Describe("list", func() { Client: clientfake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { urlPrefix := "/api/v1/namespaces/" + namespace return map[string]*http.Response{ - "/namespaces/" + namespace + "/clusters": httpResp(&appsv1alpha1.ClusterList{Items: []appsv1alpha1.Cluster{*cluster}}), - "/namespaces/" + namespace + "/clusters/test": httpResp(cluster), - "/namespaces/" + namespace + "/secrets": httpResp(testing.FakeSecrets(namespace, clusterName)), - "/api/v1/nodes/" + testing.NodeName: httpResp(testing.FakeNode()), - urlPrefix + "/services": httpResp(&corev1.ServiceList{}), - urlPrefix + "/secrets": httpResp(testing.FakeSecrets(namespace, clusterName)), - urlPrefix + "/pods": httpResp(pods), - urlPrefix + "/events": httpResp(testing.FakeEvents()), + "/namespaces/" + namespace + "/clusters": httpResp(&appsv1alpha1.ClusterList{Items: []appsv1alpha1.Cluster{*cluster}}), + "/namespaces/" + namespace + "/clusters/" + clusterName: httpResp(cluster), + "/namespaces/" + namespace + "/clusters/" + clusterName1: httpResp(clusterWithCondition), + "/namespaces/" + namespace + "/secrets": httpResp(testing.FakeSecrets(namespace, clusterName)), + "/api/v1/nodes/" + testing.NodeName: httpResp(testing.FakeNode()), + urlPrefix + "/services": httpResp(&corev1.ServiceList{}), + urlPrefix + "/secrets": httpResp(testing.FakeSecrets(namespace, clusterName)), + urlPrefix + "/pods": httpResp(pods), + urlPrefix + "/events": httpResp(testing.FakeEvents()), }[req.URL.Path], nil }), } tf.Client = tf.UnstructuredClient - tf.FakeDynamicClient = testing.FakeDynamicClient(cluster, testing.FakeClusterDef(), testing.FakeClusterVersion()) + tf.FakeDynamicClient = testing.FakeDynamicClient(cluster, clusterWithCondition, testing.FakeClusterDef(), testing.FakeClusterVersion()) }) AfterEach(func() { @@ -91,8 +100,11 @@ var _ = Describe("list", func() { cmd := NewListCmd(tf, streams) Expect(cmd).ShouldNot(BeNil()) - cmd.Run(cmd, []string{"test"}) + cmd.Run(cmd, []string{clusterName}) Expect(out.String()).Should(ContainSubstring(testing.ClusterDefName)) + + cmd.Run(cmd, []string{clusterName1}) + Expect(out.String()).Should(ContainSubstring(verticalScalingReason)) }) It("list instances", func() { diff --git a/internal/cli/testing/fake.go b/internal/cli/testing/fake.go index 5c459345b..9335333c5 100644 --- a/internal/cli/testing/fake.go +++ b/internal/cli/testing/fake.go @@ -58,7 +58,7 @@ func GetRandomStr() string { return seq } -func FakeCluster(name string, namespace string) *appsv1alpha1.Cluster { +func FakeCluster(name, namespace string, conditions ...metav1.Condition) *appsv1alpha1.Cluster { var replicas int32 = 1 return &appsv1alpha1.Cluster{ TypeMeta: metav1.TypeMeta{ @@ -82,6 +82,7 @@ func FakeCluster(name string, namespace string) *appsv1alpha1.Cluster { }, }, }, + Conditions: conditions, }, Spec: appsv1alpha1.ClusterSpec{ ClusterDefRef: ClusterDefName, diff --git a/internal/controller/lifecycle/cluster_plan_builder.go b/internal/controller/lifecycle/cluster_plan_builder.go index caa95b8bd..864c306ec 100644 --- a/internal/controller/lifecycle/cluster_plan_builder.go +++ b/internal/controller/lifecycle/cluster_plan_builder.go @@ -25,6 +25,8 @@ import ( appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/tools/record" ctrl "sigs.k8s.io/controller-runtime" @@ -33,7 +35,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" dataprotectionv1alpha1 "github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1" - "github.com/apecloud/kubeblocks/controllers/apps/components/util" + opsutil "github.com/apecloud/kubeblocks/controllers/apps/operations/util" "github.com/apecloud/kubeblocks/internal/constant" types2 "github.com/apecloud/kubeblocks/internal/controller/client" "github.com/apecloud/kubeblocks/internal/controller/graph" @@ -42,20 +44,22 @@ import ( // clusterPlanBuilder a graph.PlanBuilder implementation for Cluster reconciliation type clusterPlanBuilder struct { - ctx intctrlutil.RequestCtx - cli client.Client - req ctrl.Request - recorder record.EventRecorder - cluster *appsv1alpha1.Cluster - conMgr clusterConditionManager2 + ctx intctrlutil.RequestCtx + cli client.Client + req ctrl.Request + recorder record.EventRecorder + cluster *appsv1alpha1.Cluster + originCluster appsv1alpha1.Cluster } // clusterPlan a graph.Plan implementation for Cluster reconciliation type clusterPlan struct { + ctx intctrlutil.RequestCtx + cli client.Client + recorder record.EventRecorder dag *graph.DAG walkFunc graph.WalkFunc cluster *appsv1alpha1.Cluster - conMgr clusterConditionManager2 } var _ graph.PlanBuilder = &clusterPlanBuilder{} @@ -67,19 +71,58 @@ func (c *clusterPlanBuilder) Init() error { return err } c.cluster = cluster + c.originCluster = *cluster.DeepCopy() + // handles the cluster phase and ops condition first to indicates what the current cluster is doing. + c.handleClusterPhase() + c.handleLatestOpsRequestProcessingCondition() return nil } +// updateClusterPhase handles the cluster phase and ops condition first to indicates what the current cluster is doing. +func (c *clusterPlanBuilder) handleClusterPhase() { + clusterPhase := c.cluster.Status.Phase + if isClusterUpdating(*c.cluster) { + if clusterPhase == "" { + c.cluster.Status.Phase = appsv1alpha1.CreatingClusterPhase + } else if clusterPhase != appsv1alpha1.CreatingClusterPhase { + c.cluster.Status.Phase = appsv1alpha1.SpecReconcilingClusterPhase + } + } +} + +// updateLatestOpsRequestProcessingCondition handles the latest opsRequest processing condition. +func (c *clusterPlanBuilder) handleLatestOpsRequestProcessingCondition() { + opsRecords, _ := opsutil.GetOpsRequestSliceFromCluster(c.cluster) + if len(opsRecords) == 0 { + return + } + ops := opsRecords[0] + opsBehaviour, ok := appsv1alpha1.OpsRequestBehaviourMapper[ops.Type] + if !ok { + return + } + opsCondition := newOpsRequestProcessingCondition(ops.Name, string(ops.Type), opsBehaviour.ProcessingReasonInClusterCondition) + oldCondition := meta.FindStatusCondition(c.cluster.Status.Conditions, opsCondition.Type) + if oldCondition == nil { + // if this condition not exists, insert it to the first position. + opsCondition.LastTransitionTime = metav1.Now() + c.cluster.Status.Conditions = append([]metav1.Condition{opsCondition}, c.cluster.Status.Conditions...) + } else { + meta.SetStatusCondition(&c.cluster.Status.Conditions, opsCondition) + } +} + func (c *clusterPlanBuilder) Validate() error { + var err error + defer func() { + if err != nil { + _ = c.updateClusterStatusWithCondition(newFailedProvisioningStartedCondition(err.Error(), ReasonPreCheckFailed)) + } + }() + validateExistence := func(key client.ObjectKey, object client.Object) error { - err := c.cli.Get(c.ctx.Ctx, key, object) + err = c.cli.Get(c.ctx.Ctx, key, object) if err != nil { - if apierrors.IsNotFound(err) { - if setErr := c.conMgr.setPreCheckErrorCondition(c.cluster, err); util.IgnoreNoOps(setErr) != nil { - return setErr - } - c.recorder.Eventf(c.cluster, corev1.EventTypeWarning, constant.ReasonNotFoundCR, err.Error()) - } return newRequeueError(requeueDuration, err.Error()) } return nil @@ -87,13 +130,13 @@ func (c *clusterPlanBuilder) Validate() error { // validate cd & cv existences cd := &appsv1alpha1.ClusterDefinition{} - if err := validateExistence(types.NamespacedName{Name: c.cluster.Spec.ClusterDefRef}, cd); err != nil { + if err = validateExistence(types.NamespacedName{Name: c.cluster.Spec.ClusterDefRef}, cd); err != nil { return err } var cv *appsv1alpha1.ClusterVersion if len(c.cluster.Spec.ClusterVersionRef) > 0 { cv = &appsv1alpha1.ClusterVersion{} - if err := validateExistence(types.NamespacedName{Name: c.cluster.Spec.ClusterVersionRef}, cv); err != nil { + if err = validateExistence(types.NamespacedName{Name: c.cluster.Spec.ClusterVersionRef}, cv); err != nil { return err } } @@ -101,11 +144,7 @@ func (c *clusterPlanBuilder) Validate() error { // validate cd & cv availability if cd.Status.Phase != appsv1alpha1.AvailablePhase || (cv != nil && cv.Status.Phase != appsv1alpha1.AvailablePhase) { message := fmt.Sprintf("ref resource is unavailable, this problem needs to be solved first. cd: %v, cv: %v", cd, cv) - if err := c.conMgr.setReferenceCRUnavailableCondition(c.cluster, message); util.IgnoreNoOps(err) != nil { - if apierrors.IsNotFound(err) { - return nil - } - } + err = errors.New(message) return newRequeueError(requeueDuration, message) } @@ -114,21 +153,31 @@ func (c *clusterPlanBuilder) Validate() error { chain := &graph.ValidatorChain{ &enableLogsValidator{cluster: c.cluster, clusterDef: cd}, } - if err := chain.WalkThrough(); err != nil { - _ = c.conMgr.setPreCheckErrorCondition(c.cluster, err) + if err = chain.WalkThrough(); err != nil { return newRequeueError(requeueDuration, err.Error()) } return nil } +func (c *clusterPlanBuilder) handleProvisionStartedCondition() { + // set provisioning cluster condition + condition := newProvisioningStartedCondition(c.cluster.Name, c.cluster.Generation) + oldCondition := meta.FindStatusCondition(c.cluster.Status.Conditions, condition.Type) + if conditionIsChanged(oldCondition, condition) { + meta.SetStatusCondition(&c.cluster.Status.Conditions, condition) + c.recorder.Event(c.cluster, corev1.EventTypeNormal, condition.Reason, condition.Message) + } +} + // Build only cluster Creation, Update and Deletion supported. func (c *clusterPlanBuilder) Build() (graph.Plan, error) { - _ = c.conMgr.setProvisioningStartedCondition(c.cluster) + // set provisioning cluster condition + c.handleProvisionStartedCondition() var err error defer func() { if err != nil { - _ = c.conMgr.setApplyResourcesFailedCondition(c.cluster, err) + _ = c.updateClusterStatusWithCondition(newFailedApplyResourcesCondition(err.Error())) } }() @@ -143,7 +192,7 @@ func (c *clusterPlanBuilder) Build() (graph.Plan, error) { // build transformer chain chain := &graph.TransformerChain{ // init dag, that is put cluster vertex into dag - &initTransformer{cluster: c.cluster}, + &initTransformer{cluster: c.cluster, originCluster: &c.originCluster}, // fix cd&cv labels of cluster &fixClusterLabelsTransformer{}, // cluster to K8s objects and put them into dag @@ -167,7 +216,7 @@ func (c *clusterPlanBuilder) Build() (graph.Plan, error) { // replication set horizontal scaling &rplSetHorizontalScalingTransformer{cr: *cr, cli: c.cli, ctx: c.ctx}, // finally, update cluster status - &clusterStatusTransformer{cc: *cr, cli: c.cli, ctx: c.ctx, recorder: c.recorder, conMgr: c.conMgr}, + newClusterStatusTransformer(c.ctx, c.cli, c.recorder, *cr), } // new a DAG and apply chain on it, after that we should get the final Plan @@ -179,39 +228,72 @@ func (c *clusterPlanBuilder) Build() (graph.Plan, error) { c.ctx.Log.Info(fmt.Sprintf("DAG: %s", dag)) // we got the execution plan plan := &clusterPlan{ + ctx: c.ctx, + cli: c.cli, + recorder: c.recorder, dag: dag, walkFunc: c.defaultWalkFunc, cluster: c.cluster, - conMgr: c.conMgr, } return plan, nil } +func (c *clusterPlanBuilder) updateClusterStatusWithCondition(condition metav1.Condition) error { + oldCondition := meta.FindStatusCondition(c.cluster.Status.Conditions, condition.Type) + meta.SetStatusCondition(&c.cluster.Status.Conditions, condition) + if !reflect.DeepEqual(c.cluster.Status, c.originCluster.Status) { + if err := c.cli.Status().Patch(c.ctx.Ctx, c.cluster, client.MergeFrom(c.originCluster.DeepCopy())); err != nil { + return err + } + } + // Normal events are only sent once. + if !conditionIsChanged(oldCondition, condition) && condition.Status == metav1.ConditionTrue { + return nil + } + eventType := corev1.EventTypeWarning + if condition.Status == metav1.ConditionTrue { + eventType = corev1.EventTypeNormal + } + c.recorder.Event(c.cluster, eventType, condition.Reason, condition.Message) + return nil +} + // NewClusterPlanBuilder returns a clusterPlanBuilder powered PlanBuilder // TODO: change ctx to context.Context func NewClusterPlanBuilder(ctx intctrlutil.RequestCtx, cli client.Client, req ctrl.Request, recorder record.EventRecorder) graph.PlanBuilder { - conMgr := clusterConditionManager2{ - Client: cli, - Recorder: recorder, - ctx: ctx.Ctx, - } return &clusterPlanBuilder{ ctx: ctx, cli: cli, req: req, recorder: recorder, - conMgr: conMgr, } } func (p *clusterPlan) Execute() error { err := p.dag.WalkReverseTopoOrder(p.walkFunc) if err != nil { - _ = p.conMgr.setApplyResourcesFailedCondition(p.cluster, err) + if hErr := p.handleDAGWalkError(err); hErr != nil { + return hErr + } } return err } +func (p *clusterPlan) handleDAGWalkError(err error) error { + condition := newFailedApplyResourcesCondition(err.Error()) + meta.SetStatusCondition(&p.cluster.Status.Conditions, condition) + p.recorder.Event(p.cluster, corev1.EventTypeWarning, condition.Reason, condition.Message) + rootVertex, _ := findRootVertex(p.dag) + if rootVertex == nil { + return nil + } + originCluster, _ := rootVertex.oriObj.(*appsv1alpha1.Cluster) + if originCluster == nil || reflect.DeepEqual(originCluster.Status, p.cluster.Status) { + return nil + } + return p.cli.Status().Patch(p.ctx.Ctx, p.cluster, client.MergeFrom(originCluster.DeepCopy())) +} + func (c *clusterPlanBuilder) getClusterRefResources() (*clusterRefResources, error) { cluster := c.cluster cd := &appsv1alpha1.ClusterDefinition{} @@ -333,8 +415,18 @@ func (c *clusterPlanBuilder) defaultWalkFunc(vertex graph.Vertex) error { } case STATUS: + if node.immutable { + return nil + } patch := client.MergeFrom(node.oriObj) - return c.cli.Status().Patch(c.ctx.Ctx, node.obj, patch) + if err := c.cli.Status().Patch(c.ctx.Ctx, node.obj, patch); err != nil { + return err + } + for _, postHandle := range node.postHandleAfterStatusPatch { + if err := postHandle(); err != nil { + return err + } + } } return nil } @@ -405,15 +497,7 @@ func (c *clusterPlanBuilder) buildUpdateObj(node *lifecycleVertex) (client.Objec func (c *clusterPlanBuilder) handleClusterDeletion(cluster *appsv1alpha1.Cluster) error { switch cluster.Spec.TerminationPolicy { case appsv1alpha1.DoNotTerminate: - // if cluster.Status.Phase != appsv1alpha1.DeletingClusterPhase { - // patch := client.MergeFrom(cluster.DeepCopy()) - // cluster.Status.ObservedGeneration = cluster.Generation - // // cluster.Status.Message = fmt.Sprintf("spec.terminationPolicy %s is preventing deletion.", cluster.Spec.TerminationPolicy) - // if err := r.Status().Patch(reqCtx.Ctx, cluster, patch); err != nil { - // return intctrlutil.ResultToP(intctrlutil.CheckedRequeueWithError(err, reqCtx.Log, "")) - // } - // } - // TODO: add warning event + c.recorder.Eventf(cluster, corev1.EventTypeWarning, "DoNotTerminate", "spec.terminationPolicy %s is preventing deletion.", cluster.Spec.TerminationPolicy) return nil case appsv1alpha1.Delete, appsv1alpha1.WipeOut: if err := c.deletePVCs(cluster); err != nil && !apierrors.IsNotFound(err) { diff --git a/internal/controller/lifecycle/cluster_plan_utils.go b/internal/controller/lifecycle/cluster_plan_utils.go index 3bbf4467a..c9d1bb544 100644 --- a/internal/controller/lifecycle/cluster_plan_utils.go +++ b/internal/controller/lifecycle/cluster_plan_utils.go @@ -54,15 +54,10 @@ func mergeServiceAnnotations(originalAnnotations, targetAnnotations map[string]s } // updateComponentPhaseWithOperation if workload of component changes, should update the component phase. -// REVIEW: this function need provide return value to determine mutation or not -// Deprecated: func updateComponentPhaseWithOperation(cluster *appsv1alpha1.Cluster, componentName string) { - if len(componentName) == 0 { - return - } componentPhase := appsv1alpha1.SpecReconcilingClusterCompPhase - if cluster.Status.Phase == appsv1alpha1.StartingClusterPhase { - componentPhase = appsv1alpha1.StartingClusterCompPhase + if cluster.Status.Phase == appsv1alpha1.CreatingClusterPhase { + componentPhase = appsv1alpha1.CreatingClusterCompPhase } compStatus := cluster.Status.Components[componentName] // synchronous component phase is consistent with cluster phase diff --git a/internal/controller/lifecycle/cluster_status_conditions.go b/internal/controller/lifecycle/cluster_status_conditions.go index 3521b6991..bb24b8d61 100644 --- a/internal/controller/lifecycle/cluster_status_conditions.go +++ b/internal/controller/lifecycle/cluster_status_conditions.go @@ -17,143 +17,53 @@ limitations under the License. package lifecycle import ( - "context" "fmt" "reflect" + "strings" "golang.org/x/exp/maps" "golang.org/x/exp/slices" - corev1 "k8s.io/api/core/v1" - apierrors "k8s.io/apimachinery/pkg/api/errors" - "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/client-go/tools/record" - "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" - "github.com/apecloud/kubeblocks/controllers/apps/components/util" - "github.com/apecloud/kubeblocks/internal/constant" ) -type clusterConditionManager2 struct { - client.Client - Recorder record.EventRecorder - ctx context.Context -} - const ( - // ConditionTypeProvisioningStarted the operator starts resource provisioning to create or change the cluster - ConditionTypeProvisioningStarted = "ProvisioningStarted" - // ConditionTypeApplyResources the operator start to apply resources to create or change the cluster - ConditionTypeApplyResources = "ApplyResources" - // ConditionTypeReplicasReady all pods of components are ready - ConditionTypeReplicasReady = "ReplicasReady" - // ConditionTypeReady all components are running - ConditionTypeReady = "Ready" - - // ReasonPreCheckSucceed preChecks succeed for provisioning started - ReasonPreCheckSucceed = "PreCheckSucceed" - // ReasonPreCheckFailed preChecks failed for provisioning started - ReasonPreCheckFailed = "PreCheckFailed" - // ReasonApplyResourcesFailed applies resources failed to create or change the cluster - ReasonApplyResourcesFailed = "ApplyResourcesFailed" - // ReasonApplyResourcesSucceed applies resources succeed to create or change the cluster - ReasonApplyResourcesSucceed = "ApplyResourcesSucceed" - // ReasonReplicasNotReady the pods of components are not ready - ReasonReplicasNotReady = "ReplicasNotReady" - // ReasonAllReplicasReady the pods of components are ready - ReasonAllReplicasReady = "AllReplicasReady" - // ReasonComponentsNotReady the components of cluster are not ready - ReasonComponentsNotReady = "ComponentsNotReady" - // ReasonClusterReady the components of cluster are ready, the component phase are running - ReasonClusterReady = "ClusterReady" - - // ClusterControllerErrorDuration if there is an error in the cluster controller, - // it will not be automatically repaired unless there is network jitter. - // so if the error lasts more than 5s, the cluster will enter the ConditionsError phase - // and prompt the user to repair manually according to the message. - // ClusterControllerErrorDuration = 5 * time.Second - - // ControllerErrorRequeueTime the requeue time to reconcile the error event of the cluster controller - // which need to respond to user repair events timely. - // ControllerErrorRequeueTime = 5 * time.Second + ReasonOpsRequestProcessed = "Processed" // ReasonOpsRequestProcessed the latest OpsRequest has been processed. + ReasonPreCheckSucceed = "PreCheckSucceed" // ReasonPreCheckSucceed preChecks succeed for provisioning started + ReasonPreCheckFailed = "PreCheckFailed" // ReasonPreCheckFailed preChecks failed for provisioning started + ReasonApplyResourcesFailed = "ApplyResourcesFailed" // ReasonApplyResourcesFailed applies resources failed to create or change the cluster + ReasonApplyResourcesSucceed = "ApplyResourcesSucceed" // ReasonApplyResourcesSucceed applies resources succeed to create or change the cluster + ReasonReplicasNotReady = "ReplicasNotReady" // ReasonReplicasNotReady the pods of components are not ready + ReasonAllReplicasReady = "AllReplicasReady" // ReasonAllReplicasReady the pods of components are ready + ReasonComponentsNotReady = "ComponentsNotReady" // ReasonComponentsNotReady the components of cluster are not ready + ReasonClusterReady = "ClusterReady" // ReasonClusterReady the components of cluster are ready, the component phase are running ) -// REVIEW: this handling patches co-relation object upon condition patch succeed (cascade issue), -// need better handling technique; function handling is monolithic, call for refactor. -// -// updateClusterConditions updates cluster.status condition and records event. -// Deprecated: avoid monolithic and cascade processing -func (conMgr clusterConditionManager2) updateStatusConditions(cluster *appsv1alpha1.Cluster, condition metav1.Condition, isFailCondition bool) error { - patch := client.MergeFrom(cluster.DeepCopy()) - oldCondition := meta.FindStatusCondition(cluster.Status.Conditions, condition.Type) - conditionChanged := !reflect.DeepEqual(oldCondition, condition) - if conditionChanged { - meta.SetStatusCondition(&cluster.Status.Conditions, condition) - if isFailCondition { - if err := conMgr.Client.Status().Patch(conMgr.ctx, cluster, patch); err != nil { - return err - } - } - } - if conditionChanged { - eventType := corev1.EventTypeWarning - if condition.Status == metav1.ConditionTrue { - eventType = corev1.EventTypeNormal - } - conMgr.Recorder.Event(cluster, eventType, condition.Reason, condition.Message) - } - // REVIEW/TODO: tmp remove following for interaction with OpsRequest - // if phaseChanged { - // // if cluster status changed, do it - // return opsutil.MarkRunningOpsRequestAnnotation(conMgr.ctx, conMgr.Client, conMgr.cluster) - // } - return util.ErrNoOps -} - -// setProvisioningStartedCondition sets the provisioning started condition in cluster conditions. -// @return could return ErrNoOps -// Deprecated: avoid monolithic handling -func (conMgr clusterConditionManager2) setProvisioningStartedCondition(cluster *appsv1alpha1.Cluster) error { - condition := metav1.Condition{ - Type: ConditionTypeProvisioningStarted, - Status: metav1.ConditionTrue, - Message: fmt.Sprintf("The operator has started the provisioning of Cluster: %s", cluster.Name), - Reason: ReasonPreCheckSucceed, +// conditionIsChanged checks if the condition is changed. +func conditionIsChanged(oldCondition *metav1.Condition, newCondition metav1.Condition) bool { + if newCondition.LastTransitionTime.IsZero() && oldCondition != nil { + // assign the old condition's LastTransitionTime to the new condition for "DeepEqual" checking. + newCondition.LastTransitionTime = oldCondition.LastTransitionTime } - return conMgr.updateStatusConditions(cluster, condition, false) + return !reflect.DeepEqual(oldCondition, &newCondition) } -// setPreCheckErrorCondition sets the error condition when preCheck failed. -// @return could return ErrNoOps -// Deprecated: avoid monolithic handling -func (conMgr clusterConditionManager2) setPreCheckErrorCondition(cluster *appsv1alpha1.Cluster, err error) error { - reason := ReasonPreCheckFailed - if apierrors.IsNotFound(err) { - reason = constant.ReasonNotFoundCR +// newProvisioningStartedCondition creates the provisioning started condition in cluster conditions. +func newProvisioningStartedCondition(clusterName string, clusterGeneration int64) metav1.Condition { + return metav1.Condition{ + Type: appsv1alpha1.ConditionTypeProvisioningStarted, + ObservedGeneration: clusterGeneration, + Status: metav1.ConditionTrue, + Message: fmt.Sprintf("The operator has started the provisioning of Cluster: %s", clusterName), + Reason: ReasonPreCheckSucceed, } - return conMgr.updateStatusConditions(cluster, newFailedProvisioningStartedCondition(err.Error(), reason), true) -} - -// setUnavailableCondition sets the condition that reference CRs are unavailable. -// @return could return ErrNoOps -// Deprecated: avoid monolithic handling -func (conMgr clusterConditionManager2) setReferenceCRUnavailableCondition(cluster *appsv1alpha1.Cluster, message string) error { - return conMgr.updateStatusConditions(cluster, newFailedProvisioningStartedCondition( - message, constant.ReasonRefCRUnavailable), true) -} - -// setApplyResourcesFailedCondition sets applied resources failed condition in cluster conditions. -// @return could return ErrNoOps -// Deprecated: avoid monolithic handling -func (conMgr clusterConditionManager2) setApplyResourcesFailedCondition(cluster *appsv1alpha1.Cluster, err error) error { - return conMgr.updateStatusConditions(cluster, newFailedApplyResourcesCondition(err.Error()), true) } // newApplyResourcesCondition creates a condition when applied resources succeed. func newFailedProvisioningStartedCondition(message, reason string) metav1.Condition { return metav1.Condition{ - Type: ConditionTypeProvisioningStarted, + Type: appsv1alpha1.ConditionTypeProvisioningStarted, Status: metav1.ConditionFalse, Message: message, Reason: reason, @@ -161,19 +71,20 @@ func newFailedProvisioningStartedCondition(message, reason string) metav1.Condit } // newApplyResourcesCondition creates a condition when applied resources succeed. -func newApplyResourcesCondition() metav1.Condition { +func newApplyResourcesCondition(clusterGeneration int64) metav1.Condition { return metav1.Condition{ - Type: ConditionTypeApplyResources, - Status: metav1.ConditionTrue, - Message: "Successfully applied for resources", - Reason: ReasonApplyResourcesSucceed, + Type: appsv1alpha1.ConditionTypeApplyResources, + ObservedGeneration: clusterGeneration, + Status: metav1.ConditionTrue, + Message: "Successfully applied for resources", + Reason: ReasonApplyResourcesSucceed, } } // newApplyResourcesCondition creates a condition when applied resources succeed. func newFailedApplyResourcesCondition(message string) metav1.Condition { return metav1.Condition{ - Type: ConditionTypeApplyResources, + Type: appsv1alpha1.ConditionTypeApplyResources, Status: metav1.ConditionFalse, Message: message, Reason: ReasonApplyResourcesFailed, @@ -183,7 +94,7 @@ func newFailedApplyResourcesCondition(message string) metav1.Condition { // newAllReplicasPodsReadyConditions creates a condition when all pods of components are ready func newAllReplicasPodsReadyConditions() metav1.Condition { return metav1.Condition{ - Type: ConditionTypeReplicasReady, + Type: appsv1alpha1.ConditionTypeReplicasReady, Status: metav1.ConditionTrue, Message: "all pods of components are ready, waiting for the probe detection successful", Reason: ReasonAllReplicasReady, @@ -195,7 +106,7 @@ func newReplicasNotReadyCondition(notReadyComponentNames map[string]struct{}) me cNameSlice := maps.Keys(notReadyComponentNames) slices.Sort(cNameSlice) return metav1.Condition{ - Type: ConditionTypeReplicasReady, + Type: appsv1alpha1.ConditionTypeReplicasReady, Status: metav1.ConditionFalse, Message: fmt.Sprintf("pods are not ready in Components: %v, refer to related component message in Cluster.status.components", cNameSlice), Reason: ReasonReplicasNotReady, @@ -205,7 +116,7 @@ func newReplicasNotReadyCondition(notReadyComponentNames map[string]struct{}) me // newClusterReadyCondition creates a condition when all components of cluster are running func newClusterReadyCondition(clusterName string) metav1.Condition { return metav1.Condition{ - Type: ConditionTypeReady, + Type: appsv1alpha1.ConditionTypeReady, Status: metav1.ConditionTrue, Message: fmt.Sprintf("Cluster: %s is ready, current phase is Running", clusterName), Reason: ReasonClusterReady, @@ -217,55 +128,29 @@ func newComponentsNotReadyCondition(notReadyComponentNames map[string]struct{}) cNameSlice := maps.Keys(notReadyComponentNames) slices.Sort(cNameSlice) return metav1.Condition{ - Type: ConditionTypeReady, + Type: appsv1alpha1.ConditionTypeReady, Status: metav1.ConditionFalse, Message: fmt.Sprintf("pods are unavailable in Components: %v, refer to related component message in Cluster.status.components", cNameSlice), Reason: ReasonComponentsNotReady, } } -// checkConditionIsChanged checks if the condition is changed. -func checkConditionIsChanged(oldCondition *metav1.Condition, newCondition metav1.Condition) bool { - if oldCondition == nil { - return true +// newOpsRequestProcessingCondition creates a condition when the latest opsRequest of cluster is processing +func newOpsRequestProcessingCondition(opsName, opsType, reason string) metav1.Condition { + return metav1.Condition{ + Type: appsv1alpha1.ConditionTypeLatestOpsRequestProcessed, + Status: metav1.ConditionFalse, + Message: fmt.Sprintf("%s opsRequest: %s is processing", opsType, opsName), + Reason: reason, } - return oldCondition.Message != newCondition.Message } -// handleClusterReadyCondition handles the cluster conditions with ClusterReady and ReplicasReady type. -// @return could return ErrNoOps -func handleNotReadyConditionForCluster(cluster *appsv1alpha1.Cluster, - recorder record.EventRecorder, - replicasNotReadyCompNames map[string]struct{}, - notReadyCompNames map[string]struct{}) (postHandler, error) { - oldReplicasReadyCondition := meta.FindStatusCondition(cluster.Status.Conditions, ConditionTypeReplicasReady) - if len(replicasNotReadyCompNames) == 0 { - // if all replicas of cluster are ready, set ReasonAllReplicasReady to status.conditions - readyCondition := newAllReplicasPodsReadyConditions() - if checkConditionIsChanged(oldReplicasReadyCondition, readyCondition) { - meta.SetStatusCondition(&cluster.Status.Conditions, readyCondition) - postFunc := func(cluster *appsv1alpha1.Cluster) error { - // send an event when all pods of the components are ready. - recorder.Event(cluster, corev1.EventTypeNormal, readyCondition.Reason, readyCondition.Message) - return nil - } - return postFunc, nil - } - } else { - replicasNotReadyCond := newReplicasNotReadyCondition(replicasNotReadyCompNames) - if checkConditionIsChanged(oldReplicasReadyCondition, replicasNotReadyCond) { - meta.SetStatusCondition(&cluster.Status.Conditions, replicasNotReadyCond) - return nil, nil - } - } - - if len(notReadyCompNames) > 0 { - oldClusterReadyCondition := meta.FindStatusCondition(cluster.Status.Conditions, ConditionTypeReady) - clusterNotReadyCondition := newComponentsNotReadyCondition(notReadyCompNames) - if checkConditionIsChanged(oldClusterReadyCondition, clusterNotReadyCondition) { - meta.SetStatusCondition(&cluster.Status.Conditions, clusterNotReadyCondition) - return nil, nil - } +// newOpsRequestProcessedCondition creates a condition when the latest opsRequest of cluster has been processed +func newOpsRequestProcessedCondition(processingMessage string) metav1.Condition { + return metav1.Condition{ + Type: appsv1alpha1.ConditionTypeLatestOpsRequestProcessed, + Status: metav1.ConditionTrue, + Message: strings.Replace(processingMessage, "is processing", "has been processed", 1), + Reason: ReasonOpsRequestProcessed, } - return nil, util.ErrNoOps } diff --git a/internal/controller/lifecycle/transform_types.go b/internal/controller/lifecycle/transform_types.go index ed3994ff6..695041b4a 100644 --- a/internal/controller/lifecycle/transform_types.go +++ b/internal/controller/lifecycle/transform_types.go @@ -89,6 +89,8 @@ type lifecycleVertex struct { immutable bool isOrphan bool action *Action + // postHandleAfterStatusPatch is called after the object status has changed + postHandleAfterStatusPatch []func() error } func (v lifecycleVertex) String() string { @@ -122,14 +124,6 @@ func (r *realRequeueError) Reason() string { return r.reason } -// TODO: dedup -// postHandler defines the handler after patching cluster status. -type postHandler func(cluster *appsv1alpha1.Cluster) error - -// TODO: dedup -// clusterStatusHandler a cluster status handler which changes of Cluster.status will be patched uniformly by doChainClusterStatusHandler. -type clusterStatusHandler func(cluster *appsv1alpha1.Cluster) (postHandler, error) - type delegateClient struct { client.Client } diff --git a/internal/controller/lifecycle/transformer_cluster_status.go b/internal/controller/lifecycle/transformer_cluster_status.go index 1a62e0511..abe0c72b8 100644 --- a/internal/controller/lifecycle/transformer_cluster_status.go +++ b/internal/controller/lifecycle/transformer_cluster_status.go @@ -17,15 +17,14 @@ limitations under the License. package lifecycle import ( - "context" - "encoding/json" "fmt" - "time" + "reflect" "golang.org/x/exp/slices" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" @@ -38,14 +37,45 @@ import ( intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" ) +// phaseSyncLevel defines a phase synchronization level to notify the status synchronizer how to handle cluster phase. +type phaseSyncLevel int + +const ( + clusterPhaseNoChange phaseSyncLevel = iota + clusterIsRunning // cluster is running + clusterIsStopped // cluster is stopped + clusterExistFailedOrAbnormal // cluster exists failed or abnormal component +) + type clusterStatusTransformer struct { cc clusterRefResources cli client.Client ctx intctrlutil.RequestCtx recorder record.EventRecorder - conMgr clusterConditionManager2 + // phaseSyncLevel defines a phase synchronization level to indicate how to handle cluster phase. + phaseSyncLevel phaseSyncLevel + // existsAbnormalOrFailed indicates whether the cluster exists abnormal or failed component. + existsAbnormalOrFailed bool + // replicasNotReadyCompNames records the component names that are not ready. + notReadyCompNames map[string]struct{} + // replicasNotReadyCompNames records the component names which replicas are not ready. + replicasNotReadyCompNames map[string]struct{} } +func newClusterStatusTransformer(ctx intctrlutil.RequestCtx, + cli client.Client, + recorder record.EventRecorder, + cc clusterRefResources) *clusterStatusTransformer { + return &clusterStatusTransformer{ + ctx: ctx, + cc: cc, + cli: cli, + recorder: recorder, + phaseSyncLevel: clusterPhaseNoChange, + notReadyCompNames: map[string]struct{}{}, + replicasNotReadyCompNames: map[string]struct{}{}, + } +} func (c *clusterStatusTransformer) Transform(dag *graph.DAG) error { rootVertex, err := findRootVertex(dag) if err != nil { @@ -73,41 +103,36 @@ func (c *clusterStatusTransformer) Transform(dag *graph.DAG) error { // if cluster is deleting, set root(cluster) vertex.action to DELETE rootVertex.action = actionPtr(DELETE) case isClusterUpdating(*origCluster): - c.ctx.Log.Info("update cluster status") + c.ctx.Log.Info("update cluster status after applying resources ") defer func() { - rootVertex.action = actionPtr(STATUS) // update components' phase in cluster.status updateComponentPhase() + rootVertex.action = actionPtr(STATUS) + rootVertex.immutable = reflect.DeepEqual(cluster.Status, origCluster.Status) }() cluster.Status.ObservedGeneration = cluster.Generation cluster.Status.ClusterDefGeneration = c.cc.cd.Generation - if cluster.Status.Phase == "" { - // REVIEW: may need to start with "validating" phase - cluster.Status.Phase = appsv1alpha1.StartingClusterPhase + applyResourcesCondition := newApplyResourcesCondition(cluster.Generation) + oldApplyCondition := meta.FindStatusCondition(cluster.Status.Conditions, applyResourcesCondition.Type) + if !conditionIsChanged(oldApplyCondition, applyResourcesCondition) { return nil } - if cluster.Status.Phase != appsv1alpha1.StartingClusterPhase { - cluster.Status.Phase = appsv1alpha1.SpecReconcilingClusterPhase - } - if err := c.conMgr.setProvisioningStartedCondition(cluster); util.IgnoreNoOps(err) != nil { - return err - } - applyResourcesCondition := newApplyResourcesCondition() - oldApplyCondition := meta.FindStatusCondition(cluster.Status.Conditions, applyResourcesCondition.Type) meta.SetStatusCondition(&cluster.Status.Conditions, applyResourcesCondition) - if oldApplyCondition == nil || oldApplyCondition.Status != applyResourcesCondition.Status { + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { c.recorder.Event(cluster, corev1.EventTypeNormal, applyResourcesCondition.Reason, applyResourcesCondition.Message) - } + return nil + }) case isClusterStatusUpdating(*origCluster): - defer func() { rootVertex.action = actionPtr(STATUS) }() + defer func() { + rootVertex.action = actionPtr(STATUS) + rootVertex.immutable = reflect.DeepEqual(cluster.Status, origCluster.Status) + }() // checks if the controller is handling the garbage of restore. - if err := c.handleGarbageOfRestoreBeforeRunning(cluster); err == nil { - return nil - } else if util.IgnoreNoOps(err) != nil { + if err := c.handleGarbageOfRestoreBeforeRunning(cluster); err != nil { return err } // reconcile the phase and conditions of the Cluster.status - if err := c.reconcileClusterStatus(cluster, c.cc.cd); util.IgnoreNoOps(err) != nil { + if err := c.reconcileClusterStatus(cluster, rootVertex); err != nil { return err } c.cleanupAnnotationsAfterRunning(cluster) @@ -116,163 +141,170 @@ func (c *clusterStatusTransformer) Transform(dag *graph.DAG) error { return nil } -// REVIEW: this handling rather monolithic // reconcileClusterStatus reconciles phase and conditions of the Cluster.status. -// @return ErrNoOps if no operation -// Deprecated: -func (c *clusterStatusTransformer) reconcileClusterStatus(cluster *appsv1alpha1.Cluster, - clusterDef appsv1alpha1.ClusterDefinition) error { +func (c *clusterStatusTransformer) reconcileClusterStatus(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) error { if len(cluster.Status.Components) == 0 { return nil } + // removes the invalid component of status.components which is deleted from spec.components. + c.removeInvalidCompStatus(cluster) - var ( - currentClusterPhase appsv1alpha1.ClusterPhase - existsAbnormalOrFailed bool - notReadyCompNames = map[string]struct{}{} - replicasNotReadyCompNames = map[string]struct{}{} - ) + // do analysis of Cluster.Status.component and update the results to status synchronizer. + c.doAnalysisAndUpdateSynchronizer(cluster) - // analysis the status of components and calculate the cluster phase . - analysisComponentsStatus := func(cluster *appsv1alpha1.Cluster) { - var ( - runningCompCount int - stoppedCompCount int - ) - for k, v := range cluster.Status.Components { - if v.PodsReady == nil || !*v.PodsReady { - replicasNotReadyCompNames[k] = struct{}{} - notReadyCompNames[k] = struct{}{} - } - switch v.Phase { - case appsv1alpha1.AbnormalClusterCompPhase, appsv1alpha1.FailedClusterCompPhase: - existsAbnormalOrFailed = true - notReadyCompNames[k] = struct{}{} - case appsv1alpha1.RunningClusterCompPhase: - runningCompCount += 1 - case appsv1alpha1.StoppedClusterCompPhase: - stoppedCompCount += 1 - } + // sync the LatestOpsRequestProcessed condition. + c.syncOpsRequestProcessedCondition(cluster, rootVertex) + + // handle the ready condition. + c.syncReadyConditionForCluster(cluster, rootVertex) + + // sync the cluster phase. + switch c.phaseSyncLevel { + case clusterIsRunning: + if cluster.Status.Phase != appsv1alpha1.RunningClusterPhase { + c.syncClusterPhaseToRunning(cluster, rootVertex) } - compLen := len(cluster.Status.Components) - notReadyLen := len(notReadyCompNames) - if existsAbnormalOrFailed && notReadyLen > 0 { - if compLen == notReadyLen { - currentClusterPhase = appsv1alpha1.FailedClusterPhase - } else { - currentClusterPhase = appsv1alpha1.AbnormalClusterPhase - } - return + case clusterIsStopped: + if cluster.Status.Phase != appsv1alpha1.StoppedClusterPhase { + c.syncClusterPhaseToStopped(cluster, rootVertex) } - switch len(cluster.Status.Components) { - case 0: - // if no components, return, and how could this possible? - return - case runningCompCount: - currentClusterPhase = appsv1alpha1.RunningClusterPhase - case stoppedCompCount: - // cluster is Stopped when cluster is not Running and all components are Stopped or Running - currentClusterPhase = appsv1alpha1.StoppedClusterPhase + case clusterExistFailedOrAbnormal: + c.handleExistAbnormalOrFailed(cluster, rootVertex) + } + return nil +} + +// removeInvalidCompStatus removes the invalid component of status.components which is deleted from spec.components. +func (c *clusterStatusTransformer) removeInvalidCompStatus(cluster *appsv1alpha1.Cluster) { + // remove the invalid component in status.components when the component is deleted from spec.components. + tmpCompsStatus := map[string]appsv1alpha1.ClusterComponentStatus{} + compsStatus := cluster.Status.Components + for _, v := range cluster.Spec.ComponentSpecs { + if compStatus, ok := compsStatus[v.Name]; ok { + tmpCompsStatus[v.Name] = compStatus } } + // keep valid components' status + cluster.Status.Components = tmpCompsStatus +} - // remove the invalid component in status.components when spec.components changed and analysis the status of components. - removeInvalidComponentsAndAnalysis := func(cluster *appsv1alpha1.Cluster) (postHandler, error) { - tmpCompsStatus := map[string]appsv1alpha1.ClusterComponentStatus{} - compsStatus := cluster.Status.Components - for _, v := range cluster.Spec.ComponentSpecs { - if compStatus, ok := compsStatus[v.Name]; ok { - tmpCompsStatus[v.Name] = compStatus - } +// doAnalysisAndUpdateSynchronizer analyses the Cluster.Status.Components and updates the results to the synchronizer. +func (c *clusterStatusTransformer) doAnalysisAndUpdateSynchronizer(cluster *appsv1alpha1.Cluster) { + var ( + runningCompCount int + stoppedCompCount int + ) + // analysis the status of components and calculate the cluster phase. + for k, v := range cluster.Status.Components { + if v.PodsReady == nil || !*v.PodsReady { + c.replicasNotReadyCompNames[k] = struct{}{} + c.notReadyCompNames[k] = struct{}{} } - if len(tmpCompsStatus) != len(compsStatus) { - // keep valid components' status - cluster.Status.Components = tmpCompsStatus - return nil, nil + switch v.Phase { + case appsv1alpha1.AbnormalClusterCompPhase, appsv1alpha1.FailedClusterCompPhase: + c.existsAbnormalOrFailed, c.notReadyCompNames[k] = true, struct{}{} + case appsv1alpha1.RunningClusterCompPhase: + runningCompCount += 1 + case appsv1alpha1.StoppedClusterCompPhase: + stoppedCompCount += 1 } - analysisComponentsStatus(cluster) - return nil, util.ErrNoOps } - - // handle the cluster conditions with ClusterReady and ReplicasReady type. - handleClusterReadyCondition := func(cluster *appsv1alpha1.Cluster) (postHandler, error) { - return handleNotReadyConditionForCluster(cluster, c.recorder, replicasNotReadyCompNames, notReadyCompNames) + if c.existsAbnormalOrFailed { + c.phaseSyncLevel = clusterExistFailedOrAbnormal + return } + switch len(cluster.Status.Components) { + case runningCompCount: + c.phaseSyncLevel = clusterIsRunning + case stoppedCompCount: + // cluster is Stopped when cluster is not Running and all components are Stopped or Running + c.phaseSyncLevel = clusterIsStopped + } +} - // processes cluster phase changes. - processClusterPhaseChanges := func(cluster *appsv1alpha1.Cluster, - oldPhase, - currPhase appsv1alpha1.ClusterPhase, - eventType string, - eventMessage string, - doAction func(cluster *appsv1alpha1.Cluster)) (postHandler, error) { - if oldPhase == currPhase { - return nil, util.ErrNoOps - } - cluster.Status.Phase = currPhase - if doAction != nil { - doAction(cluster) - } - postFuncAfterPatch := func(currCluster *appsv1alpha1.Cluster) error { - c.recorder.Event(currCluster, eventType, string(currPhase), eventMessage) - return opsutil.MarkRunningOpsRequestAnnotation(c.ctx.Ctx, c.cli, currCluster) - } - return postFuncAfterPatch, nil +// handleOpsRequestProcessedCondition syncs the condition that OpsRequest has been processed. +func (c *clusterStatusTransformer) syncOpsRequestProcessedCondition(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) { + opsCondition := meta.FindStatusCondition(cluster.Status.Conditions, appsv1alpha1.ConditionTypeLatestOpsRequestProcessed) + if opsCondition == nil || opsCondition.Status == metav1.ConditionTrue { + return } - // handle the Cluster.status when some components of cluster are Abnormal or Failed. - handleExistAbnormalOrFailed := func(cluster *appsv1alpha1.Cluster) (postHandler, error) { - if !existsAbnormalOrFailed { - return nil, util.ErrNoOps - } - oldPhase := cluster.Status.Phase - componentMap, clusterAvailabilityEffectMap, _ := getComponentRelatedInfo(cluster, - clusterDef, "") - // handle the cluster status when some components are not ready. - handleClusterPhaseWhenCompsNotReady(cluster, componentMap, clusterAvailabilityEffectMap) - currPhase := cluster.Status.Phase - if !slices.Contains(appsv1alpha1.GetClusterFailedPhases(), currPhase) { - return nil, util.ErrNoOps - } - message := fmt.Sprintf("Cluster: %s is %s, check according to the components message", - cluster.Name, currPhase) - return processClusterPhaseChanges(cluster, oldPhase, currPhase, - corev1.EventTypeWarning, message, nil) + opsRecords, _ := opsutil.GetOpsRequestSliceFromCluster(cluster) + if len(opsRecords) != 0 { + return } + processedCondition := newOpsRequestProcessedCondition(opsCondition.Message) + oldCondition := meta.FindStatusCondition(cluster.Status.Conditions, processedCondition.Type) + if !conditionIsChanged(oldCondition, processedCondition) { + return + } + meta.SetStatusCondition(&cluster.Status.Conditions, processedCondition) + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { + // send an event when all pods of the components are ready. + c.recorder.Event(cluster, corev1.EventTypeNormal, processedCondition.Reason, processedCondition.Message) + return nil + }) +} - // handle the Cluster.status when cluster is Stopped. - handleClusterIsStopped := func(cluster *appsv1alpha1.Cluster) (postHandler, error) { - if currentClusterPhase != appsv1alpha1.StoppedClusterPhase { - return nil, util.ErrNoOps +// syncReadyConditionForCluster syncs the cluster conditions with ClusterReady and ReplicasReady type. +func (c *clusterStatusTransformer) syncReadyConditionForCluster(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) { + if len(c.replicasNotReadyCompNames) == 0 { + oldReplicasReadyCondition := meta.FindStatusCondition(cluster.Status.Conditions, appsv1alpha1.ConditionTypeReplicasReady) + // if all replicas of cluster are ready, set ReasonAllReplicasReady to status.conditions + readyCondition := newAllReplicasPodsReadyConditions() + if oldReplicasReadyCondition == nil || oldReplicasReadyCondition.Status == metav1.ConditionFalse { + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { + // send an event when all pods of the components are ready. + c.recorder.Event(cluster, corev1.EventTypeNormal, readyCondition.Reason, readyCondition.Message) + return nil + }) } - message := fmt.Sprintf("Cluster: %s stopped successfully.", cluster.Name) - oldPhase := cluster.Status.Phase - return processClusterPhaseChanges(cluster, oldPhase, currentClusterPhase, - corev1.EventTypeNormal, message, nil) + meta.SetStatusCondition(&cluster.Status.Conditions, readyCondition) + } else { + meta.SetStatusCondition(&cluster.Status.Conditions, newReplicasNotReadyCondition(c.replicasNotReadyCompNames)) } - // handle the Cluster.status when cluster is Running. - handleClusterIsRunning := func(cluster *appsv1alpha1.Cluster) (postHandler, error) { - if currentClusterPhase != appsv1alpha1.RunningClusterPhase { - return nil, util.ErrNoOps - } - message := fmt.Sprintf("Cluster: %s is ready, current phase is Running.", cluster.Name) - action := func(currCluster *appsv1alpha1.Cluster) { - meta.SetStatusCondition(&currCluster.Status.Conditions, - newClusterReadyCondition(currCluster.Name)) - } - oldPhase := cluster.Status.Phase - return processClusterPhaseChanges(cluster, oldPhase, currentClusterPhase, - corev1.EventTypeNormal, message, action) + if len(c.notReadyCompNames) > 0 { + meta.SetStatusCondition(&cluster.Status.Conditions, newComponentsNotReadyCondition(c.notReadyCompNames)) } - if err := doChainClusterStatusHandler(cluster, - removeInvalidComponentsAndAnalysis, - handleClusterReadyCondition, - handleExistAbnormalOrFailed, - handleClusterIsStopped, - handleClusterIsRunning); err != nil { - return err +} + +// syncClusterPhaseToRunning syncs the cluster phase to Running. +func (c *clusterStatusTransformer) syncClusterPhaseToRunning(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) { + cluster.Status.Phase = appsv1alpha1.RunningClusterPhase + meta.SetStatusCondition(&cluster.Status.Conditions, newClusterReadyCondition(cluster.Name)) + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { + message := fmt.Sprintf("Cluster: %s is ready, current phase is Running", cluster.Name) + c.recorder.Event(cluster, corev1.EventTypeNormal, string(appsv1alpha1.RunningClusterPhase), message) + return opsutil.MarkRunningOpsRequestAnnotation(c.ctx.Ctx, c.cli, cluster) + }) +} + +// syncClusterToStopped syncs the cluster phase to Stopped. +func (c *clusterStatusTransformer) syncClusterPhaseToStopped(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) { + cluster.Status.Phase = appsv1alpha1.StoppedClusterPhase + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { + message := fmt.Sprintf("Cluster: %s stopped successfully.", cluster.Name) + c.recorder.Event(cluster, corev1.EventTypeNormal, string(cluster.Status.Phase), message) + return opsutil.MarkRunningOpsRequestAnnotation(c.ctx.Ctx, c.cli, cluster) + }) +} + +// handleExistAbnormalOrFailed handles the cluster status when some components are not ready. +func (c *clusterStatusTransformer) handleExistAbnormalOrFailed(cluster *appsv1alpha1.Cluster, rootVertex *lifecycleVertex) { + oldPhase := cluster.Status.Phase + componentMap, clusterAvailabilityEffectMap, _ := getComponentRelatedInfo(cluster, + c.cc.cd, "") + // handle the cluster status when some components are not ready. + handleClusterPhaseWhenCompsNotReady(cluster, componentMap, clusterAvailabilityEffectMap) + currPhase := cluster.Status.Phase + if slices.Contains(appsv1alpha1.GetClusterFailedPhases(), currPhase) && oldPhase != currPhase { + rootVertex.postHandleAfterStatusPatch = append(rootVertex.postHandleAfterStatusPatch, func() error { + message := fmt.Sprintf("Cluster: %s is %s, check according to the components message", + cluster.Name, currPhase) + c.recorder.Event(cluster, corev1.EventTypeWarning, string(cluster.Status.Phase), message) + return opsutil.MarkRunningOpsRequestAnnotation(c.ctx.Ctx, c.cli, cluster) + }) } - return nil } // cleanupAnnotationsAfterRunning cleans up the cluster annotations after cluster is Running. @@ -296,12 +328,12 @@ func (c *clusterStatusTransformer) handleGarbageOfRestoreBeforeRunning(cluster * return err } if clusterBackupResourceMap == nil { - return util.ErrNoOps + return nil } // check if all components are running. for _, v := range cluster.Status.Components { if v.Phase != appsv1alpha1.RunningClusterCompPhase { - return util.ErrNoOps + return nil } } // remove the garbage for restore if the cluster restores from backup. @@ -324,7 +356,7 @@ func (c *clusterStatusTransformer) removeGarbageWithRestore( return err } } - return util.ErrNoOps + return nil } // removeStsInitContainerForRestore removes the statefulSet's init container which restores data from backup. @@ -355,7 +387,7 @@ func (c *clusterStatusTransformer) removeStsInitContainerForRestore( if doRemoveInitContainers { // if need to remove init container, reset component to Creating. compStatus := cluster.Status.Components[componentName] - compStatus.Phase = appsv1alpha1.StartingClusterCompPhase + compStatus.Phase = appsv1alpha1.CreatingClusterCompPhase cluster.Status.Components[componentName] = compStatus } return doRemoveInitContainers, nil @@ -368,15 +400,20 @@ func handleClusterPhaseWhenCompsNotReady(cluster *appsv1alpha1.Cluster, componentMap map[string]string, clusterAvailabilityEffectMap map[string]bool) { var ( - clusterIsFailed bool - failedCompCount int + clusterIsFailed bool + failedCompCount int + isVolumeExpanding bool ) + + opsRecords, _ := opsutil.GetOpsRequestSliceFromCluster(cluster) + if len(opsRecords) != 0 && opsRecords[0].Type == appsv1alpha1.VolumeExpansionType { + isVolumeExpanding = true + } for k, v := range cluster.Status.Components { // determine whether other components are still doing operation, i.e., create/restart/scaling. // waiting for operation to complete except for volumeExpansion operation. // because this operation will not affect cluster availability. - // TODO: for appsv1alpha1.VolumeExpandingPhas requires extra handling - if !util.IsCompleted(v.Phase) { + if !slices.Contains(appsv1alpha1.GetComponentTerminalPhases(), v.Phase) && !isVolumeExpanding { return } if v.Phase == appsv1alpha1.FailedClusterCompPhase { @@ -432,74 +469,3 @@ func getComponentRelatedInfo(cluster *appsv1alpha1.Cluster, clusterDef appsv1alp } return componentMap, clusterAvailabilityEffectMap, componentDef } - -// doChainClusterStatusHandler chain processing clusterStatusHandler. -func doChainClusterStatusHandler(cluster *appsv1alpha1.Cluster, - handlers ...clusterStatusHandler) error { - var ( - needPatchStatus bool - postHandlers = make([]func(cluster *appsv1alpha1.Cluster) error, 0, len(handlers)) - ) - for _, statusHandler := range handlers { - postFunc, err := statusHandler(cluster) - if err != nil { - if err == util.ErrNoOps { - continue - } - return err - } - needPatchStatus = true - if postFunc != nil { - postHandlers = append(postHandlers, postFunc) - } - } - if !needPatchStatus { - return util.ErrNoOps - } - // perform the handlers after patched the cluster status. - for _, postFunc := range postHandlers { - if err := postFunc(cluster); err != nil { - return err - } - } - return nil -} - -// TODO: dedup the following funcs - -// PatchOpsRequestReconcileAnnotation patches the reconcile annotation to OpsRequest -func PatchOpsRequestReconcileAnnotation(ctx context.Context, cli client.Client, namespace string, opsRequestName string) error { - opsRequest := &appsv1alpha1.OpsRequest{} - if err := cli.Get(ctx, client.ObjectKey{Name: opsRequestName, Namespace: namespace}, opsRequest); err != nil { - return err - } - patch := client.MergeFrom(opsRequest.DeepCopy()) - if opsRequest.Annotations == nil { - opsRequest.Annotations = map[string]string{} - } - // because many changes may be triggered within one second, if the accuracy is only seconds, the event may be lost. - // so we used RFC3339Nano format. - opsRequest.Annotations[constant.ReconcileAnnotationKey] = time.Now().Format(time.RFC3339Nano) - return cli.Patch(ctx, opsRequest, patch) -} - -// GetOpsRequestSliceFromCluster gets OpsRequest slice from cluster annotations. -// this records what OpsRequests are running in cluster -func GetOpsRequestSliceFromCluster(cluster *appsv1alpha1.Cluster) ([]appsv1alpha1.OpsRecorder, error) { - var ( - opsRequestValue string - opsRequestSlice []appsv1alpha1.OpsRecorder - ok bool - ) - if cluster == nil || cluster.Annotations == nil { - return nil, nil - } - if opsRequestValue, ok = cluster.Annotations[constant.OpsRequestAnnotationKey]; !ok { - return nil, nil - } - // opsRequest annotation value in cluster to slice - if err := json.Unmarshal([]byte(opsRequestValue), &opsRequestSlice); err != nil { - return nil, err - } - return opsRequestSlice, nil -} diff --git a/internal/controller/lifecycle/transformer_init.go b/internal/controller/lifecycle/transformer_init.go index 038ac8a6f..e8f753a1b 100644 --- a/internal/controller/lifecycle/transformer_init.go +++ b/internal/controller/lifecycle/transformer_init.go @@ -22,13 +22,13 @@ import ( ) type initTransformer struct { - cluster *appsv1alpha1.Cluster + cluster *appsv1alpha1.Cluster + originCluster *appsv1alpha1.Cluster } func (i *initTransformer) Transform(dag *graph.DAG) error { // put the cluster object first, it will be root vertex of DAG - oriCluster := i.cluster.DeepCopy() - rootVertex := &lifecycleVertex{obj: i.cluster, oriObj: oriCluster} + rootVertex := &lifecycleVertex{obj: i.cluster, oriObj: i.originCluster} dag.AddVertex(rootVertex) return nil } diff --git a/internal/controller/lifecycle/transformer_object_action.go b/internal/controller/lifecycle/transformer_object_action.go index 9c055e457..e20d62702 100644 --- a/internal/controller/lifecycle/transformer_object_action.go +++ b/internal/controller/lifecycle/transformer_object_action.go @@ -99,6 +99,10 @@ func (c *objectActionTransformer) Transform(dag *graph.DAG) error { newNameVertices := make(map[gvkName]graph.Vertex) for _, vertex := range dag.Vertices() { v, _ := vertex.(*lifecycleVertex) + if v == rootVertex { + // ignore root vertex, i.e, cluster object. + continue + } name, err := getGVKName(v.obj, scheme) if err != nil { return err diff --git a/internal/controller/lifecycle/transformer_sts_horizontal_scaling.go b/internal/controller/lifecycle/transformer_sts_horizontal_scaling.go index 3ca273d47..015c26dcf 100644 --- a/internal/controller/lifecycle/transformer_sts_horizontal_scaling.go +++ b/internal/controller/lifecycle/transformer_sts_horizontal_scaling.go @@ -683,7 +683,7 @@ func createBackup(reqCtx intctrlutil.RequestCtx, root graph.Vertex) error { ctx := reqCtx.Ctx - createBackupPolicy := func() (backupPolicyName string, err error) { + createBackupPolicy := func() (backupPolicyName string, Vertex *lifecycleVertex, err error) { backupPolicyName = "" backupPolicyList := dataprotectionv1alpha1.BackupPolicyList{} ml := getBackupMatchingLabels(cluster.Name, sts.Labels[constant.KBAppComponentLabelKey]) @@ -698,6 +698,9 @@ func createBackup(reqCtx intctrlutil.RequestCtx, if err != nil { return } + if err = controllerutil.SetControllerReference(cluster, backupPolicy, scheme); err != nil { + return + } backupPolicyName = backupPolicy.Name vertex := &lifecycleVertex{obj: backupPolicy, action: actionPtr(CREATE)} dag.AddVertex(vertex) @@ -705,7 +708,15 @@ func createBackup(reqCtx intctrlutil.RequestCtx, return } - createBackup := func(backupPolicyName string) error { + createBackup := func(backupPolicyName string, policyVertex *lifecycleVertex) error { + backupPolicy := &dataprotectionv1alpha1.BackupPolicy{} + if err := cli.Get(ctx, client.ObjectKey{Namespace: backupKey.Namespace, Name: backupPolicyName}, backupPolicy); err != nil && !apierrors.IsNotFound(err) { + return err + } + // wait for backupPolicy created + if len(backupPolicy.Name) == 0 { + return nil + } backupList := dataprotectionv1alpha1.BackupList{} ml := getBackupMatchingLabels(cluster.Name, sts.Labels[constant.KBAppComponentLabelKey]) if err := cli.List(ctx, &backupList, ml); err != nil { @@ -734,11 +745,11 @@ func createBackup(reqCtx intctrlutil.RequestCtx, return nil } - backupPolicyName, err := createBackupPolicy() + backupPolicyName, policyVertex, err := createBackupPolicy() if err != nil { return err } - if err := createBackup(backupPolicyName); err != nil { + if err := createBackup(backupPolicyName, policyVertex); err != nil { return err } From 47f66c7b45860b7be20a7994efed2d4e0f6ba381 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Fri, 31 Mar 2023 09:33:38 +0800 Subject: [PATCH 05/80] chore: improvement the prompt for the failed reconfiguring (#2282) --- apis/apps/v1alpha1/opsrequest_conditions.go | 17 ++++++++ controllers/apps/operations/reconfigure.go | 41 ++++++++++--------- .../apps/operations/reconfigure_util.go | 10 ++++- internal/configuration/config.go | 6 +-- 4 files changed, 50 insertions(+), 24 deletions(-) diff --git a/apis/apps/v1alpha1/opsrequest_conditions.go b/apis/apps/v1alpha1/opsrequest_conditions.go index 1d9b059d1..df9e58233 100644 --- a/apis/apps/v1alpha1/opsrequest_conditions.go +++ b/apis/apps/v1alpha1/opsrequest_conditions.go @@ -241,3 +241,20 @@ func NewReconfigureRunningCondition(ops *OpsRequest, conditionType string, confi Message: message, } } + +// NewReconfigureFailedCondition creates a condition for the failed reconfigure. +func NewReconfigureFailedCondition(ops *OpsRequest, err error) *metav1.Condition { + var msg string + if err != nil { + msg = err.Error() + } else { + msg = fmt.Sprintf("Failed to reconfigure: %s in cluster: %s", ops.Name, ops.Spec.ClusterRef) + } + return &metav1.Condition{ + Type: ReasonReconfigureFailed, + Status: metav1.ConditionFalse, + Reason: "ReconfigureFailed", + LastTransitionTime: metav1.Now(), + Message: msg, + } +} diff --git a/controllers/apps/operations/reconfigure.go b/controllers/apps/operations/reconfigure.go index 78859c5ee..0f5e18d6d 100644 --- a/controllers/apps/operations/reconfigure.go +++ b/controllers/apps/operations/reconfigure.go @@ -106,7 +106,7 @@ func (r *reconfigureAction) Handle(eventContext cfgcore.ConfigEventContext, last appsv1alpha1.ReasonReconfigureFailed, eventContext.ConfigSpecName, formatConfigPatchToMessage(eventContext.ConfigPatch, &eventContext.PolicyStatus)), - appsv1alpha1.NewFailedCondition(opsRequest, cfgError)) + appsv1alpha1.NewReconfigureFailedCondition(opsRequest, cfgError)) default: return PatchOpsStatusWithOpsDeepCopy(ctx, cli, opsRes, opsDeepCopy, appsv1alpha1.OpsRunningPhase, appsv1alpha1.NewReconfigureRunningCondition(opsRequest, @@ -255,15 +255,16 @@ func (r *reconfigureAction) Action(reqCtx intctrlutil.RequestCtx, cli client.Cli return err } - tpls, err := cfgcore.GetConfigTemplatesFromComponent( + configSpecs, err := cfgcore.GetConfigTemplatesFromComponent( cluster.Spec.ComponentSpecs, clusterDefinition.Spec.ComponentDefs, clusterVersion.Spec.ComponentVersions, componentName) if err != nil { - return cfgcore.WrapError(err, "failed to get config template[%s]", componentName) + return processMergedFailed(resource, true, + cfgcore.WrapError(err, "failed to get config template in the component[%s]", componentName)) } - return r.doMergeAndPersist(reqCtx, cli, clusterName, componentName, spec.Reconfigure, resource, tpls) + return r.doMergeAndPersist(reqCtx, cli, clusterName, componentName, spec.Reconfigure, resource, configSpecs) } func (r *reconfigureAction) doMergeAndPersist(reqCtx intctrlutil.RequestCtx, @@ -271,14 +272,14 @@ func (r *reconfigureAction) doMergeAndPersist(reqCtx intctrlutil.RequestCtx, clusterName, componentName string, reconfigure *appsv1alpha1.Reconfigure, resource *OpsResource, - tpls []appsv1alpha1.ComponentConfigSpec) error { - findTpl := func(tplName string) *appsv1alpha1.ComponentConfigSpec { - if len(tplName) == 0 && len(tpls) == 1 { - return &tpls[0] + configSpecs []appsv1alpha1.ComponentConfigSpec) error { + foundConfigSpec := func(configSpecName string) *appsv1alpha1.ComponentConfigSpec { + if len(configSpecName) == 0 && len(configSpecs) == 1 { + return &configSpecs[0] } - for _, tpl := range tpls { - if tpl.Name == tplName { - return &tpl + for _, configSpec := range configSpecs { + if configSpec.Name == configSpecName { + return &configSpec } } return nil @@ -287,17 +288,17 @@ func (r *reconfigureAction) doMergeAndPersist(reqCtx intctrlutil.RequestCtx, // Update params to configmap // TODO support multi tpl conditions merge for _, config := range reconfigure.Configurations { - tpl := findTpl(config.Name) - if tpl == nil { + configSpec := foundConfigSpec(config.Name) + if configSpec == nil { return processMergedFailed(resource, true, - cfgcore.MakeError("failed to reconfigure, not exist config[%s], all configs: %v", config.Name, tpls)) + cfgcore.MakeError("failed to reconfigure, not exist config[%s], all configs: %v", config.Name, getConfigSpecName(configSpecs))) } - if len(tpl.ConfigConstraintRef) == 0 { + if len(configSpec.ConfigConstraintRef) == 0 { return processMergedFailed(resource, true, - cfgcore.MakeError("current tpl not support reconfigure, tpl: %v", tpl)) + cfgcore.MakeError("current configSpec not support reconfigure, configSpec: %v", configSpec.Name)) } - result := updateCfgParams(config, *tpl, client.ObjectKey{ - Name: cfgcore.GetComponentCfgName(clusterName, componentName, tpl.VolumeName), + result := updateCfgParams(config, *configSpec, client.ObjectKey{ + Name: cfgcore.GetComponentCfgName(clusterName, componentName, configSpec.VolumeName), Namespace: resource.Cluster.Namespace, }, reqCtx.Ctx, cli, resource.OpsRequest.Name) if result.err != nil { @@ -305,11 +306,11 @@ func (r *reconfigureAction) doMergeAndPersist(reqCtx intctrlutil.RequestCtx, } // merged successfully - if err := patchReconfigureOpsStatus(resource, tpl.Name, + if err := patchReconfigureOpsStatus(resource, configSpec.Name, handleNewReconfigureRequest(result.configPatch, result.lastAppliedConfigs)); err != nil { return err } - condition := constructReconfiguringConditions(result, resource, tpl) + condition := constructReconfiguringConditions(result, resource, configSpec) resource.OpsRequest.SetStatusCondition(*condition) } return nil diff --git a/controllers/apps/operations/reconfigure_util.go b/controllers/apps/operations/reconfigure_util.go index ee3b96078..38dc826cb 100644 --- a/controllers/apps/operations/reconfigure_util.go +++ b/controllers/apps/operations/reconfigure_util.go @@ -132,6 +132,14 @@ func makeReconfiguringResult(err error, ops ...func(*reconfiguringResult)) recon return result } +func getConfigSpecName(configSpec []appsv1alpha1.ComponentConfigSpec) []string { + names := make([]string, len(configSpec)) + for i, spec := range configSpec { + names[i] = spec.Name + } + return names +} + func constructReconfiguringConditions(result reconfiguringResult, resource *OpsResource, configSpec *appsv1alpha1.ComponentConfigSpec) *metav1.Condition { if result.configPatch.IsModify { return appsv1alpha1.NewReconfigureRunningCondition( @@ -176,7 +184,7 @@ func processMergedFailed(resource *OpsResource, isInvalid bool, err error) error } // if failed to validate configure, set opsRequest to failed and return - failedCondition := appsv1alpha1.NewFailedCondition(resource.OpsRequest, err) + failedCondition := appsv1alpha1.NewReconfigureFailedCondition(resource.OpsRequest, err) resource.OpsRequest.SetStatusCondition(*failedCondition) return nil } diff --git a/internal/configuration/config.go b/internal/configuration/config.go index fc97e65f8..b2510602c 100644 --- a/internal/configuration/config.go +++ b/internal/configuration/config.go @@ -195,7 +195,7 @@ type Option func(ctx *CfgOpOption) func (c *cfgWrapper) MergeFrom(params map[string]interface{}, option CfgOpOption) error { cfg := c.getConfigObject(option) if cfg == nil { - return MakeError("not any configuration. option:[%v]", option) + return MakeError("not found the config file:[%s]", option.FileName) } // TODO support param delete @@ -302,14 +302,14 @@ func (c *cfgWrapper) Query(jsonpath string, option CfgOpOption) ([]byte, error) cfg := c.getConfigObject(option) if cfg == nil { - return nil, MakeError("not any configuration. option:[%v]", option) + return nil, MakeError("not found the config file:[%s]", option.FileName) } iniContext := option.IniContext if iniContext != nil && len(iniContext.SectionName) > 0 { cfg = cfg.SubConfig(iniContext.SectionName) if cfg == nil { - return nil, MakeError("configuration not exist section [%s]", iniContext.SectionName) + return nil, MakeError("the section[%s] does not exist in the config file", iniContext.SectionName) } } From 8aa05e3db8e8bf6baf2a545b4a574f5ab028bcf9 Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Fri, 31 Mar 2023 09:58:04 +0800 Subject: [PATCH 06/80] feat: support ChatGPT-Retrieval-Plugin (#2332) --- deploy/chatgpt-retrieval-plugin/Chart.yaml | 12 ++ .../templates/NOTES.txt | 14 ++ .../templates/_helpers.tpl | 62 +++++++ .../templates/clusterrole.yaml | 8 + .../templates/clusterrolebinding.yaml | 12 ++ .../templates/deployment.yaml | 136 +++++++++++++++ .../templates/service.yaml | 15 ++ .../templates/serviceaccount.yaml | 12 ++ .../templates/tests/test-connection.yaml | 15 ++ deploy/chatgpt-retrieval-plugin/values.yaml | 162 ++++++++++++++++++ 10 files changed, 448 insertions(+) create mode 100644 deploy/chatgpt-retrieval-plugin/Chart.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/NOTES.txt create mode 100644 deploy/chatgpt-retrieval-plugin/templates/_helpers.tpl create mode 100644 deploy/chatgpt-retrieval-plugin/templates/clusterrole.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/clusterrolebinding.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/deployment.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/service.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/serviceaccount.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/templates/tests/test-connection.yaml create mode 100644 deploy/chatgpt-retrieval-plugin/values.yaml diff --git a/deploy/chatgpt-retrieval-plugin/Chart.yaml b/deploy/chatgpt-retrieval-plugin/Chart.yaml new file mode 100644 index 000000000..f035bc99e --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/Chart.yaml @@ -0,0 +1,12 @@ +apiVersion: v2 +name: chatgpt-retrieval-plugin +description: A demo application for ChatGPT plugin. + +type: application + + +version: 0.1.0 + +appVersion: 0.1.0 + +kubeVersion: '>=1.22.0-0' diff --git a/deploy/chatgpt-retrieval-plugin/templates/NOTES.txt b/deploy/chatgpt-retrieval-plugin/templates/NOTES.txt new file mode 100644 index 000000000..d5038ec48 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/NOTES.txt @@ -0,0 +1,14 @@ +1. Get the application URL by running these commands: +{{- if contains "NodePort" .Values.service.type }} + export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "gptplugin.fullname" . }}) + export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") + echo http://$NODE_IP:$NODE_PORT +{{- else if contains "LoadBalancer" .Values.service.type }} + NOTE: It may take a few minutes for the LoadBalancer IP to be available. + You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "gptplugin.fullname" . }}' + export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "gptplugin.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}") + echo http://$SERVICE_IP:{{ .Values.service.port }} +{{- else if contains "ClusterIP" .Values.service.type }} + kubectl --namespace {{ .Release.Namespace }} port-forward service/gptplugin 8080:8080 + echo "Visit http://127.0.0.1:8080 to use ChatGPT-Retrieval-Plugin demo application." +{{- end }} diff --git a/deploy/chatgpt-retrieval-plugin/templates/_helpers.tpl b/deploy/chatgpt-retrieval-plugin/templates/_helpers.tpl new file mode 100644 index 000000000..87d78c2ab --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/_helpers.tpl @@ -0,0 +1,62 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "gptplugin.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "gptplugin.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "gptplugin.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "gptplugin.labels" -}} +helm.sh/chart: {{ include "gptplugin.chart" . }} +{{ include "gptplugin.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "gptplugin.selectorLabels" -}} +app.kubernetes.io/name: {{ include "gptplugin.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "gptplugin.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "gptplugin.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} diff --git a/deploy/chatgpt-retrieval-plugin/templates/clusterrole.yaml b/deploy/chatgpt-retrieval-plugin/templates/clusterrole.yaml new file mode 100644 index 000000000..83b941682 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/clusterrole.yaml @@ -0,0 +1,8 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "gptplugin.fullname" . }} +rules: + - apiGroups: [""] + resources: ["services", "pods", "secrets"] + verbs: ["get", "list"] diff --git a/deploy/chatgpt-retrieval-plugin/templates/clusterrolebinding.yaml b/deploy/chatgpt-retrieval-plugin/templates/clusterrolebinding.yaml new file mode 100644 index 000000000..e0eca0c92 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/clusterrolebinding.yaml @@ -0,0 +1,12 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "gptplugin.fullname" . }} +subjects: + - kind: ServiceAccount + name: {{ include "gptplugin.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +roleRef: + kind: ClusterRole + name: {{ include "gptplugin.fullname" . }} + apiGroup: rbac.authorization.k8s.io diff --git a/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml new file mode 100644 index 000000000..b92eecb48 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml @@ -0,0 +1,136 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "gptplugin.fullname" . }} + labels: + {{- include "gptplugin.labels" . | nindent 4 }} +spec: + {{- if not .Values.autoscaling.enabled }} + replicas: {{ .Values.replicaCount }} + {{- end }} + selector: + matchLabels: + {{- include "gptplugin.selectorLabels" . | nindent 6 }} + template: + metadata: + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + labels: + {{- include "gptplugin.selectorLabels" . | nindent 8 }} + spec: + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ include "gptplugin.serviceAccountName" . }} + securityContext: + {{- toYaml .Values.podSecurityContext | nindent 8 }} + containers: + - name: {{ .Chart.Name }} + securityContext: + {{- toYaml .Values.securityContext | nindent 12 }} + image: "{{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + ports: + - name: http + containerPort: 8080 + protocol: TCP + livenessProbe: + httpGet: + path: / + port: http + readinessProbe: + httpGet: + path: / + port: http + resources: + {{- toYaml .Values.resources | nindent 12 }} + env: + - name: DATASTORE + value: {{.Values.datastore.DATASTORE | default "milvus" | quote}} + - name: BEARER_TOKEN + value: {{.Values.datastore.BEARER_TOKEN | default | quote}} + - name: OPENAI_API_KEY + value: {{.Values.datastore.OPENAI_API_KEY | default | quote}} + - name: PINECONE_API_KEY + value: {{.Values.datastore.PINECONE_API_KEY | default | quote}} + - name: PINECONE_ENVIRONMENT + value: {{.Values.datastore.PINECONE_ENVIRONMENT | default | quote}} + - name: PINECONE_INDEX + value: {{.Values.datastore.PINECONE_INDEX | default | quote}} + - name: WEAVIATE_HOST + value: {{.Values.datastore.WEAVIATE_HOST | default "http://127.0.0.1" | quote}} + - name: WEAVIATE_PORT + value: {{.Values.datastore.WEAVIATE_PORT | default 8080 | quote}} + - name: WEAVIATE_INDEX + value: {{.Values.datastore.WEAVIATE_INDEX | default "OpenAIDocument" | quote}} + - name: WEAVIATE_USERNAME + value: {{.Values.datastore.WEAVIATE_USERNAME | default | quote}} + - name: WEAVIATE_PASSWORD + value: {{.Values.datastore.WEAVIATE_PASSWORD | default | quote}} + - name: WEAVIATE_SCOPES + value: {{.Values.datastore.WEAVIATE_SCOPES | default | quote}} + - name: WEAVIATE_BATCH_SIZE + value: {{.Values.datastore.WEAVIATE_BATCH_SIZE | default 20 | quote}} + - name: WEAVIATE_BATCH_DYNAMIC + value: {{.Values.datastore.WEAVIATE_BATCH_DYNAMIC | default false | quote}} + - name: WEAVIATE_BATCH_TIMEOUT_RETRIES + value: {{.Values.datastore.WEAVIATE_BATCH_TIMEOUT_RETRIES | default 3 | quote}} + - name: WEAVIATE_BATCH_NUM_WORKERS + value: {{.Values.datastore.WEAVIATE_BATCH_NUM_WORKERS | default 1 | quote}} + - name: ZILLIZ_COLLECTION + value: {{.Values.datastore.ZILLIZ_COLLECTION | default | quote}} + - name: ZILLIZ_URI + value: {{.Values.datastore.ZILLIZ_URI | default | quote}} + - name: ZILLIZ_USER + value: {{.Values.datastore.ZILLIZ_USER | default | quote}} + - name: ZILLIZ_PASSWORD + value: {{.Values.datastore.ZILLIZ_PASSWORD | default | quote}} + - name: MILVUS_COLLECTION + value: {{.Values.datastore.MILVUS_COLLECTION | default | quote}} + - name: MILVUS_HOST + value: {{.Values.datastore.MILVUS_HOST | default "localhost" | quote}} + - name: MILVUS_PORT + value: {{.Values.datastore.MILVUS_PORT | default 19530 | quote}} + - name: MILVUS_USER + value: {{.Values.datastore.MILVUS_USER | default | quote}} + - name: MILVUS_PASSWORD + value: {{.Values.datastore.MILVUS_PASSWORD | default | quote}} + - name: QDRANT_URL + value: {{.Values.datastore.QDRANT_URL | default "http://localhost" | quote}} + - name: QDRANT_PORT + value: {{.Values.datastore.QDRANT_PORT | default 6333 | quote}} + - name: QDRANT_GRPC_PORT + value: {{.Values.datastore.QDRANT_GRPC_PORT | default 6334 | quote}} + - name: QDRANT_API_KEY + value: {{.Values.datastore.QDRANT_API_KEY | default | quote}} + - name: QDRANT_COLLECTION + value: {{.Values.datastore.QDRANT_COLLECTION | default "document_chunks" | quote}} + - name: REDIS_HOST + value: {{.Values.datastore.REDIS_HOST | default "localhost" | quote}} + - name: REDIS_PORT + value: {{.Values.datastore.REDIS_PORT | default 6379 | quote}} + - name: REDIS_PASSWORD + value: {{.Values.datastore.REDIS_PASSWORD | default | quote}} + - name: REDIS_INDEX_NAME + value: {{.Values.datastore.REDIS_INDEX_NAME | default "index" | quote}} + - name: REDIS_DOC_PREFIX + value: {{.Values.datastore.REDIS_DOC_PREFIX | default "doc" | quote}} + - name: REDIS_DISTANCE_METRIC + value: {{.Values.datastore.REDIS_DISTANCE_METRIC | default "COSINE" | quote}} + - name: REDIS_INDEX_TYPE + value: {{.Values.datastore.REDIS_INDEX_TYPE | default "FLAT" | quote}} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/deploy/chatgpt-retrieval-plugin/templates/service.yaml b/deploy/chatgpt-retrieval-plugin/templates/service.yaml new file mode 100644 index 000000000..334d0cc8a --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "gptplugin.fullname" . }} + labels: + {{- include "gptplugin.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: http + protocol: TCP + name: http + selector: + {{- include "gptplugin.selectorLabels" . | nindent 4 }} diff --git a/deploy/chatgpt-retrieval-plugin/templates/serviceaccount.yaml b/deploy/chatgpt-retrieval-plugin/templates/serviceaccount.yaml new file mode 100644 index 000000000..3d7263e79 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "gptplugin.serviceAccountName" . }} + labels: + {{- include "gptplugin.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/deploy/chatgpt-retrieval-plugin/templates/tests/test-connection.yaml b/deploy/chatgpt-retrieval-plugin/templates/tests/test-connection.yaml new file mode 100644 index 000000000..4a648ae63 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/tests/test-connection.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Pod +metadata: + name: "{{ include "gptplugin.fullname" . }}-test-connection" + labels: + {{- include "gptplugin.labels" . | nindent 4 }} + annotations: + "helm.sh/hook": test +spec: + containers: + - name: wget + image: busybox + command: ['wget'] + args: ['{{ include "gptplugin.fullname" . }}:{{ .Values.service.port }}'] + restartPolicy: Never diff --git a/deploy/chatgpt-retrieval-plugin/values.yaml b/deploy/chatgpt-retrieval-plugin/values.yaml new file mode 100644 index 000000000..91eabe370 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/values.yaml @@ -0,0 +1,162 @@ +# Default values for gptplugin. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +replicaCount: 1 + +image: + registry: registry.cn-hangzhou.aliyuncs.com + repository: apecloud/chatgpt-retrieval-plugin + pullPolicy: IfNotPresent + # Overrides the image tag whose default is the chart appVersion. + tag: "latest" + +imagePullSecrets: [] +nameOverride: "" +fullnameOverride: "" + +serviceAccount: + # Specifies whether a service account should be created + create: true + # Annotations to add to the service account + annotations: {} + # The name of the service account to use. + # If not set and create is true, a name is generated using the fullname template + name: "" + +podAnnotations: {} + +podSecurityContext: {} + # fsGroup: 2000 + +securityContext: {} + # capabilities: + # drop: + # - ALL + # readOnlyRootFilesystem: true + # runAsNonRoot: true + # runAsUser: 1000 + +service: + type: ClusterIP + port: 8080 + +datastore: + # in list of (pinecone, weaviate, zilliz, milvus, qdrant, redis) + DATASTORE: + # Yes Your secret token to protect the local plugin API + BEARER_TOKEN: + # Your remote OpenAI API key + OPENAI_API_KEY: + + # Required Your Pinecone API key, found in the Pinecone console (https://app.pinecone.io/) + PINECONE_API_KEY: + # Required Your Pinecone environment, found in the Pinecone console, e.g. us-west1-gcp, us-east-1-aws, etc. + PINECONE_ENVIRONMENT: + # Required Your chosen Pinecone index name. Note: Index name must consist of lower case alphanumeric characters or '-' + PINECONE_INDEX: + + # Optional Your Weaviate instance host address default: http://127.0.0.1 + WEAVIATE_HOST: http://127.0.0.1 + # Optional Your Weaviate port number default: 8080 + WEAVIATE_PORT: 8080 + # Optional Your chosen Weaviate class/collection name to store your documents default: OpenAIDocument + WEAVIATE_INDEX: OpenAIDocument + # Required Your OIDC or WCS username + WEAVIATE_USERNAME: + # Required Your OIDC or WCS password + WEAVIATE_PASSWORD: + # Optional Space-separated list of scopes + WEAVIATE_SCOPES: + # Optional Number of insert/updates per batch operation default: 20 + WEAVIATE_BATCH_SIZE: 20 + # Optional If lets the batch process decide the batch size default: False + WEAVIATE_BATCH_DYNAMIC: False + # Optional Number of retry-on-timeout attempts default: 3 + WEAVIATE_BATCH_TIMEOUT_RETRIES: 3 + # Optional The max number of concurrent threads to run batch operations, default: 1 + WEAVIATE_BATCH_NUM_WORKERS: 1 + + # Optional Zilliz collection name. Defaults to a random UUID + ZILLIZ_COLLECTION: + # Required URI for the Zilliz instance + ZILLIZ_URI: + # Required Zilliz username + ZILLIZ_USER: + # Required Zilliz password + ZILLIZ_PASSWORD: + + # Optional Milvus collection name, defaults to a random UUID + MILVUS_COLLECTION: + # Optional Milvus host IP, defaults to localhost + MILVUS_HOST: localhost + # Optional Milvus port, defaults to 19530 + MILVUS_PORT: 19530 + # Optional Milvus username if RBAC is enabled, default: None + MILVUS_USER: + # Optional Milvus password if required, defaults: None + MILVUS_PASSWORD: + + # Required Qdrant instance URL http://localhost + QDRANT_URL: http://localhost + # Optional TCP port for Qdrant HTTP communication 6333 + QDRANT_PORT: 6333 + # Optional TCP port for Qdrant GRPC communication 6334 + QDRANT_GRPC_PORT: 6334 + # Optional Qdrant API key for Qdrant Cloud + QDRANT_API_KEY: + # Optional Qdrant collection name document_chunks + QDRANT_COLLECTION: document_chunks + + # Optional Redis host url default: localhost + REDIS_HOST: localhost + # Optional Redis port default:6379 + REDIS_PORT: 6379 + # Optional Redis password default:none + REDIS_PASSWORD: + # Optional Redis vector index name default:index + REDIS_INDEX_NAME: index + # Optional Redis key prefix for the index default:doc + REDIS_DOC_PREFIX: doc + # Optional Vector similarity distance metric default:COSINE + REDIS_DISTANCE_METRIC: COSINE + # Optional Vector index algorithm type default:FLAT + REDIS_INDEX_TYPE: FLAT + +resources: {} + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # limits: + # cpu: 100m + # memory: 128Mi + # requests: + # cpu: 100m + # memory: 128Mi + +autoscaling: + enabled: false + minReplicas: 1 + maxReplicas: 100 + targetCPUUtilizationPercentage: 80 + # targetMemoryUtilizationPercentage: 80 + +nodeSelector: {} + +tolerations: + - key: kb-controller + operator: Equal + value: "true" + effect: NoSchedule + +affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + preference: + matchExpressions: + - key: kb-controller + operator: In + values: + - "true" From c23b99ce6f01e37f163141f1888a125a1d0e413e Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Fri, 31 Mar 2023 09:58:40 +0800 Subject: [PATCH 07/80] feat: support qdrant standalone (#2334) --- Makefile | 6 +- deploy/qdrant-cluster/Chart.yaml | 9 + deploy/qdrant-cluster/templates/NOTES.txt | 2 + deploy/qdrant-cluster/templates/_helpers.tpl | 62 +++++++ deploy/qdrant-cluster/templates/cluster.yaml | 46 +++++ deploy/qdrant-cluster/values.yaml | 29 +++ deploy/qdrant/Chart.yaml | 23 +++ deploy/qdrant/templates/NOTES.txt | 2 + deploy/qdrant/templates/_helpers.tpl | 52 ++++++ .../templates/backuppolicytemplate.yaml | 15 ++ .../qdrant/templates/clusterdefinition.yaml | 97 ++++++++++ deploy/qdrant/templates/clusterversion.yaml | 14 ++ deploy/qdrant/templates/configmap.yaml | 169 ++++++++++++++++++ deploy/qdrant/values.yaml | 25 +++ 14 files changed, 550 insertions(+), 1 deletion(-) create mode 100644 deploy/qdrant-cluster/Chart.yaml create mode 100644 deploy/qdrant-cluster/templates/NOTES.txt create mode 100644 deploy/qdrant-cluster/templates/_helpers.tpl create mode 100644 deploy/qdrant-cluster/templates/cluster.yaml create mode 100644 deploy/qdrant-cluster/values.yaml create mode 100644 deploy/qdrant/Chart.yaml create mode 100644 deploy/qdrant/templates/NOTES.txt create mode 100644 deploy/qdrant/templates/_helpers.tpl create mode 100644 deploy/qdrant/templates/backuppolicytemplate.yaml create mode 100644 deploy/qdrant/templates/clusterdefinition.yaml create mode 100644 deploy/qdrant/templates/clusterversion.yaml create mode 100644 deploy/qdrant/templates/configmap.yaml create mode 100644 deploy/qdrant/values.yaml diff --git a/Makefile b/Makefile index 96df96004..723bae660 100644 --- a/Makefile +++ b/Makefile @@ -394,7 +394,11 @@ bump-chart-ver: \ bump-single-chart-ver.postgresql-patroni-ha \ bump-single-chart-ver.postgresql-patroni-ha-cluster \ bump-single-chart-ver.redis \ - bump-single-chart-ver.redis-cluster + bump-single-chart-ver.redis-cluster \ + bump-single-chart-ver.milvus \ + bump-single-chart-ver.qdrant \ + bump-single-chart-ver.qdrant-cluster \ + bump-single-chart-ver.chatgpt-retrieval-plugin bump-chart-ver: ## Bump helm chart version. LOADBALANCER_CHART_VERSION= diff --git a/deploy/qdrant-cluster/Chart.yaml b/deploy/qdrant-cluster/Chart.yaml new file mode 100644 index 000000000..601fdbf63 --- /dev/null +++ b/deploy/qdrant-cluster/Chart.yaml @@ -0,0 +1,9 @@ +apiVersion: v2 +name: qdrant-cluster +description: A Qdrant cluster Helm chart for KubeBlocks. + +type: application + +version: 0.1.0 + +appVersion: "1.1.0" diff --git a/deploy/qdrant-cluster/templates/NOTES.txt b/deploy/qdrant-cluster/templates/NOTES.txt new file mode 100644 index 000000000..c3b3453e3 --- /dev/null +++ b/deploy/qdrant-cluster/templates/NOTES.txt @@ -0,0 +1,2 @@ +1. Get the application URL by running these commands: + diff --git a/deploy/qdrant-cluster/templates/_helpers.tpl b/deploy/qdrant-cluster/templates/_helpers.tpl new file mode 100644 index 000000000..2ae432f3e --- /dev/null +++ b/deploy/qdrant-cluster/templates/_helpers.tpl @@ -0,0 +1,62 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "qdrant.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "qdrant.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "qdrant.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "qdrant.labels" -}} +helm.sh/chart: {{ include "qdrant.chart" . }} +{{ include "qdrant.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "qdrant.selectorLabels" -}} +app.kubernetes.io/name: {{ include "qdrant.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "qdrant.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "qdrant.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} diff --git a/deploy/qdrant-cluster/templates/cluster.yaml b/deploy/qdrant-cluster/templates/cluster.yaml new file mode 100644 index 000000000..e088659e1 --- /dev/null +++ b/deploy/qdrant-cluster/templates/cluster.yaml @@ -0,0 +1,46 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: Cluster +metadata: + name: {{ .Release.Name }} + labels: {{ include "qdrant.labels" . | nindent 4 }} +spec: + clusterDefinitionRef: qdrant-standalone # ref clusterdefinition.name + clusterVersionRef: qdrant-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} # ref clusterversion.name + terminationPolicy: {{ .Values.terminationPolicy }} + affinity: + {{- with .Values.topologyKeys }} + topologyKeys: {{ . | toYaml | nindent 6 }} + {{- end }} + {{- with $.Values.tolerations }} + tolerations: {{ . | toYaml | nindent 4 }} + {{- end }} + componentSpecs: + - name: qdrant # user-defined + componentDefRef: qdrant # ref clusterdefinition components.name + monitor: {{ .Values.monitor.enabled | default false }} + replicas: {{ .Values.replicaCount | default 1 }} + {{- with .Values.resources }} + resources: + {{- with .limits }} + limits: + cpu: {{ .cpu | quote }} + memory: {{ .memory | quote }} + {{- end }} + {{- with .requests }} + requests: + cpu: {{ .cpu | quote }} + memory: {{ .memory | quote }} + {{- end }} + {{- end }} + {{- if .Values.persistence.enabled }} + volumeClaimTemplates: + - name: data # ref clusterdefinition components.containers.volumeMounts.name + spec: + storageClassName: {{ .Values.persistence.data.storageClassName }} + accessModes: + - ReadWriteOnce + resources: + requests: + storage: {{ .Values.persistence.data.size }} + {{- end }} + diff --git a/deploy/qdrant-cluster/values.yaml b/deploy/qdrant-cluster/values.yaml new file mode 100644 index 000000000..3f14e3c39 --- /dev/null +++ b/deploy/qdrant-cluster/values.yaml @@ -0,0 +1,29 @@ +# Default values for wesqlcluster. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +replicaCount: 1 +terminationPolicy: Delete + +clusterVersionOverride: "" + +monitor: + enabled: false + +resources: { } + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + + # limits: + # cpu: 500m + # memory: 2Gi + # requests: + # cpu: 100m + # memory: 1Gi +persistence: + enabled: true + data: + storageClassName: + size: 10Gi diff --git a/deploy/qdrant/Chart.yaml b/deploy/qdrant/Chart.yaml new file mode 100644 index 000000000..4a278bd3f --- /dev/null +++ b/deploy/qdrant/Chart.yaml @@ -0,0 +1,23 @@ +apiVersion: v2 +name: qdrant-standalone +description: . + +type: application + +# This is the chart version. +version: 0.1.0 + +# This is the version number of qdrant. +appVersion: "1.1.0" + +home: https://qdrant.tech/ +icon: https://qdrant.tech/images/logo_with_text.svg + + +maintainers: + - name: ApeCloud + url: https://github.com/apecloud/kubeblocks/ + + +sources: + - https://github.com/apecloud/kubeblocks/ diff --git a/deploy/qdrant/templates/NOTES.txt b/deploy/qdrant/templates/NOTES.txt new file mode 100644 index 000000000..c3b3453e3 --- /dev/null +++ b/deploy/qdrant/templates/NOTES.txt @@ -0,0 +1,2 @@ +1. Get the application URL by running these commands: + diff --git a/deploy/qdrant/templates/_helpers.tpl b/deploy/qdrant/templates/_helpers.tpl new file mode 100644 index 000000000..840320090 --- /dev/null +++ b/deploy/qdrant/templates/_helpers.tpl @@ -0,0 +1,52 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "qdrant.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "qdrant.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "qdrant.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "qdrant.labels" -}} +helm.sh/chart: {{ include "qdrant.chart" . }} +{{ include "qdrant.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "qdrant.selectorLabels" -}} +app.kubernetes.io/name: {{ include "qdrant.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + diff --git a/deploy/qdrant/templates/backuppolicytemplate.yaml b/deploy/qdrant/templates/backuppolicytemplate.yaml new file mode 100644 index 000000000..b07273534 --- /dev/null +++ b/deploy/qdrant/templates/backuppolicytemplate.yaml @@ -0,0 +1,15 @@ +apiVersion: dataprotection.kubeblocks.io/v1alpha1 +kind: BackupPolicyTemplate +metadata: + name: backup-policy-template-qdrant + labels: + clusterdefinition.kubeblocks.io/name: qdrant-standalone + {{- include "qdrant.labels" . | nindent 4 }} +spec: + # which backup tool to perform database backup, only support one tool. + backupToolName: volumesnapshot + ttl: 168h0m0s + + credentialKeyword: + userKeyword: username + passwordKeyword: password diff --git a/deploy/qdrant/templates/clusterdefinition.yaml b/deploy/qdrant/templates/clusterdefinition.yaml new file mode 100644 index 000000000..07dff600e --- /dev/null +++ b/deploy/qdrant/templates/clusterdefinition.yaml @@ -0,0 +1,97 @@ +--- +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterDefinition +metadata: + name: qdrant-standalone + labels: + {{- include "qdrant.labels" . | nindent 4 }} +spec: + connectionCredential: + username: root + password: "$(RANDOM_PASSWD)" + endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-qdrant)" + host: "$(SVC_FQDN)" + port: "$(SVC_PORT_tcp-qdrant)" + componentDefs: + - name: qdrant + workloadType: Stateful + characterType: qdrant + probes: + monitor: + builtIn: false + exporterConfig: + scrapePath: /metrics + scrapePort: 9187 + logConfigs: + configSpecs: + - name: qdrant-standalone-config-template + templateRef: qdrant-standalone-config-template + volumeName: qdrant-config + namespace: {{ .Release.Namespace }} + service: + ports: + - name: tcp-qdrant + protocol: TCP + port: 6333 + targetPort: tcp-qdrant + - name: grpc-qdrant + protocol: TCP + port: 6334 + targetPort: grpc-qdrant + volumeTypes: + - name: data + type: data + podSpec: + securityContext: + fsGroup: 1001 + containers: + - name: qdrant + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + securityContext: + runAsUser: 0 + livenessProbe: + failureThreshold: 3 + httpGet: + path: / + port: 6333 + scheme: HTTP + periodSeconds: 15 + successThreshold: 1 + timeoutSeconds: 10 + readinessProbe: + failureThreshold: 2 + httpGet: + path: / + port: 6333 + scheme: HTTP + initialDelaySeconds: 5 + periodSeconds: 15 + successThreshold: 1 + timeoutSeconds: 3 + startupProbe: + failureThreshold: 18 + httpGet: + path: / + port: 6333 + scheme: HTTP + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 3 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /qdrant/config/ + name: qdrant-config + - mountPath: /qdrant/storage + name: data + dnsPolicy: ClusterFirst + enableServiceLinks: true + ports: + - name: tcp-qdrant + containerPort: 6333 + - name: tcp-metrics + containerPort: 9091 + command: + - ./qdrant + env: + diff --git a/deploy/qdrant/templates/clusterversion.yaml b/deploy/qdrant/templates/clusterversion.yaml new file mode 100644 index 000000000..c57d1e9bc --- /dev/null +++ b/deploy/qdrant/templates/clusterversion.yaml @@ -0,0 +1,14 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterVersion +metadata: + name: qdrant-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} + labels: + {{- include "qdrant.labels" . | nindent 4 }} +spec: + clusterDefinitionRef: qdrant-standalone + componentVersions: + - componentDefRef: qdrant + versionsContext: + containers: + - name: qdrant + image: {{ .Values.images.qdrant.repository }}:{{ default .Chart.AppVersion .Values.images.qdrant.tag }} diff --git a/deploy/qdrant/templates/configmap.yaml b/deploy/qdrant/templates/configmap.yaml new file mode 100644 index 000000000..913fdeef2 --- /dev/null +++ b/deploy/qdrant/templates/configmap.yaml @@ -0,0 +1,169 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: qdrant-standalone-config-template + namespace: {{ .Release.Namespace | quote }} + labels: + {{- include "qdrant.labels" . | nindent 4 }} + +data: + production.yaml: |- + debug: false + log_level: INFO + + service: + host: 0.0.0.0 + http_port: 6333 + grpc_port: 6334 + config.yaml: |- + storage: + # Where to store all the data + storage_path: ./storage + + # Where to store snapshots + snapshots_path: ./snapshots + + # If true - point's payload will not be stored in memory. + # It will be read from the disk every time it is requested. + # This setting saves RAM by (slightly) increasing the response time. + # Note: those payload values that are involved in filtering and are indexed - remain in RAM. + on_disk_payload: true + + # Write-ahead-log related configuration + wal: + # Size of a single WAL segment + wal_capacity_mb: 32 + + # Number of WAL segments to create ahead of actual data requirement + wal_segments_ahead: 0 + + # Normal node - receives all updates and answers all queries + node_type: "Normal" + + # Listener node - receives all updates, but does not answer search/read queries + # Useful for setting up a dedicated backup node + # node_type: "Listener" + + performance: + # Number of parallel threads used for search operations. If 0 - auto selection. + max_search_threads: 0 + # Max total number of threads, which can be used for running optimization processes across all collections. + # Note: Each optimization thread will also use `max_indexing_threads` for index building. + # So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads` + max_optimization_threads: 1 + + optimizers: + # The minimal fraction of deleted vectors in a segment, required to perform segment optimization + deleted_threshold: 0.2 + + # The minimal number of vectors in a segment, required to perform segment optimization + vacuum_min_vector_number: 1000 + + # Target amount of segments optimizer will try to keep. + # Real amount of segments may vary depending on multiple parameters: + # - Amount of stored points + # - Current write RPS + # + # It is recommended to select default number of segments as a factor of the number of search threads, + # so that each segment would be handled evenly by one of the threads. + # If `default_segment_number = 0`, will be automatically selected by the number of available CPUs + default_segment_number: 0 + + # Do not create segments larger this size (in KiloBytes). + # Large segments might require disproportionately long indexation times, + # therefore it makes sense to limit the size of segments. + # + # If indexation speed have more priority for your - make this parameter lower. + # If search speed is more important - make this parameter higher. + # Note: 1Kb = 1 vector of size 256 + # If not set, will be automatically selected considering the number of available CPUs. + max_segment_size_kb: null + + # Maximum size (in KiloBytes) of vectors to store in-memory per segment. + # Segments larger than this threshold will be stored as read-only memmaped file. + # To enable memmap storage, lower the threshold + # Note: 1Kb = 1 vector of size 256 + # If not set, mmap will not be used. + memmap_threshold_kb: null + + # Maximum size (in KiloBytes) of vectors allowed for plain index. + # Default value based on https://github.com/google-research/google-research/blob/master/scann/docs/algorithms.md + # Note: 1Kb = 1 vector of size 256 + indexing_threshold_kb: 20000 + + # Interval between forced flushes. + flush_interval_sec: 5 + + # Max number of threads, which can be used for optimization per collection. + # Note: Each optimization thread will also use `max_indexing_threads` for index building. + # So total number of threads used for optimization will be `max_optimization_threads * max_indexing_threads` + # If `max_optimization_threads = 0`, optimization will be disabled. + max_optimization_threads: 1 + + # Default parameters of HNSW Index. Could be overridden for each collection individually + hnsw_index: + # Number of edges per node in the index graph. Larger the value - more accurate the search, more space required. + m: 16 + # Number of neighbours to consider during the index building. Larger the value - more accurate the search, more time required to build index. + ef_construct: 100 + # Minimal size (in KiloBytes) of vectors for additional payload-based indexing. + # If payload chunk is smaller than `full_scan_threshold_kb` additional indexing won't be used - + # in this case full-scan search should be preferred by query planner and additional indexing is not required. + # Note: 1Kb = 1 vector of size 256 + full_scan_threshold_kb: 10000 + # Number of parallel threads used for background index building. If 0 - auto selection. + max_indexing_threads: 0 + # Store HNSW index on disk. If set to false, index will be stored in RAM. Default: false + on_disk: false + # Custom M param for hnsw graph built for payload index. If not set, default M will be used. + payload_m: null + + service: + + # Maximum size of POST data in a single request in megabytes + max_request_size_mb: 32 + + # Number of parallel workers used for serving the api. If 0 - equal to the number of available cores. + # If missing - Same as storage.max_search_threads + max_workers: 0 + + # Host to bind the service on + host: 0.0.0.0 + + # HTTP port to bind the service on + http_port: 6333 + + # gRPC port to bind the service on. + # If `null` - gRPC is disabled. Default: null + grpc_port: 6334 + # Uncomment to enable gRPC: + # grpc_port: 6334 + + # Enable CORS headers in REST API. + # If enabled, browsers would be allowed to query REST endpoints regardless of query origin. + # More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS + # Default: true + enable_cors: true + + cluster: + # Use `enabled: true` to run Qdrant in distributed deployment mode + enabled: false + + # Configuration of the inter-cluster communication + p2p: + # Port for internal communication between peers + port: 6335 + + # Configuration related to distributed consensus algorithm + consensus: + # How frequently peers should ping each other. + # Setting this parameter to lower value will allow consensus + # to detect disconnected nodes earlier, but too frequent + # tick period may create significant network and CPU overhead. + # We encourage you NOT to change this parameter unless you know what you are doing. + tick_period_ms: 100 + + + # Set to true to prevent service from sending usage statistics to the developers. + # Read more: https://qdrant.tech/documentation/telemetry + telemetry_disabled: false \ No newline at end of file diff --git a/deploy/qdrant/values.yaml b/deploy/qdrant/values.yaml new file mode 100644 index 000000000..7843eb6f2 --- /dev/null +++ b/deploy/qdrant/values.yaml @@ -0,0 +1,25 @@ +# Default values for qdrant. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + + +clusterVersionOverride: "" +nameOverride: "" +fullnameOverride: "" + + +## @param commonLabels Labels to add to all deployed objects +## +commonLabels: {} + +## @param application images +## +images: + pullPolicy: IfNotPresent + qdrant: + repository: docker.io/qdrant/qdrant + tag: latest + +## @param debugEnabled enables containers' debug logging +## +debugEnabled: true \ No newline at end of file From e7537d21f9b512b159715d5c5892a13201f56427 Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Fri, 31 Mar 2023 09:59:12 +0800 Subject: [PATCH 08/80] feat: support milvus standalone mode (#2310) --- deploy/milvus/Chart.yaml | 23 ++ deploy/milvus/configs/milvus-user.yaml.tpl | 20 ++ deploy/milvus/templates/NOTES.txt | 2 + deploy/milvus/templates/_helpers.tpl | 73 +++++ .../templates/backuppolicytemplate.yaml | 15 + .../milvus/templates/clusterdefinition.yaml | 303 ++++++++++++++++++ deploy/milvus/templates/clusterversion.yaml | 27 ++ deploy/milvus/templates/configconstraint.yaml | 15 + deploy/milvus/templates/configmap.yaml | 11 + deploy/milvus/values.yaml | 34 ++ 10 files changed, 523 insertions(+) create mode 100644 deploy/milvus/Chart.yaml create mode 100644 deploy/milvus/configs/milvus-user.yaml.tpl create mode 100644 deploy/milvus/templates/NOTES.txt create mode 100644 deploy/milvus/templates/_helpers.tpl create mode 100644 deploy/milvus/templates/backuppolicytemplate.yaml create mode 100644 deploy/milvus/templates/clusterdefinition.yaml create mode 100644 deploy/milvus/templates/clusterversion.yaml create mode 100644 deploy/milvus/templates/configconstraint.yaml create mode 100644 deploy/milvus/templates/configmap.yaml create mode 100644 deploy/milvus/values.yaml diff --git a/deploy/milvus/Chart.yaml b/deploy/milvus/Chart.yaml new file mode 100644 index 000000000..b7750366c --- /dev/null +++ b/deploy/milvus/Chart.yaml @@ -0,0 +1,23 @@ +apiVersion: v2 +name: milvus-standalone +description: . + +type: application + +# This is the chart version +version: 0.1.0 + +# This is the version number of milvus +appVersion: "2.2.4" + +home: https://milvus.io/ +icon: https://github.com/milvus-io/artwork/blob/master/horizontal/color/milvus-horizontal-color.png + + +maintainers: + - name: ApeCloud + url: https://github.com/apecloud/kubeblocks/ + + +sources: + - https://github.com/apecloud/kubeblocks/ diff --git a/deploy/milvus/configs/milvus-user.yaml.tpl b/deploy/milvus/configs/milvus-user.yaml.tpl new file mode 100644 index 000000000..c3a3ff4a1 --- /dev/null +++ b/deploy/milvus/configs/milvus-user.yaml.tpl @@ -0,0 +1,20 @@ +{{- $clusterName := $.cluster.metadata.name }} +{{- $namespace := $.cluster.metadata.namespace }} +{{- $userName := getEnvByName ( index $.podSpec.containers 0 ) "MINIO_ACCESS_KEY" }} +{{- $secret := getEnvByName ( index $.podSpec.containers 0 ) "MINIO_SECRET_KEY" | b64dec }} + +etcd: + endpoints: + - {{$clusterName}}-etcd-headless.{{$namespace}}.svc.cluster.local:2379 + rootPath: {{$clusterName}} +messageQueue: rocksmq +minio: + accessKeyID: minioadmin + address: {{$clusterName}}-minio-headless.{{$namespace}}.svc.cluster.local + bucketName: {{$clusterName}} + port: 9000 + secretAccessKey: minioadmin +msgChannel: + chanNamePrefix: + cluster: {{$clusterName}} + rocksmq: {} \ No newline at end of file diff --git a/deploy/milvus/templates/NOTES.txt b/deploy/milvus/templates/NOTES.txt new file mode 100644 index 000000000..c3b3453e3 --- /dev/null +++ b/deploy/milvus/templates/NOTES.txt @@ -0,0 +1,2 @@ +1. Get the application URL by running these commands: + diff --git a/deploy/milvus/templates/_helpers.tpl b/deploy/milvus/templates/_helpers.tpl new file mode 100644 index 000000000..637112ebd --- /dev/null +++ b/deploy/milvus/templates/_helpers.tpl @@ -0,0 +1,73 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "milvus.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "milvus.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "milvus.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "milvus.labels" -}} +helm.sh/chart: {{ include "milvus.chart" . }} +{{ include "milvus.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "milvus.selectorLabels" -}} +app.kubernetes.io/name: {{ include "milvus.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use + +{{- define "milvus.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "milvus.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{- define "milvus.checkerServiceAccountName" -}} +{{- if .Values.installDependencies.enable }} +{{- if .Values.installDependencies.serviceAccount.create }} +{{- default (printf "%s-checker" (include "milvus.fullname" .)) .Values.installDependencies.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} +{{- end }} +*/}} \ No newline at end of file diff --git a/deploy/milvus/templates/backuppolicytemplate.yaml b/deploy/milvus/templates/backuppolicytemplate.yaml new file mode 100644 index 000000000..38c1d98c3 --- /dev/null +++ b/deploy/milvus/templates/backuppolicytemplate.yaml @@ -0,0 +1,15 @@ +apiVersion: dataprotection.kubeblocks.io/v1alpha1 +kind: BackupPolicyTemplate +metadata: + name: backup-policy-template-milvus + labels: + clusterdefinition.kubeblocks.io/name: milvus-standalone + {{- include "milvus.labels" . | nindent 4 }} +spec: + # which backup tool to perform database backup, only support one tool. + backupToolName: volumesnapshot + ttl: 168h0m0s + + credentialKeyword: + userKeyword: username + passwordKeyword: password diff --git a/deploy/milvus/templates/clusterdefinition.yaml b/deploy/milvus/templates/clusterdefinition.yaml new file mode 100644 index 000000000..db5a18da3 --- /dev/null +++ b/deploy/milvus/templates/clusterdefinition.yaml @@ -0,0 +1,303 @@ +--- +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterDefinition +metadata: + name: milvus-standalone + labels: + {{- include "milvus.labels" . | nindent 4 }} +spec: + connectionCredential: + username: root + password: "$(RANDOM_PASSWD)" + endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-milvus)" + host: "$(SVC_FQDN)" + port: "$(SVC_PORT_tcp-milvus)" + accesskey: minioadmin + secretkey: minioadmin + componentDefs: + - name: milvus + workloadType: Stateful + characterType: milvus + probes: + monitor: + builtIn: false + exporterConfig: + scrapePath: /metrics + scrapePort: 9187 + logConfigs: + configSpecs: + - name: milvus-standalone-config-template + templateRef: milvus-standalone-config-template + volumeName: milvus-config + namespace: {{.Release.Namespace}} + service: + ports: + - name: tcp-milvus + protocol: TCP + port: 19530 + targetPort: tcp-milvus + volumeTypes: + - name: data + type: data + podSpec: + securityContext: + fsGroup: 1001 + containers: + - name: milvus + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + securityContext: + runAsUser: 0 + livenessProbe: + failureThreshold: 3 + httpGet: + path: /healthz + port: 9091 + scheme: HTTP + periodSeconds: 15 + successThreshold: 1 + timeoutSeconds: 10 + readinessProbe: + failureThreshold: 2 + httpGet: + path: /healthz + port: 9091 + scheme: HTTP + initialDelaySeconds: 5 + periodSeconds: 15 + successThreshold: 1 + timeoutSeconds: 3 + startupProbe: + failureThreshold: 18 + httpGet: + path: /healthz + port: 9091 + scheme: HTTP + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 3 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /milvus/configs/user.yaml + name: milvus-config + readOnly: true + subPath: user.yaml + - mountPath: /milvus/tools + name: tools + dnsPolicy: ClusterFirst + enableServiceLinks: true + ports: + - name: tcp-milvus + containerPort: 19530 + - name: tcp-metrics + containerPort: 9091 + args: + - /milvus/tools/run.sh + - milvus + - run + - standalone + env: + - name: CACHE_SIZE + valueFrom: + resourceFieldRef: + divisor: 1Gi + resource: limits.memory + - name: MINIO_ACCESS_KEY + valueFrom: + secretKeyRef: + key: accesskey + name: $(CONN_CREDENTIAL_SECRET_NAME) + - name: MINIO_SECRET_KEY + valueFrom: + secretKeyRef: + key: secretkey + name: $(CONN_CREDENTIAL_SECRET_NAME) + initContainers: + - name: milvus-init + command: + - /cp + - /run.sh,/merge + - /milvus/tools/run.sh,/milvus/tools/merge + image: milvusdb/milvus-operator:v0.7.8 + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + resources: {} + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /milvus/tools + name: tools + serviceAccountName: default + terminationGracePeriodSeconds: 300 + volumes: + - emptyDir: {} + name: tools + + - name: etcd + workloadType: Stateful + characterType: etcd + probes: + logconfigs: + configSpecs: + scriptsSpecs: + podSpec: + containers: + - name: etcd + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + securityContext: + runAsNonRoot: true + runAsUser: 1001 + livenessProbe: + exec: + command: + - /opt/bitnami/scripts/etcd/healthcheck.sh + failureThreshold: 5 + initialDelaySeconds: 60 + periodSeconds: 30 + successThreshold: 1 + timeoutSeconds: 5 + readinessProbe: + exec: + command: + - /opt/bitnami/scripts/etcd/healthcheck.sh + failureThreshold: 5 + initialDelaySeconds: 60 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 5 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /bitnami/etcd + name: data + dnsPolicy: ClusterFirst + ports: + - name: client + containerPort: 2379 + protocol: TCP + - name: peer + containerPort: 2380 + protocol: TCP + env: + - name: BITNAMI_DEBUG + value: "false" + - name: MY_POD_IP + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.podIP + - name: MY_POD_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.name + - name: ETCDCTL_API + value: "3" + - name: ETCD_ON_K8S + value: "yes" + - name: ETCD_START_FROM_SNAPSHOT + value: "no" + - name: ETCD_DISASTER_RECOVERY + value: "no" + - name: ETCD_NAME + value: $(MY_POD_NAME) + - name: ETCD_DATA_DIR + value: /bitnami/etcd/data + - name: ETCD_LOG_LEVEL + value: info + - name: ALLOW_NONE_AUTHENTICATION + value: "yes" + - name: ETCD_ADVERTISE_CLIENT_URLS + value: http://$(MY_POD_NAME).$(KB_CLUSTER_NAME)-etcd-headless.default.svc.cluster.local:2379 + - name: ETCD_LISTEN_CLIENT_URLS + value: http://0.0.0.0:2379 + - name: ETCD_INITIAL_ADVERTISE_PEER_URLS + value: http://$(MY_POD_NAME).$(KB_CLUSTER_NAME)-etcd-headless.default.svc.cluster.local:2380 + - name: ETCD_LISTEN_PEER_URLS + value: http://0.0.0.0:2380 + - name: ETCD_AUTO_COMPACTION_MODE + value: revision + - name: ETCD_AUTO_COMPACTION_RETENTION + value: "1000" + - name: ETCD_QUOTA_BACKEND_BYTES + value: "4294967296" + - name: ETCD_HEARTBEAT_INTERVAL + value: "500" + - name: ETCD_ELECTION_TIMEOUT + value: "2500" + + - name: minio + workloadType: Stateful + characterType: minio + probes: + logconfigs: + configSpecs: + scriptsSpecs: + service: + ports: + - name: http + protocol: TCP + port: 9000 + targetPort: 9000 + volumeTypes: + - name: data + type: data + podSpec: + containers: + - name: minio + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + securityContext: + fsGroup: 1000 + runAsGroup: 1000 + runAsUser: 1000 + livenessProbe: + failureThreshold: 5 + httpGet: + path: /minio/health/live + port: 9000 + scheme: HTTP + initialDelaySeconds: 5 + periodSeconds: 5 + successThreshold: 1 + timeoutSeconds: 5 + readinessProbe: + failureThreshold: 5 + initialDelaySeconds: 5 + periodSeconds: 5 + successThreshold: 1 + tcpSocket: + port: 9000 + timeoutSeconds: 1 + startupProbe: + failureThreshold: 60 + periodSeconds: 10 + successThreshold: 1 + tcpSocket: + port: 9000 + timeoutSeconds: 5 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + serviceAccountName: "$(KB_CLUSTER_NAME-minio)" + volumeMounts: + - mountPath: /export + name: export + volumes: + - name: minio-user + secret: + defaultMode: 420 + secretName: $(CONN_CREDENTIAL_SECRET_NAME) + command: + - /bin/sh + - -ce + - /usr/bin/docker-entrypoint.sh minio -S /etc/minio/certs/ server /export + env: + - name: MINIO_ACCESS_KEY + valueFrom: + secretKeyRef: + key: accesskey + name: $(CONN_CREDENTIAL_SECRET_NAME) + - name: MINIO_SECRET_KEY + valueFrom: + secretKeyRef: + key: secretkey + name: $(CONN_CREDENTIAL_SECRET_NAME) + diff --git a/deploy/milvus/templates/clusterversion.yaml b/deploy/milvus/templates/clusterversion.yaml new file mode 100644 index 000000000..f13a210cd --- /dev/null +++ b/deploy/milvus/templates/clusterversion.yaml @@ -0,0 +1,27 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterVersion +metadata: + name: milvus-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} + labels: + {{- include "milvus.labels" . | nindent 4 }} +spec: + clusterDefinitionRef: milvus-standalone + componentVersions: + - componentDefRef: minio + versionsContext: + containers: + - name: minio + image: {{ .Values.images.minio.repository }}:{{ default .Chart.AppVersion .Values.images.minio.tag }} + - componentDefRef: etcd + versionsContext: + containers: + - name: etcd + image: {{ .Values.images.etcd.repository }}:{{ default .Chart.AppVersion .Values.images.etcd.tag }} + - componentDefRef: milvus + versionsContext: + initContainers: + - name: milvus-init + image: {{ .Values.images.milvus_init.repository }}:{{ default .Chart.AppVersion .Values.images.milvus_init.tag }} + containers: + - name: milvus + image: {{ .Values.images.milvus.repository }}:{{ default .Chart.AppVersion .Values.images.milvus.tag }} diff --git a/deploy/milvus/templates/configconstraint.yaml b/deploy/milvus/templates/configconstraint.yaml new file mode 100644 index 000000000..c1b159eab --- /dev/null +++ b/deploy/milvus/templates/configconstraint.yaml @@ -0,0 +1,15 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ConfigConstraint +metadata: + name: milvus-cc + labels: + {{- include "milvus.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +spec: + formatterConfig: + format: yaml \ No newline at end of file diff --git a/deploy/milvus/templates/configmap.yaml b/deploy/milvus/templates/configmap.yaml new file mode 100644 index 000000000..864337864 --- /dev/null +++ b/deploy/milvus/templates/configmap.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: milvus-standalone-config-template + namespace: {{ .Release.Namespace | quote }} + labels: + {{- include "milvus.labels" . | nindent 4 }} + +data: + user.yaml: |- + {{- .Files.Get "configs/milvus-user.yaml.tpl" | nindent 4 }} \ No newline at end of file diff --git a/deploy/milvus/values.yaml b/deploy/milvus/values.yaml new file mode 100644 index 000000000..7d71b424d --- /dev/null +++ b/deploy/milvus/values.yaml @@ -0,0 +1,34 @@ +# Default values for milvus. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + + +clusterVersionOverride: "" +nameOverride: "" +fullnameOverride: "" + + +## @param commonLabels Labels to add to all deployed objects +## +commonLabels: {} + +## @param application images +## +images: + pullPolicy: IfNotPresent + minio: + repository: docker.io/minio/minio + tag: RELEASE.2022-03-17T06-34-49Z + etcd: + repository: docker.io/milvusdb/etcd + tag: 3.5.5-r2 + milvus_init: + repository: docker.io/milvusdb/milvus-operator + tag: v0.7.8 + milvus: + repository: docker.io/milvusdb/milvus + tag: v2.2.4 + +## @param debugEnabled enables containers' debug logging +## +debugEnabled: true \ No newline at end of file From 1c8b4df3b4b5d0d81c736f0e630cf3cb75cd8857 Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Fri, 31 Mar 2023 10:15:23 +0800 Subject: [PATCH 09/80] fix: set default storage size to 10Gi for TKE (#2317) --- .../templates/addons/prometheus-addon.yaml | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/deploy/helm/templates/addons/prometheus-addon.yaml b/deploy/helm/templates/addons/prometheus-addon.yaml index 9488e3ddf..56c0f2a23 100644 --- a/deploy/helm/templates/addons/prometheus-addon.yaml +++ b/deploy/helm/templates/addons/prometheus-addon.yaml @@ -72,14 +72,12 @@ spec: requests: storage: 4Gi + # for ACK, the smallest storage size is 20Gi, the format of GitVersion is v1.24.6-aliyun.1 - selectors: - key: KubeGitVersion operator: Contains values: - # for ACK, the smallest storage size is 20Gi, the format of GitVersion is v1.24.6-aliyun.1 - aliyun - # for TKE, the smallest storage size is 10Gi, we also use 20Gi storage, the format of GitVersion is v1.24.4-tke.5 - - tke replicas: 1 storageClass: alicloud-disk-efficiency resources: @@ -96,5 +94,25 @@ spec: requests: storage: 20Gi + # for TKE, the smallest storage size is 10Gi, the format of GitVersion is v1.24.4-tke.5 + - selectors: + - key: KubeGitVersion + operator: Contains + values: + - tke + replicas: 1 + resources: + requests: + storage: 10Gi + memory: 512Mi + limits: + memory: 4Gi + extras: + - name: alertmanager + replicas: 1 + resources: + requests: + storage: 10Gi + installable: autoInstall: {{ .Values.prometheus.enabled }} \ No newline at end of file From 315731b1d02aee6e037247866182f47d6449c4ba Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Fri, 31 Mar 2023 10:16:27 +0800 Subject: [PATCH 10/80] fix: ifconfig url use https (#2340) --- internal/cli/util/util.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/cli/util/util.go b/internal/cli/util/util.go index 3821d7aa0..5609c9756 100644 --- a/internal/cli/util/util.go +++ b/internal/cli/util/util.go @@ -522,7 +522,7 @@ func IsSupportReconfigureParams(tpl appsv1alpha1.ComponentConfigSpec, values map func getIPLocation() (string, error) { client := &http.Client{} - req, err := http.NewRequest("GET", "http://ifconfig.io/country_code", nil) + req, err := http.NewRequest("GET", "https://ifconfig.io/country_code", nil) if err != nil { return "", err } From 8a6258f98180f3345ae67863743113459d00f7e2 Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Fri, 31 Mar 2023 10:38:18 +0800 Subject: [PATCH 11/80] feat: support weaviate standalone (#2345) --- deploy/weaviate-cluster/Chart.yaml | 9 + deploy/weaviate-cluster/templates/NOTES.txt | 2 + .../weaviate-cluster/templates/_helpers.tpl | 62 ++ .../weaviate-cluster/templates/cluster.yaml | 46 + deploy/weaviate-cluster/values.yaml | 29 + deploy/weaviate/Chart.yaml | 23 + deploy/weaviate/templates/NOTES.txt | 2 + deploy/weaviate/templates/_helpers.tpl | 52 + .../templates/backuppolicytemplate.yaml | 15 + .../weaviate/templates/clusterdefinition.yaml | 130 +++ deploy/weaviate/templates/clusterversion.yaml | 14 + deploy/weaviate/templates/configmap.yaml | 20 + .../weaviate/templates/headlessService.yaml | 18 + deploy/weaviate/values.yaml | 897 ++++++++++++++++++ 14 files changed, 1319 insertions(+) create mode 100644 deploy/weaviate-cluster/Chart.yaml create mode 100644 deploy/weaviate-cluster/templates/NOTES.txt create mode 100644 deploy/weaviate-cluster/templates/_helpers.tpl create mode 100644 deploy/weaviate-cluster/templates/cluster.yaml create mode 100644 deploy/weaviate-cluster/values.yaml create mode 100644 deploy/weaviate/Chart.yaml create mode 100644 deploy/weaviate/templates/NOTES.txt create mode 100644 deploy/weaviate/templates/_helpers.tpl create mode 100644 deploy/weaviate/templates/backuppolicytemplate.yaml create mode 100644 deploy/weaviate/templates/clusterdefinition.yaml create mode 100644 deploy/weaviate/templates/clusterversion.yaml create mode 100644 deploy/weaviate/templates/configmap.yaml create mode 100644 deploy/weaviate/templates/headlessService.yaml create mode 100644 deploy/weaviate/values.yaml diff --git a/deploy/weaviate-cluster/Chart.yaml b/deploy/weaviate-cluster/Chart.yaml new file mode 100644 index 000000000..a9395d6dd --- /dev/null +++ b/deploy/weaviate-cluster/Chart.yaml @@ -0,0 +1,9 @@ +apiVersion: v2 +name: weaviate-cluster +description: A weaviate cluster Helm chart for KubeBlocks. + +type: application + +version: 0.1.0 + +appVersion: "1.18.0" diff --git a/deploy/weaviate-cluster/templates/NOTES.txt b/deploy/weaviate-cluster/templates/NOTES.txt new file mode 100644 index 000000000..c3b3453e3 --- /dev/null +++ b/deploy/weaviate-cluster/templates/NOTES.txt @@ -0,0 +1,2 @@ +1. Get the application URL by running these commands: + diff --git a/deploy/weaviate-cluster/templates/_helpers.tpl b/deploy/weaviate-cluster/templates/_helpers.tpl new file mode 100644 index 000000000..82d09cdb0 --- /dev/null +++ b/deploy/weaviate-cluster/templates/_helpers.tpl @@ -0,0 +1,62 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "weaviate.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "weaviate.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "weaviate.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "weaviate.labels" -}} +helm.sh/chart: {{ include "weaviate.chart" . }} +{{ include "weaviate.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "weaviate.selectorLabels" -}} +app.kubernetes.io/name: {{ include "weaviate.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "weaviate.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "weaviate.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} diff --git a/deploy/weaviate-cluster/templates/cluster.yaml b/deploy/weaviate-cluster/templates/cluster.yaml new file mode 100644 index 000000000..0bf4a0804 --- /dev/null +++ b/deploy/weaviate-cluster/templates/cluster.yaml @@ -0,0 +1,46 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: Cluster +metadata: + name: {{ .Release.Name }} + labels: {{ include "weaviate.labels" . | nindent 4 }} +spec: + clusterDefinitionRef: weaviate-standalone # ref clusterdefinition.name + clusterVersionRef: weaviate-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} # ref clusterversion.name + terminationPolicy: {{ .Values.terminationPolicy }} + affinity: + {{- with .Values.topologyKeys }} + topologyKeys: {{ . | toYaml | nindent 6 }} + {{- end }} + {{- with $.Values.tolerations }} + tolerations: {{ . | toYaml | nindent 4 }} + {{- end }} + componentSpecs: + - name: weaviate # user-defined + componentDefRef: weaviate # ref clusterdefinition components.name + monitor: {{ .Values.monitor.enabled | default false }} + replicas: {{ .Values.replicaCount | default 1 }} + {{- with .Values.resources }} + resources: + {{- with .limits }} + limits: + cpu: {{ .cpu | quote }} + memory: {{ .memory | quote }} + {{- end }} + {{- with .requests }} + requests: + cpu: {{ .cpu | quote }} + memory: {{ .memory | quote }} + {{- end }} + {{- end }} + {{- if .Values.persistence.enabled }} + volumeClaimTemplates: + - name: data # ref clusterdefinition components.containers.volumeMounts.name + spec: + storageClassName: {{ .Values.persistence.data.storageClassName }} + accessModes: + - ReadWriteOnce + resources: + requests: + storage: {{ .Values.persistence.data.size }} + {{- end }} + diff --git a/deploy/weaviate-cluster/values.yaml b/deploy/weaviate-cluster/values.yaml new file mode 100644 index 000000000..3f14e3c39 --- /dev/null +++ b/deploy/weaviate-cluster/values.yaml @@ -0,0 +1,29 @@ +# Default values for wesqlcluster. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + +replicaCount: 1 +terminationPolicy: Delete + +clusterVersionOverride: "" + +monitor: + enabled: false + +resources: { } + # We usually recommend not to specify default resources and to leave this as a conscious + # choice for the user. This also increases chances charts run on environments with little + # resources, such as Minikube. If you do want to specify resources, uncomment the following + # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + + # limits: + # cpu: 500m + # memory: 2Gi + # requests: + # cpu: 100m + # memory: 1Gi +persistence: + enabled: true + data: + storageClassName: + size: 10Gi diff --git a/deploy/weaviate/Chart.yaml b/deploy/weaviate/Chart.yaml new file mode 100644 index 000000000..c198dced1 --- /dev/null +++ b/deploy/weaviate/Chart.yaml @@ -0,0 +1,23 @@ +apiVersion: v2 +name: weaviate-standalone +description: . + +type: application + +# This is the chart version. +version: 0.1.0 + +# This is the version number of weaviate. +appVersion: "1.18.0" + +home: https://weaviate.tech/ +icon: https://weaviate.tech/images/logo_with_text.svg + + +maintainers: + - name: ApeCloud + url: https://github.com/apecloud/kubeblocks/ + + +sources: + - https://github.com/apecloud/kubeblocks/ diff --git a/deploy/weaviate/templates/NOTES.txt b/deploy/weaviate/templates/NOTES.txt new file mode 100644 index 000000000..c3b3453e3 --- /dev/null +++ b/deploy/weaviate/templates/NOTES.txt @@ -0,0 +1,2 @@ +1. Get the application URL by running these commands: + diff --git a/deploy/weaviate/templates/_helpers.tpl b/deploy/weaviate/templates/_helpers.tpl new file mode 100644 index 000000000..d6d5b00aa --- /dev/null +++ b/deploy/weaviate/templates/_helpers.tpl @@ -0,0 +1,52 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "weaviate.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "weaviate.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "weaviate.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "weaviate.labels" -}} +helm.sh/chart: {{ include "weaviate.chart" . }} +{{ include "weaviate.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "weaviate.selectorLabels" -}} +app.kubernetes.io/name: {{ include "weaviate.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + diff --git a/deploy/weaviate/templates/backuppolicytemplate.yaml b/deploy/weaviate/templates/backuppolicytemplate.yaml new file mode 100644 index 000000000..778b6181d --- /dev/null +++ b/deploy/weaviate/templates/backuppolicytemplate.yaml @@ -0,0 +1,15 @@ +apiVersion: dataprotection.kubeblocks.io/v1alpha1 +kind: BackupPolicyTemplate +metadata: + name: backup-policy-template-weaviate + labels: + clusterdefinition.kubeblocks.io/name: weaviate-standalone + {{- include "weaviate.labels" . | nindent 4 }} +spec: + # which backup tool to perform database backup, only support one tool. + backupToolName: volumesnapshot + ttl: 168h0m0s + + credentialKeyword: + userKeyword: username + passwordKeyword: password diff --git a/deploy/weaviate/templates/clusterdefinition.yaml b/deploy/weaviate/templates/clusterdefinition.yaml new file mode 100644 index 000000000..e30b95f96 --- /dev/null +++ b/deploy/weaviate/templates/clusterdefinition.yaml @@ -0,0 +1,130 @@ +--- +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterDefinition +metadata: + name: weaviate-standalone + labels: + {{- include "weaviate.labels" . | nindent 4 }} +spec: + connectionCredential: + username: root + password: "$(RANDOM_PASSWD)" + endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-weaviate)" + host: "$(SVC_FQDN)" + port: "$(SVC_PORT_tcp-weaviate)" + componentDefs: + - name: weaviate + workloadType: Stateful + characterType: weaviate + probes: + monitor: + builtIn: false + exporterConfig: + scrapePath: /metrics + scrapePort: 9187 + logConfigs: + configSpecs: + - name: weaviate-standalone-config-template + templateRef: weaviate-standalone-config-template + volumeName: weaviate-config + namespace: {{ .Release.Namespace }} + service: + ports: + - name: tcp-weaviate + protocol: TCP + port: 8080 + targetPort: tcp-weaviate + volumeTypes: + - name: data + type: data + podSpec: + securityContext: + fsGroup: 1001 + containers: + - name: weaviate + imagePullPolicy: {{default .Values.images.pullPolicy "IfNotPresent"}} + command: + - /bin/weaviate + args: + - --host + - 0.0.0.0 + - --port + - "8080" + - --scheme + - http + - --config-file + - /weaviate-config/conf.yaml + - --read-timeout=60s + - --write-timeout=60s + securityContext: + runAsUser: 0 + livenessProbe: + failureThreshold: 30 + httpGet: + path: /v1/.well-known/live + port: 8080 + scheme: HTTP + initialDelaySeconds: 900 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 3 + readinessProbe: + failureThreshold: 3 + httpGet: + path: /v1/.well-known/ready + port: 8080 + scheme: HTTP + initialDelaySeconds: 3 + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 3 + startupProbe: + failureThreshold: 3 + httpGet: + path: /v1/.well-known/ready + port: 8080 + scheme: HTTP + periodSeconds: 10 + successThreshold: 1 + timeoutSeconds: 3 + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + volumeMounts: + - mountPath: /weaviate-config + name: weaviate-config + - mountPath: /var/lib/weaviate + name: data + dnsPolicy: ClusterFirst + enableServiceLinks: true + ports: + - name: tcp-weaviate + containerPort: 8080 + - name: tcp-metrics + containerPort: 9091 + env: + - name: CLUSTER_DATA_BIND_PORT + value: "7001" + - name: CLUSTER_GOSSIP_BIND_PORT + value: "7000" + - name: GOGC + value: "100" + - name: PROMETHEUS_MONITORING_ENABLED + value: "false" + - name: QUERY_MAXIMUM_RESULTS + value: "100000" + - name: REINDEX_VECTOR_DIMENSIONS_AT_STARTUP + value: "false" + - name: TRACK_VECTOR_DIMENSIONS + value: "false" + - name: STANDALONE_MODE + value: 'true' + - name: PERSISTENCE_DATA_PATH + value: '/var/lib/weaviate' + - name: DEFAULT_VECTORIZER_MODULE + value: none + - name: CLUSTER_GOSSIP_BIND_PORT + value: "7000" + - name: CLUSTER_DATA_BIND_PORT + value: "7001" + - name: CLUSTER_JOIN + value: weaviate-headless.default.svc.cluster.local diff --git a/deploy/weaviate/templates/clusterversion.yaml b/deploy/weaviate/templates/clusterversion.yaml new file mode 100644 index 000000000..1b3a69b78 --- /dev/null +++ b/deploy/weaviate/templates/clusterversion.yaml @@ -0,0 +1,14 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClusterVersion +metadata: + name: weaviate-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} + labels: + {{- include "weaviate.labels" . | nindent 4 }} +spec: + clusterDefinitionRef: weaviate-standalone + componentVersions: + - componentDefRef: weaviate + versionsContext: + containers: + - name: weaviate + image: {{ .Values.images.weaviate.repository }}:{{ default .Chart.AppVersion .Values.images.weaviate.tag }} diff --git a/deploy/weaviate/templates/configmap.yaml b/deploy/weaviate/templates/configmap.yaml new file mode 100644 index 000000000..69e621073 --- /dev/null +++ b/deploy/weaviate/templates/configmap.yaml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: weaviate-standalone-config-template + namespace: {{ .Release.Namespace | quote }} + labels: + {{- include "weaviate.labels" . | nindent 4 }} + +data: + conf.yaml: |- + --- + authentication: + anonymous_access: + enabled: true + authorization: + admin_list: + enabled: false + query_defaults: + limit: 100 + debug: false \ No newline at end of file diff --git a/deploy/weaviate/templates/headlessService.yaml b/deploy/weaviate/templates/headlessService.yaml new file mode 100644 index 000000000..5c138d5d3 --- /dev/null +++ b/deploy/weaviate/templates/headlessService.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: v1 +kind: Service +metadata: + name: weaviate-headless + labels: + app.kubernetes.io/name: weaviate + app.kubernetes.io/managed-by: helm +spec: + type: ClusterIP + clusterIP: None + selector: + app: weaviate + ports: + - protocol: TCP + port: 80 + targetPort: 7000 + publishNotReadyAddresses: true diff --git a/deploy/weaviate/values.yaml b/deploy/weaviate/values.yaml new file mode 100644 index 000000000..b69d52ff0 --- /dev/null +++ b/deploy/weaviate/values.yaml @@ -0,0 +1,897 @@ +# Default values for weaviate. +# This is a YAML-formatted file. +# Declare variables to be passed into your templates. + + +clusterVersionOverride: "" +nameOverride: "" +fullnameOverride: "" + + +## @param commonLabels Labels to add to all deployed objects +## +commonLabels: {} + +## @param application images +## +images: + pullPolicy: IfNotPresent + weaviate: + repository: docker.io/semitechnologies/weaviate + tag: 1.18.0 + +## @param debugEnabled enables containers' debug logging +## +debugEnabled: true + +# overwrite command and args if you want to run specific startup scripts, for +# example setting the nofile limit +command: ["/bin/weaviate"] +args: + - '--host' + - '0.0.0.0' + - '--port' + - '8080' + - '--scheme' + - 'http' + - '--config-file' + - '/weaviate-config/conf.yaml' + - --read-timeout=60s + - --write-timeout=60s + +# below is an example that can be used to set an arbitrary nofile limit at +# startup: +# +# command: +# - "/bin/sh" +# args: +# - "-c" +# - "ulimit -n 65535 && /bin/weaviate --host 0.0.0.0 --port 8080 --scheme http --config-file /weaviate-config/conf.yaml" + +# Scale replicas of Weaviate. Note that as of v1.8.0 dynamic scaling is limited +# to cases where no data is imported yet. Scaling down after importing data may +# break usability. Full dynamic scalability will be added in a future release. +replicas: 1 +resources: {} + # requests: + # cpu: '500m' + # memory: '300Mi' + # limits: + # cpu: '1000m' +# memory: '1Gi' + + +# Add a service account ot the Weaviate pods if you need Weaviate to have permissions to +# access kubernetes resources or cloud provider resources. For example for it to have +# access to a backup up bucket, or if you want to restrict Weaviate pod in any way. +# By default, use the default ServiceAccount +serviceAccountName: + +# The Persistent Volume Claim settings for Weaviate. If there's a +# storage.fullnameOverride field set, then the default pvc will not be +# created, instead the one defined in fullnameOverride will be used +storage: + size: 32Gi + storageClassName: "" + +# The service controls how weaviate is exposed to the outside world. If you +# don't want a public load balancer, you can also choose 'ClusterIP' to make +# weaviate only accessible within your cluster. +service: + name: weaviate + ports: + - name: http + protocol: TCP + port: 80 + # Target port is going to be the same for every port + type: LoadBalancer + loadBalancerSourceRanges: [] + # optionally set cluster IP if you want to set a static IP + clusterIP: + annotations: {} + +# Adjust liveness, readiness and startup probes configuration +startupProbe: + # For kubernetes versions prior to 1.18 startupProbe is not supported thus can be disabled. + enabled: false + + initialDelaySeconds: 300 + periodSeconds: 60 + failureThreshold: 50 + successThreshold: 1 + timeoutSeconds: 3 + +livenessProbe: + initialDelaySeconds: 900 + periodSeconds: 10 + failureThreshold: 30 + successThreshold: 1 + timeoutSeconds: 3 + +readinessProbe: + initialDelaySeconds: 3 + periodSeconds: 10 + failureThreshold: 3 + successThreshold: 1 + timeoutSeconds: 3 + + +terminationGracePeriodSeconds: 600 + +# Weaviate Config +# +# The following settings allow you to customize Weaviate to your needs, for +# example set authentication and authorization options. See weaviate docs +# (https://www.weaviate.io/developers/weaviate/) for all +# configuration. +authentication: + anonymous_access: + enabled: true +authorization: + admin_list: + enabled: false +query_defaults: + limit: 100 +debug: false + + +# Insert any custom environment variables or envSecrets by putting the exact name +# and desired value into the settings below. Any env name passed will be automatically +# set for the statefulSet. +env: + CLUSTER_GOSSIP_BIND_PORT: 7000 + CLUSTER_DATA_BIND_PORT: 7001 + # The aggressiveness of the Go Garbage Collector. 100 is the default value. + GOGC: 100 + + # Expose metrics on port 2112 for Prometheus to scrape + PROMETHEUS_MONITORING_ENABLED: false + + # Set a MEM limit for the Weaviate Pod so it can help you both increase GC-related + # performance as well as avoid GC-related out-of-memory (“OOM”) situations + # GOMEMLIMIT: 6GiB + + # Maximum results Weaviate can query with/without pagination + # NOTE: Affects performance, do NOT set to a very high value. + # The default is 100K + QUERY_MAXIMUM_RESULTS: 100000 + + # whether to enable vector dimensions tracking metric + TRACK_VECTOR_DIMENSIONS: false + + # whether to re-index/-compute the vector dimensions metric (needed if upgrading from weaviate < v1.16.0) + REINDEX_VECTOR_DIMENSIONS_AT_STARTUP: false + +envSecrets: + + +# Configure backup providers +backups: + # The backup-filesystem module enables creation of the DB backups in + # the local filesystem + filesystem: + enabled: false + envconfig: + # Configure folder where backups should be saved + BACKUP_FILESYSTEM_PATH: /tmp/backups + + s3: + enabled: false + # If one is using AWS EKS and has already configured K8s Service Account + # that holds the AWS credentials one can pass a name of that service account + # here using this setting. + # NOTE: the root `serviceAccountName` config has priority over this one, and + # if the root one is set this one will NOT overwrite it. This one is here for + # backwards compatibility. + serviceAccountName: + + envconfig: + # Configure bucket where backups should be saved, this setting is mandatory + BACKUP_S3_BUCKET: weaviate-backups + + # Optional setting. Defaults to empty string. + # Set this option if you want to save backups to a given location + # inside the bucket + # BACKUP_S3_PATH: path/inside/bucket + + # Optional setting. Defaults to AWS S3 (s3.amazonaws.com). + # Set this option if you have a MinIO storage configured in your environment + # and want to use it instead of the AWS S3. + # BACKUP_S3_ENDPOINT: custom.minio.endpoint.address + + # Optional setting. Defaults to true. + # Set this option if you don't want to use SSL. + # BACKUP_S3_USE_SSL: true + + # You can pass environment AWS settings here: + # Define the region + # AWS_REGION: eu-west-1 + + # For Weaviate to be able to create bucket objects it needs a user credentials to authenticate to AWS. + # The User must have permissions to read/create/delete bucket objects. + # You can pass the User credentials (access-key id and access-secret-key) in 2 ways: + # 1. by setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY plain values in the `secrets` section below + # this chart will create a kubernetes secret for you with these key-values pairs + # 2. create Kubernetes secret/s with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys and their respective values + # Set the Key and the secret where it is set in `envSecrets` section below + secrets: {} + # AWS_ACCESS_KEY_ID: access-key-id (plain text) + # AWS_SECRET_ACCESS_KEY: secret-access-key (plain text) + + # If one has already defined secrets with AWS credentials one can pass them using + # this setting: + envSecrets: {} + # AWS_ACCESS_KEY_ID: name-of-the-k8s-secret-containing-the-key-id + # AWS_SECRET_ACCESS_KEY: name-of-the-k8s-secret-containing-the-key + + gcs: + enabled: false + envconfig: + # Configure bucket where backups should be saved, this setting is mandatory + BACKUP_GCS_BUCKET: weaviate-backups + + # Optional setting. Defaults to empty string. + # Set this option if you want to save backups to a given location + # inside the bucket + # BACKUP_GCS_PATH: path/inside/bucket + + # You can pass environment Google settings here: + # Define the project + # GOOGLE_CLOUD_PROJECT: project-id + + # For Weaviate to be able to create bucket objects it needs a ServiceAccount credentials to authenticate to GCP. + # The ServiceAccount must have permissions to read/create/delete bucket objects. + # You can pass the ServiceAccount credentials (as JSON) in 2 ways: + # 1. by setting the GOOGLE_APPLICATION_CREDENTIALS json as plain text in the `secrets` section below + # this chart will create a kubernetes secret for you with this key-values pairs + # 2. create a Kubernetes secret with GOOGLE_APPLICATION_CREDENTIALS key and its respective value + # Set the Key and the secret where it is set in `envSecrets` section below + secrets: {} + # GOOGLE_APPLICATION_CREDENTIALS: credentials-json-string + + # If one has already defined a secret with GOOGLE_APPLICATION_CREDENTIALS one can pass them using + # this setting: + envSecrets: {} + # GOOGLE_APPLICATION_CREDENTIALS: name-of-the-k8s-secret-containing-the-key + + azure: + enabled: false + envconfig: + # Configure container where backups should be saved, this setting is mandatory + BACKUP_AZURE_CONTAINER: weaviate-backups + + # Optional setting. Defaults to empty string. + # Set this option if you want to save backups to a given location + # inside the container + # BACKUP_AZURE_PATH: path/inside/container + + # For Weaviate to be able to create container objects it needs a user credentials to authenticate to Azure Storage. + # The User must have permissions to read/create/delete container objects. + # You can pass the User credentials (account-name id and account-key or connection-string) in 2 ways: + # 1. by setting the AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY + # or AZURE_STORAGE_CONNECTION_STRING plain values in the `secrets` section below + # this chart will create a kubernetes secret for you with these key-values pairs + # 2. create Kubernetes secret/s with AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY + # or AZURE_STORAGE_CONNECTION_STRING and their respective values + # Set the Key and the secret where it is set in `envSecrets` section below + secrets: {} + # AZURE_STORAGE_ACCOUNT: account-name (plain text) + # AZURE_STORAGE_KEY: account-key (plain text) + # AZURE_STORAGE_CONNECTION_STRING: connection-string (plain text) + + # If one has already defined secrets with Azure Storage credentials one can pass them using + # this setting: + envSecrets: {} + # AZURE_STORAGE_ACCOUNT: name-of-the-k8s-secret-containing-the-account-name + # AZURE_STORAGE_KEY: name-of-the-k8s-secret-containing-account-key + # AZURE_STORAGE_CONNECTION_STRING: name-of-the-k8s-secret-containing-connection-string + + +# modules are extensions to Weaviate, they can be used to support various +# ML-models, but also other features unrelated to model inference. +# An inference/vectorizer module is not required, you can also run without any +# modules and import your own vectors. +modules: + + # The text2vec-contextionary module uses a fastText-based vector-space to + # derive vector embeddings for your objects. It is very efficient on CPUs, + # but in some situations it cannot reach the same level of accuracy as + # transformers-based models. + text2vec-contextionary: + # disable if you want to use transformers or import or own vectors + enabled: false + + # The configuration below is ignored if enabled==false + fullnameOverride: contextionary + tag: en0.16.0-v1.0.2 + repo: semitechnologies/contextionary + registry: docker.io + replicas: 1 + envconfig: + occurrence_weight_linear_factor: 0.75 + neighbor_occurrence_ignore_percentile: 5 + enable_compound_splitting: false + extensions_storage_mode: weaviate + resources: + requests: + cpu: '500m' + memory: '500Mi' + limits: + cpu: '1000m' + memory: '5000Mi' + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The text2vec-transformers modules uses neural networks, such as BERT, + # DistilBERT, etc. to dynamically compute vector embeddings based on the + # sentence's context. It is very slow on CPUs and should run with + # CUDA-enabled GPUs for optimal performance. + text2vec-transformers: + + # enable if you want to use transformers instead of the + # text2vec-contextionary module + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + # The configuration below is ignored if enabled==false + + # replace with model of choice, see + # https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-transformers + # for all supported models or build your own container. + tag: distilbert-base-uncased + repo: semitechnologies/transformers-inference + registry: docker.io + replicas: 1 + fullnameOverride: transformers-inference + probeInitialDelaySeconds: 120 + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + passageQueryServices: + passage: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + + tag: facebook-dpr-ctx_encoder-single-nq-base + repo: semitechnologies/transformers-inference + registry: docker.io + replicas: 1 + fullnameOverride: transformers-inference-passage + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + query: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + + tag: facebook-dpr-question_encoder-single-nq-base + repo: semitechnologies/transformers-inference + registry: docker.io + replicas: 1 + fullnameOverride: transformers-inference-query + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The text2vec-openai module uses OpenAI Embeddings API + # to dynamically compute vector embeddings based on the + # sentence's context. + # More information about OpenAI Embeddings API can be found here: + # https://beta.openai.com/docs/guides/embeddings/what-are-embeddings + text2vec-openai: + + # enable if you want to use OpenAI module + enabled: false + + # Set your OpenAI API Key to be passed to Weaviate pod as + # an environment variable + apiKey: '' + + # The text2vec-huggingface module uses HuggingFace API + # to dynamically compute vector embeddings based on the + # sentence's context. + # More information about HuggingFace API can be found here: + # https://huggingface.co/docs/api-inference/detailed_parameters#feature-extraction-task + text2vec-huggingface: + + # enable if you want to use HuggingFace module + enabled: false + + # Set your HuggingFace API Key to be passed to Weaviate pod as + # an environment variable + apiKey: '' + + # The text2vec-cohere module uses Cohere API + # to dynamically compute vector embeddings based on the + # sentence's context. + # More information about Cohere API can be found here: https://docs.cohere.ai/ + text2vec-cohere: + + # enable if you want to use Cohere module + enabled: false + + # Set your Cohere API Key to be passed to Weaviate pod as + # an environment variable + apiKey: '' + + # The ref2vec-centroid module + ref2vec-centroid: + + # enable if you want to use Centroid module + enabled: false + + # The multi2vec-clip modules uses CLIP transformers to vectorize both images + # and text in the same vector space. It is typically slow(er) on CPUs and should + # run with CUDA-enabled GPUs for optimal performance. + multi2vec-clip: + + # enable if you want to use transformers instead of the + # text2vec-contextionary module + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + + # The configuration below is ignored if enabled==false + + # replace with model of choice, see + # https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/multi2vec-clip + # for all supported models or build your own container. + tag: sentence-transformers-clip-ViT-B-32-multilingual-v1 + repo: semitechnologies/multi2vec-clip + registry: docker.io + replicas: 1 + fullnameOverride: clip-inference + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + annotations: + nodeSelector: + tolerations: + + # The qna-transformers module uses neural networks, such as BERT, + # DistilBERT, to find an aswer in text to a given question + qna-transformers: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + tag: bert-large-uncased-whole-word-masking-finetuned-squad-34d66b1 + repo: semitechnologies/qna-transformers + registry: docker.io + replicas: 1 + fullnameOverride: qna-transformers + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The qna-openai module uses OpenAI Completions API + # to dynamically answer given questions. + # More information about OpenAI Completions API can be found here: + # https://beta.openai.com/docs/api-reference/completions + qna-openai: + + # enable if you want to use OpenAI module + enabled: false + + # Set your OpenAI API Key to be passed to Weaviate pod as + # an environment variable + apiKey: '' + + # The generative-openai module uses OpenAI Completions API + # along with text-davinci-003 model to behave as ChatGPT. + # More information about OpenAI Completions API can be found here: + # https://beta.openai.com/docs/api-reference/completions + generative-openai: + + # enable if you want to use OpenAI module + enabled: false + + # Set your OpenAI API Key to be passed to Weaviate pod as + # an environment variable + apiKey: '' + + # The img2vec-neural module uses neural networks, to generate + # a vector representation of the image + img2vec-neural: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + tag: resnet50 + repo: semitechnologies/img2vec-pytorch + registry: docker.io + replicas: 1 + fullnameOverride: img2vec-neural + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The text-spellcheck module uses spellchecker library to check + # misspellings in a given text + text-spellcheck: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + tag: pyspellchecker-en + repo: semitechnologies/text-spellcheck-model + registry: docker.io + replicas: 1 + fullnameOverride: text-spellcheck + + resources: + requests: + cpu: '400m' + memory: '400Mi' + limits: + cpu: '500m' + memory: '500Mi' + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The ner-transformers module uses spellchecker library to check + # misspellings in a given text + ner-transformers: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + tag: dbmdz-bert-large-cased-finetuned-conll03-english-0.0.2 + repo: semitechnologies/ner-transformers + registry: docker.io + replicas: 1 + fullnameOverride: ner-transformers + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # The sum-transformers module makes result texts summarizations + sum-transformers: + enabled: false + # You can set directly an inference URL of this module without deploying it with this release. + # You can do so by setting a value for the `inferenceUrl` here AND by setting the `enable` to `false` + inferenceUrl: {} + tag: facebook-bart-large-cnn-1.0.0 + repo: semitechnologies/sum-transformers + registry: docker.io + replicas: 1 + fullnameOverride: sum-transformers + envconfig: + # enable for CUDA support. Your K8s cluster needs to be configured + # accordingly and you need to explicitly set GPU requests & limits below + enable_cuda: false + + # only used when cuda is enabled + nvidia_visible_devices: all + nvidia_driver_capabilities: compute,utility + + # only used when cuda is enabled + ld_library_path: /usr/local/nvidia/lib64 + + resources: + requests: + cpu: '1000m' + memory: '3000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + limits: + cpu: '1000m' + memory: '5000Mi' + + # enable if running with CUDA support + # nvidia.com/gpu: 1 + + # It is possible to add a ServiceAccount to this module's Pods, it can be + # used in cases where the module is in a private registry and you want to + # give access to the registry only to this pod. + # NOTE: if not set the root `serviceAccountName` config will be used. + serviceAccountName: + + # You can guide where the pods are scheduled on a per-module basis, + # as well as for Weaviate overall. Each module accepts nodeSelector, + # tolerations, and affinity configuration. If it is set on a per- + # module basis, this configuration overrides the global config. + + nodeSelector: + tolerations: + affinity: + + # by choosing the default vectorizer module, you can tell Weaviate to always + # use this module as the vectorizer if nothing else is specified. Can be + # overwritten on a per-class basis. + # set to text2vec-transformers if running with transformers instead + default_vectorizer_module: none + +# It is also possible to configure authentication and authorization through a +# custom configmap The authorization and authentication values defined in +# values.yaml will be ignored when defining a custom config map. +custom_config_map: + enabled: false + name: 'custom-config' + +# Pass any annotations to Weaviate pods +annotations: + +nodeSelector: + +tolerations: + +affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 1 + podAffinityTerm: + topologyKey: "kubernetes.io/hostname" + labelSelector: + matchExpressions: + - key: "app" + operator: In + values: + - weaviate \ No newline at end of file From 693b3d7deef2d6f9e07900a16c53b1dc6a3bfedd Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Fri, 31 Mar 2023 11:36:52 +0800 Subject: [PATCH 12/80] fix: cli kubeblocks upgrade command output dashboard info (#2290) --- internal/cli/cmd/kubeblocks/upgrade.go | 2 ++ internal/cli/cmd/playground/util.go | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/internal/cli/cmd/kubeblocks/upgrade.go b/internal/cli/cmd/kubeblocks/upgrade.go index 5b5411e22..0bdfc037d 100644 --- a/internal/cli/cmd/kubeblocks/upgrade.go +++ b/internal/cli/cmd/kubeblocks/upgrade.go @@ -137,6 +137,8 @@ func (o *InstallOptions) Upgrade() error { if !o.Quiet { fmt.Fprintf(o.Out, "\nKubeBlocks has been upgraded %s SUCCESSFULLY!\n", msg) + // set monitor to true, so that we can print notes with monitor + o.Monitor = true o.printNotes() } return nil diff --git a/internal/cli/cmd/playground/util.go b/internal/cli/cmd/playground/util.go index b2afa9be8..f81333c0b 100644 --- a/internal/cli/cmd/playground/util.go +++ b/internal/cli/cmd/playground/util.go @@ -105,7 +105,7 @@ func readClusterInfoFromFile(path string) (*cp.K8sClusterInfo, error) { defer f.Close() var info cp.K8sClusterInfo - if err := json.NewDecoder(f).Decode(&info); err != nil { + if err = json.NewDecoder(f).Decode(&info); err != nil { return nil, err } return &info, nil From f4feda6b49b5510a21026b391446d0d9f8a9aa7c Mon Sep 17 00:00:00 2001 From: chantu Date: Fri, 31 Mar 2023 12:01:51 +0800 Subject: [PATCH 13/80] fix: scale in drop follower (#2165) --- cmd/manager/main.go | 9 ++ config/rbac/role.yaml | 24 ++++ .../consensusset/consensus_set_utils.go | 11 ++ controllers/apps/components/pod_controller.go | 113 ++++++++++++++++++ .../apps/components/pod_controller_test.go | 107 +++++++++++++++++ .../templates/clusterdefinition.yaml | 13 ++ deploy/apecloud-mysql/templates/scripts.yaml | 48 ++++++-- deploy/helm/config/rbac/role.yaml | 24 ++++ internal/constant/const.go | 1 + 9 files changed, 340 insertions(+), 10 deletions(-) create mode 100644 controllers/apps/components/pod_controller.go create mode 100644 controllers/apps/components/pod_controller_test.go diff --git a/cmd/manager/main.go b/cmd/manager/main.go index d82291d5e..7500c92b1 100644 --- a/cmd/manager/main.go +++ b/cmd/manager/main.go @@ -380,6 +380,15 @@ func main() { os.Exit(1) } + if err = (&components.PodReconciler{ + Client: mgr.GetClient(), + Scheme: mgr.GetScheme(), + Recorder: mgr.GetEventRecorderFor("pod-controller"), + }).SetupWithManager(mgr); err != nil { + setupLog.Error(err, "unable to create controller", "controller", "Pod") + os.Exit(1) + } + if viper.GetBool("enable_webhooks") { appsv1alpha1.RegisterWebhookManager(mgr) diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml index e6e751334..b3607a9e1 100644 --- a/config/rbac/role.yaml +++ b/config/rbac/role.yaml @@ -30,6 +30,30 @@ rules: - deployments/status verbs: - get +- apiGroups: + - apps + resources: + - pods + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - apps + resources: + - pods/finalizers + verbs: + - update +- apiGroups: + - apps + resources: + - pods/status + verbs: + - get - apiGroups: - apps resources: diff --git a/controllers/apps/components/consensusset/consensus_set_utils.go b/controllers/apps/components/consensusset/consensus_set_utils.go index 7ed77c5ff..8c5dc181a 100644 --- a/controllers/apps/components/consensusset/consensus_set_utils.go +++ b/controllers/apps/components/consensusset/consensus_set_utils.go @@ -444,6 +444,17 @@ func updateConsensusRoleInfo(ctx context.Context, } } } + // patch pods' annotations + for _, pod := range pods { + patch := client.MergeFrom(pod.DeepCopy()) + if pod.Annotations == nil { + pod.Annotations = map[string]string{} + } + pod.Annotations[constant.LeaderAnnotationKey] = leader + if err := cli.Patch(ctx, &pod, patch); err != nil { + return err + } + } return nil } diff --git a/controllers/apps/components/pod_controller.go b/controllers/apps/components/pod_controller.go new file mode 100644 index 000000000..f20167f57 --- /dev/null +++ b/controllers/apps/components/pod_controller.go @@ -0,0 +1,113 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package components + +import ( + "context" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/tools/record" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/controllers/apps/components/util" + "github.com/apecloud/kubeblocks/internal/constant" + intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" +) + +// PodReconciler reconciles a Pod object +type PodReconciler struct { + client.Client + Scheme *runtime.Scheme + Recorder record.EventRecorder +} + +// +kubebuilder:rbac:groups=apps,resources=pods,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=apps,resources=pods/status,verbs=get +// +kubebuilder:rbac:groups=apps,resources=pods/finalizers,verbs=update + +// Reconcile is part of the main kubernetes reconciliation loop which aims to +// move the current state of the cluster closer to the desired state. +// +// For more details, check Reconcile and its Result here: +// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.12.2/pkg/reconcile +func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + var ( + pod = &corev1.Pod{} + err error + cluster *appsv1alpha1.Cluster + ok bool + componentName string + componentStatus appsv1alpha1.ClusterComponentStatus + ) + + reqCtx := intctrlutil.RequestCtx{ + Ctx: ctx, + Req: req, + Log: log.FromContext(ctx).WithValues("pod", req.NamespacedName), + } + + if err = r.Client.Get(reqCtx.Ctx, reqCtx.Req.NamespacedName, pod); err != nil { + return intctrlutil.CheckedRequeueWithError(err, reqCtx.Log, "") + } + + if cluster, err = util.GetClusterByObject(reqCtx.Ctx, r.Client, pod); err != nil { + return intctrlutil.CheckedRequeueWithError(err, reqCtx.Log, "") + } + if cluster == nil { + return intctrlutil.Reconciled() + } + + if componentName, ok = pod.Labels[constant.KBAppComponentLabelKey]; !ok { + return intctrlutil.Reconciled() + } + + if cluster.Status.Components == nil { + return intctrlutil.Reconciled() + } + if componentStatus, ok = cluster.Status.Components[componentName]; !ok { + return intctrlutil.Reconciled() + } + if componentStatus.ConsensusSetStatus == nil { + return intctrlutil.Reconciled() + } + if componentStatus.ConsensusSetStatus.Leader.Pod == util.ComponentStatusDefaultPodName { + return intctrlutil.Reconciled() + } + + // sync leader status from cluster.status + patch := client.MergeFrom(pod.DeepCopy()) + pod.Annotations[constant.LeaderAnnotationKey] = componentStatus.ConsensusSetStatus.Leader.Pod + if err = r.Client.Patch(reqCtx.Ctx, pod, patch); err != nil { + return intctrlutil.CheckedRequeueWithError(err, reqCtx.Log, "") + } + r.Recorder.Eventf(pod, corev1.EventTypeNormal, "AddAnnotation", "add annotation %s=%s", constant.LeaderAnnotationKey, componentStatus.ConsensusSetStatus.Leader.Pod) + + return intctrlutil.Reconciled() +} + +// SetupWithManager sets up the controller with the Manager. +func (r *PodReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + For(&corev1.Pod{}). + WithEventFilter(predicate.NewPredicateFuncs(intctrlutil.WorkloadFilterPredicate)). + Complete(r) +} diff --git a/controllers/apps/components/pod_controller_test.go b/controllers/apps/components/pod_controller_test.go new file mode 100644 index 000000000..02e406972 --- /dev/null +++ b/controllers/apps/components/pod_controller_test.go @@ -0,0 +1,107 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package components + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/constant" + intctrlutil "github.com/apecloud/kubeblocks/internal/generics" + testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" +) + +var _ = Describe("Pod Controller", func() { + + var ( + randomStr = testCtx.GetRandomStr() + clusterName = "mysql-" + randomStr + clusterDefName = "cluster-definition-consensus-" + randomStr + clusterVersionName = "cluster-version-operations-" + randomStr + ) + + const ( + revisionID = "6fdd48d9cd" + consensusCompName = "consensus" + consensusCompType = "consensus" + ) + + cleanAll := func() { + // must wait until resources deleted and no longer exist before the testcases start, + // otherwise if later it needs to create some new resource objects with the same name, + // in race conditions, it will find the existence of old objects, resulting failure to + // create the new objects. + By("clean resources") + + // delete cluster(and all dependent sub-resources), clusterversion and clusterdef + testapps.ClearClusterResources(&testCtx) + + // clear rest resources + inNS := client.InNamespace(testCtx.DefaultNamespace) + ml := client.HasLabels{testCtx.TestObjLabelKey} + // namespaced resources + testapps.ClearResources(&testCtx, intctrlutil.OpsRequestSignature, inNS, ml) + testapps.ClearResources(&testCtx, intctrlutil.StatefulSetSignature, inNS, ml) + testapps.ClearResources(&testCtx, intctrlutil.PodSignature, inNS, ml, client.GracePeriodSeconds(0)) + } + + BeforeEach(cleanAll) + + AfterEach(cleanAll) + + Context("test controller", func() { + It("test pod controller", func() { + + leaderName := "test-leader-name" + podName := "test-pod-name" + + By("mock cluster object") + _, _, cluster := testapps.InitConsensusMysql(testCtx, clusterDefName, + clusterVersionName, clusterName, consensusCompType, consensusCompName) + + By("mock cluster's consensus status") + Expect(testapps.ChangeObjStatus(&testCtx, cluster, func() { + cluster.Status.Components = map[string]appsv1alpha1.ClusterComponentStatus{} + cluster.Status.Components[consensusCompName] = appsv1alpha1.ClusterComponentStatus{ + ConsensusSetStatus: &appsv1alpha1.ConsensusSetStatus{ + Leader: appsv1alpha1.ConsensusMemberStatus{ + Pod: leaderName, + AccessMode: "ReadWrite", + }, + }, + } + })).Should(Succeed()) + + By("triggering pod reconcile") + pod := testapps.NewPodFactory(cluster.Namespace, podName). + AddContainer(corev1.Container{Name: testapps.DefaultMySQLContainerName, Image: testapps.ApeCloudMySQLImage}). + AddLabels(constant.AppInstanceLabelKey, cluster.Name). + AddLabels(constant.KBAppComponentLabelKey, consensusCompName). + Create(&testCtx).GetObject() + podKey := client.ObjectKeyFromObject(pod) + + By("checking pod has leader annotation") + testapps.CheckObj(&testCtx, podKey, func(g Gomega, pod *corev1.Pod) { + g.Expect(pod.Annotations).ShouldNot(BeNil()) + g.Expect(pod.Annotations[constant.LeaderAnnotationKey]).Should(Equal(leaderName)) + }) + }) + }) +}) diff --git a/deploy/apecloud-mysql/templates/clusterdefinition.yaml b/deploy/apecloud-mysql/templates/clusterdefinition.yaml index 1d789f4f5..133005d17 100644 --- a/deploy/apecloud-mysql/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql/templates/clusterdefinition.yaml @@ -72,6 +72,8 @@ spec: name: mysql-config - name: scripts mountPath: /scripts + - name: annotations + mountPath: /etc/annotations ports: - containerPort: 3306 name: mysql @@ -111,6 +113,10 @@ spec: - name: MYSQL_DYNAMIC_CONFIG value: {{ if .Values.cluster.dynamicConfig }}{{ .Values.cluster.dynamicConfig }}{{ end }} command: ["/scripts/setup.sh"] + lifecycle: + preStop: + exec: + command: ["/scripts/pre-stop.sh"] - name: metrics image: {{ .Values.metrics.image.registry | default "docker.io" }}/{{ .Values.metrics.image.repository }}:{{ .Values.metrics.image.tag }} imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }} @@ -154,6 +160,13 @@ spec: httpGet: path: / port: http-metrics + volumes: + - name: annotations + downwardAPI: + items: + - path: "leader" + fieldRef: + fieldPath: metadata.annotations['cs.apps.kubeblocks.io/leader'] systemAccounts: cmdExecutorConfig: image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} diff --git a/deploy/apecloud-mysql/templates/scripts.yaml b/deploy/apecloud-mysql/templates/scripts.yaml index 39b5e17a1..8c46f63a2 100644 --- a/deploy/apecloud-mysql/templates/scripts.yaml +++ b/deploy/apecloud-mysql/templates/scripts.yaml @@ -9,17 +9,17 @@ data: #!/bin/bash leader=$KB_MYSQL_LEADER followers=$KB_MYSQL_FOLLOWERS - echo $leader - echo $followers + echo "leader=$leader" + echo "followers=$followers" sub_follower=`echo "$followers" | grep "$KB_POD_NAME"` - echo $KB_POD_NAME - echo $sub_follower + echo "KB_POD_NAME=$KB_POD_NAME" + echo "sub_follower=$sub_follower" if [ -z "$leader" -o "$KB_POD_NAME" = "$leader" -o ! -z "$sub_follower" ]; then echo "no need to call add" else idx=${KB_POD_NAME##*-} host=$(eval echo \$KB_MYSQL_"$idx"_HOSTNAME) - echo "$host" + echo "host=$host" leader_idx=${leader##*-} leader_host=$(eval echo \$KB_MYSQL_"$leader_idx"_HOSTNAME) if [ ! -z $leader_host ]; then @@ -41,23 +41,27 @@ data: fi /scripts/upgrade-learner.sh & fi - cluster_info=""; for (( i=0; i< $KB_MYSQL_N; i++ )); do + cluster_info=""; + for (( i=0; i< $KB_MYSQL_N; i++ )); do if [ $i -ne 0 ]; then cluster_info="$cluster_info;"; fi; host=$(eval echo \$KB_MYSQL_"$i"_HOSTNAME) - cluster_info="$cluster_info$host:13306"; + # setup pod weight, prefer pod 0 to be leader + if [ $i -eq 0 ]; then + cluster_info="$cluster_info$host:13306#9N"; + else + cluster_info="$cluster_info$host:13306#1N"; + fi done; idx=${KB_POD_NAME##*-} echo $idx host=$(eval echo \$KB_MYSQL_"$idx"_HOSTNAME) cluster_info="$cluster_info@$(($idx+1))"; - echo $cluster_info; + echo "cluster_info=$cluster_info"; mkdir -p /data/mysql/data; mkdir -p /data/mysql/log; chmod +777 -R /data/mysql; - leader=$KB_MYSQL_LEADER - echo $leader echo "KB_MYSQL_RECREATE=$KB_MYSQL_RECREATE" if [ "$KB_MYSQL_RECREATE" == "true" ]; then echo "recreate from existing volumes, touch /data/mysql/data/.resetup_db" @@ -137,3 +141,27 @@ data: fi sleep 5 done + pre-stop.sh: | + #!/bin/bash + leader=`cat /etc/annotations/leader` + echo "leader=$leader" + echo "KB_POD_NAME=$KB_POD_NAME" + if [ -z "$leader" -o "$KB_POD_NAME" = "$leader" ]; then + echo "no leader or self is leader, exit" + exit 0 + fi + idx=${KB_POD_NAME##*-} + host=$(eval echo \$KB_MYSQL_"$idx"_HOSTNAME) + echo "host=$host" + leader_idx=${leader##*-} + leader_host=$(eval echo \$KB_MYSQL_"$leader_idx"_HOSTNAME) + if [ ! -z $leader_host ]; then + host_flag="-h$leader_host" + fi + if [ ! -z $MYSQL_ROOT_PASSWORD ]; then + password_flag="-p$MYSQL_ROOT_PASSWORD" + fi + echo "mysql $host_flag -uroot $password_flag -e \"call dbms_consensus.downgrade_follower('$host:13306');\" 2>&1 " + mysql $host_flag -uroot $password_flag -e "call dbms_consensus.downgrade_follower('$host:13306');" 2>&1 + echo "mysql $host_flag -uroot $password_flag -e \"call dbms_consensus.drop_learner('$host:13306');\" 2>&1 " + mysql $host_flag -uroot $password_flag -e "call dbms_consensus.drop_learner('$host:13306');" 2>&1 diff --git a/deploy/helm/config/rbac/role.yaml b/deploy/helm/config/rbac/role.yaml index e6e751334..b3607a9e1 100644 --- a/deploy/helm/config/rbac/role.yaml +++ b/deploy/helm/config/rbac/role.yaml @@ -30,6 +30,30 @@ rules: - deployments/status verbs: - get +- apiGroups: + - apps + resources: + - pods + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - apps + resources: + - pods/finalizers + verbs: + - update +- apiGroups: + - apps + resources: + - pods/status + verbs: + - get - apiGroups: - apps resources: diff --git a/internal/constant/const.go b/internal/constant/const.go index 97e9e065f..417bec672 100644 --- a/internal/constant/const.go +++ b/internal/constant/const.go @@ -79,6 +79,7 @@ const ( SnapShotForStartAnnotationKey = "kubeblocks.io/snapshot-for-start" RestoreFromBackUpAnnotationKey = "kubeblocks.io/restore-from-backup" // RestoreFromBackUpAnnotationKey specifies the component to recover from the backup. ClusterSnapshotAnnotationKey = "kubeblocks.io/cluster-snapshot" // ClusterSnapshotAnnotationKey saves the snapshot of cluster. + LeaderAnnotationKey = "cs.apps.kubeblocks.io/leader" // ConfigurationTplLabelPrefixKey clusterVersion or clusterdefinition using tpl ConfigurationTplLabelPrefixKey = "config.kubeblocks.io/tpl" From 282f1923361bf54e693513fdb42163cbc6efc525 Mon Sep 17 00:00:00 2001 From: chantu Date: Fri, 31 Mar 2023 12:25:32 +0800 Subject: [PATCH 14/80] fix: merge annotations should contain original (#2185) --- .../cli/cmd/playground/testdata/kubeconfig | 2 +- .../lifecycle/cluster_plan_builder.go | 8 +++---- .../lifecycle/cluster_plan_utils.go | 18 +++++++++------ .../lifecycle/cluster_plan_utils_test.go | 22 +++++++++++++++++++ 4 files changed, 38 insertions(+), 12 deletions(-) diff --git a/internal/cli/cmd/playground/testdata/kubeconfig b/internal/cli/cmd/playground/testdata/kubeconfig index 8b00000dc..25328b694 100644 --- a/internal/cli/cmd/playground/testdata/kubeconfig +++ b/internal/cli/cmd/playground/testdata/kubeconfig @@ -4,4 +4,4 @@ contexts: null current-context: test-context kind: Config preferences: {} -users: null \ No newline at end of file +users: null diff --git a/internal/controller/lifecycle/cluster_plan_builder.go b/internal/controller/lifecycle/cluster_plan_builder.go index 864c306ec..5c37beaa6 100644 --- a/internal/controller/lifecycle/cluster_plan_builder.go +++ b/internal/controller/lifecycle/cluster_plan_builder.go @@ -446,8 +446,8 @@ func (c *clusterPlanBuilder) buildUpdateObj(node *lifecycleVertex) (client.Objec } // keep the original template annotations. // if annotations exist and are replaced, the statefulSet will be updated. - stsProto.Spec.Template.Annotations = mergeAnnotations(stsObj.Spec.Template.Annotations, - stsProto.Spec.Template.Annotations) + mergeAnnotations(stsObj.Spec.Template.Annotations, + &stsProto.Spec.Template.Annotations) stsObj.Spec.Template = stsProto.Spec.Template stsObj.Spec.Replicas = stsProto.Spec.Replicas stsObj.Spec.UpdateStrategy = stsProto.Spec.UpdateStrategy @@ -456,8 +456,8 @@ func (c *clusterPlanBuilder) buildUpdateObj(node *lifecycleVertex) (client.Objec handleDeploy := func(origObj, deployProto *appsv1.Deployment) (client.Object, error) { deployObj := origObj.DeepCopy() - deployProto.Spec.Template.Annotations = mergeAnnotations(deployObj.Spec.Template.Annotations, - deployProto.Spec.Template.Annotations) + mergeAnnotations(deployObj.Spec.Template.Annotations, + &deployProto.Spec.Template.Annotations) deployObj.Spec = deployProto.Spec return deployObj, nil } diff --git a/internal/controller/lifecycle/cluster_plan_utils.go b/internal/controller/lifecycle/cluster_plan_utils.go index c9d1bb544..13ea656e1 100644 --- a/internal/controller/lifecycle/cluster_plan_utils.go +++ b/internal/controller/lifecycle/cluster_plan_utils.go @@ -22,19 +22,23 @@ import ( "golang.org/x/exp/maps" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" - "github.com/apecloud/kubeblocks/internal/constant" ) // mergeAnnotations keeps the original annotations. // if annotations exist and are replaced, the Deployment/StatefulSet will be updated. -func mergeAnnotations(originalAnnotations, targetAnnotations map[string]string) map[string]string { - if restartAnnotation, ok := originalAnnotations[constant.RestartAnnotationKey]; ok { - if targetAnnotations == nil { - targetAnnotations = map[string]string{} +func mergeAnnotations(originalAnnotations map[string]string, targetAnnotations *map[string]string) { + if targetAnnotations == nil { + return + } + if *targetAnnotations == nil { + *targetAnnotations = map[string]string{} + } + for k, v := range originalAnnotations { + // if the annotation not exist in targetAnnotations, copy it from original. + if _, ok := (*targetAnnotations)[k]; !ok { + (*targetAnnotations)[k] = v } - targetAnnotations[constant.RestartAnnotationKey] = restartAnnotation } - return targetAnnotations } // mergeServiceAnnotations keeps the original annotations except prometheus scrape annotations. diff --git a/internal/controller/lifecycle/cluster_plan_utils_test.go b/internal/controller/lifecycle/cluster_plan_utils_test.go index 2e01fb5ab..fcd505921 100644 --- a/internal/controller/lifecycle/cluster_plan_utils_test.go +++ b/internal/controller/lifecycle/cluster_plan_utils_test.go @@ -46,5 +46,27 @@ var _ = Describe("cluster plan utils test", func() { expectAnnotations := map[string]string{"k1": "v11"} Expect(mergeServiceAnnotations(originalAnnotations, targetAnnotations)).To(Equal(expectAnnotations)) }) + + It("should merge annotations from original that not exist in target to final result", func() { + originalKey := "only-existing-in-original" + targetKey := "only-existing-in-target" + updatedKey := "updated-in-target" + originalAnnotations := map[string]string{ + originalKey: "true", + updatedKey: "false", + } + targetAnnotations := map[string]string{ + targetKey: "true", + updatedKey: "true", + } + mergeAnnotations(originalAnnotations, &targetAnnotations) + Expect(targetAnnotations[targetKey]).ShouldNot(BeEmpty()) + Expect(targetAnnotations[originalKey]).ShouldNot(BeEmpty()) + Expect(targetAnnotations[updatedKey]).Should(Equal("true")) + By("merging with target being nil") + var nilAnnotations map[string]string + mergeAnnotations(originalAnnotations, &nilAnnotations) + Expect(nilAnnotations).ShouldNot(BeNil()) + }) }) }) From 28f1b7682de03cc729436af47e3b07af066ce314 Mon Sep 17 00:00:00 2001 From: huangzhangshu <109708205+JashBook@users.noreply.github.com> Date: Fri, 31 Mar 2023 13:10:47 +0800 Subject: [PATCH 15/80] chore: upgrade release-helm-chart.yml (#2351) --- .github/workflows/cicd-pull-request.yml | 2 +- .github/workflows/cicd-push.yml | 2 +- .github/workflows/release-helm-chart.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/cicd-pull-request.yml b/.github/workflows/cicd-pull-request.yml index c8c73e4eb..c3122f053 100644 --- a/.github/workflows/cicd-pull-request.yml +++ b/.github/workflows/cicd-pull-request.yml @@ -93,7 +93,7 @@ jobs: name: check helm needs: trigger-mode if: needs.trigger-mode.outputs.trigger-mode == '[deploy]' - uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.0 + uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.2 with: MAKE_OPS: "bump-chart-ver" VERSION: "v0.4.0-check" diff --git a/.github/workflows/cicd-push.yml b/.github/workflows/cicd-push.yml index d0056d86a..abfcf92c7 100644 --- a/.github/workflows/cicd-push.yml +++ b/.github/workflows/cicd-push.yml @@ -205,7 +205,7 @@ jobs: check-helm: needs: trigger-mode if: ${{ contains(needs.trigger-mode.outputs.trigger-mode, '[deploy]') && github.ref_name != 'main' }} - uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.0 + uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.2 with: MAKE_OPS: "bump-chart-ver" VERSION: "v0.4.0-check" diff --git a/.github/workflows/release-helm-chart.yml b/.github/workflows/release-helm-chart.yml index 100fbe037..c8c3fb38e 100644 --- a/.github/workflows/release-helm-chart.yml +++ b/.github/workflows/release-helm-chart.yml @@ -32,7 +32,7 @@ jobs: release-chart: needs: chart-version - uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.0 + uses: apecloud/apecd/.github/workflows/release-charts.yml@v0.5.2 with: MAKE_OPS: "bump-chart-ver" VERSION: "${{ needs.chart-version.outputs.chart-version }}" From ec5b96fd30f4cdb55d1c5829be095bbd6323e7dc Mon Sep 17 00:00:00 2001 From: huangzhangshu <109708205+JashBook@users.noreply.github.com> Date: Fri, 31 Mar 2023 14:18:05 +0800 Subject: [PATCH 16/80] chore: change chatgpt-retrieval-plugin tag (#2356) --- deploy/chatgpt-retrieval-plugin/values.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy/chatgpt-retrieval-plugin/values.yaml b/deploy/chatgpt-retrieval-plugin/values.yaml index 91eabe370..610591ec7 100644 --- a/deploy/chatgpt-retrieval-plugin/values.yaml +++ b/deploy/chatgpt-retrieval-plugin/values.yaml @@ -9,7 +9,7 @@ image: repository: apecloud/chatgpt-retrieval-plugin pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. - tag: "latest" + tag: "arm64-latest" imagePullSecrets: [] nameOverride: "" From 06349e83765b497de6275b7f9409b55b68dacced Mon Sep 17 00:00:00 2001 From: xingran Date: Fri, 31 Mar 2023 14:18:17 +0800 Subject: [PATCH 17/80] chore: fix postgres patroni appVersion (#2357) --- deploy/postgresql-patroni-ha-cluster/Chart.yaml | 2 +- deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/deploy/postgresql-patroni-ha-cluster/Chart.yaml b/deploy/postgresql-patroni-ha-cluster/Chart.yaml index f73123fae..2bfe51851 100644 --- a/deploy/postgresql-patroni-ha-cluster/Chart.yaml +++ b/deploy/postgresql-patroni-ha-cluster/Chart.yaml @@ -6,4 +6,4 @@ type: application version: 0.5.0-alpha.0 -appVersion: "14.7.0" +appVersion: "14.5.0" diff --git a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml index 7b576cd5b..288cac1fa 100644 --- a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml +++ b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml @@ -17,7 +17,7 @@ spec: characterType: postgresql customLabelSpecs: - key: apps.kubeblocks.postgres.patroni/scope - value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)" + value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" resources: - gvk: "v1/Pod" selector: @@ -136,7 +136,7 @@ spec: - name: KUBERNETES_USE_CONFIGMAPS value: "true" - name: SCOPE - value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)" + value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" - name: KUBERNETES_SCOPE_LABEL value: "apps.kubeblocks.postgres.patroni/scope" - name: KUBERNETES_ROLE_LABEL From c204b943509133fa37e88b81cb6355069b02b02c Mon Sep 17 00:00:00 2001 From: shaojiang Date: Fri, 31 Mar 2023 15:50:30 +0800 Subject: [PATCH 18/80] fix: playground start kubeblocks crash #2289 (#2361) --- controllers/apps/cluster_controller.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/controllers/apps/cluster_controller.go b/controllers/apps/cluster_controller.go index 840c35009..180a58d06 100644 --- a/controllers/apps/cluster_controller.go +++ b/controllers/apps/cluster_controller.go @@ -28,6 +28,7 @@ import ( "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/tools/record" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/log" @@ -145,7 +146,7 @@ func (r *ClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct func (r *ClusterReconciler) SetupWithManager(mgr ctrl.Manager) error { requeueDuration = time.Duration(viper.GetInt(constant.CfgKeyCtrlrReconcileRetryDurationMS)) // TODO: add filter predicate for core API objects - return ctrl.NewControllerManagedBy(mgr). + b := ctrl.NewControllerManagedBy(mgr). For(&appsv1alpha1.Cluster{}). Owns(&appsv1.StatefulSet{}). Owns(&appsv1.Deployment{}). @@ -154,10 +155,12 @@ func (r *ClusterReconciler) SetupWithManager(mgr ctrl.Manager) error { Owns(&corev1.ConfigMap{}). Owns(&corev1.PersistentVolumeClaim{}). Owns(&policyv1.PodDisruptionBudget{}). - Owns(&snapshotv1.VolumeSnapshot{}). Owns(&dataprotectionv1alpha1.BackupPolicy{}). - Owns(&dataprotectionv1alpha1.Backup{}). - Complete(r) + Owns(&dataprotectionv1alpha1.Backup{}) + if viper.GetBool("VOLUMESNAPSHOT") { + b.Owns(&snapshotv1.VolumeSnapshot{}, builder.OnlyMetadata, builder.Predicates{}) + } + return b.Complete(r) } // Handle is the event handler for the cluster status event. From 53d219f8cef16dc76ab43a1cfddccc00c1a57891 Mon Sep 17 00:00:00 2001 From: Ziang Guo Date: Fri, 31 Mar 2023 16:58:58 +0800 Subject: [PATCH 19/80] chore: move loadbalancer sub-module to a separate repo https://github.com/apecloud/havip. (#2354) --- .github/workflows/release-image-lb.yml | 20 - Makefile | 7 +- cmd/cmd.mk | 17 - cmd/loadbalancer/internal/agent/eni.go | 375 ----- cmd/loadbalancer/internal/agent/eni_test.go | 239 --- .../internal/agent/generate_mocks.go | 20 - .../agent/mocks/node_manager_mocks.go | 114 -- .../internal/agent/mocks/node_mocks.go | 181 --- cmd/loadbalancer/internal/agent/node.go | 153 -- .../internal/agent/node_manager.go | 221 --- .../internal/agent/node_manager_test.go | 138 -- cmd/loadbalancer/internal/agent/node_test.go | 60 - cmd/loadbalancer/internal/agent/suite_test.go | 51 - .../internal/cloud/aws/aws_service.go | 776 ---------- .../internal/cloud/aws/aws_service_test.go | 391 ----- .../internal/cloud/aws/generate_mocks.go | 19 - cmd/loadbalancer/internal/cloud/aws/imds.go | 162 -- .../internal/cloud/aws/mocks/ec2_mocks.go | 294 ---- .../internal/cloud/aws/suite_test.go | 52 - cmd/loadbalancer/internal/cloud/aws/types.go | 57 - .../internal/cloud/factory/factory.go | 60 - .../internal/cloud/generate_mocks.go | 19 - .../internal/cloud/mocks/provider_mocks.go | 241 --- cmd/loadbalancer/internal/cloud/types.go | 99 -- cmd/loadbalancer/internal/config/config.go | 135 -- .../internal/config/suite_test.go | 64 - .../controllers/endpoint_controller.go | 116 -- .../controllers/endpoint_controller_test.go | 88 -- cmd/loadbalancer/internal/controllers/rbac.go | 26 - .../controllers/service_controller.go | 508 ------- .../controllers/service_controller_test.go | 351 ----- .../internal/controllers/suite_test.go | 132 -- .../internal/controllers/traffic_policy.go | 141 -- .../internal/iptables/iptables.go | 30 - cmd/loadbalancer/internal/iptables/types.go | 25 - .../internal/netlink/generate_mocks.go | 20 - .../internal/netlink/mocks/link_mocks.go | 81 - .../internal/netlink/mocks/netlink_mocks.go | 239 --- cmd/loadbalancer/internal/netlink/netlink.go | 32 - cmd/loadbalancer/internal/netlink/types.go | 47 - cmd/loadbalancer/internal/network/client.go | 39 - .../internal/network/client_darwin.go | 46 - .../internal/network/client_linux.go | 497 ------ .../internal/network/client_test.go | 262 ---- .../internal/network/generate_mocks.go | 19 - .../internal/network/mocks/network_mocks.go | 110 -- .../internal/network/suite_test.go | 37 - cmd/loadbalancer/internal/network/types.go | 29 - .../internal/procfs/generate_mocks.go | 19 - .../internal/procfs/mocks/procfs_mocks.go | 81 - cmd/loadbalancer/internal/procfs/types.go | 48 - .../internal/protocol/generate.go | 21 - .../protocol/mocks/node_client_mocks.go | 176 --- cmd/loadbalancer/internal/protocol/node.pb.go | 1352 ----------------- cmd/loadbalancer/internal/protocol/node.proto | 111 -- .../internal/protocol/node_grpc.pb.go | 286 ---- cmd/loadbalancer/main.go | 310 ---- cmd/loadbalancer/proxy.go | 170 --- config/loadbalancer/role.yaml | 43 - deploy/helm/values.yaml | 4 - deploy/loadbalancer/Chart.yaml | 37 - deploy/loadbalancer/README.md | 15 - deploy/loadbalancer/templates/_helpers.tpl | 93 -- .../loadbalancer/templates/clusterrole.yaml | 44 - .../templates/clusterrolebinding.yaml | 14 - deploy/loadbalancer/templates/daemonset.yaml | 88 -- deploy/loadbalancer/templates/deployment.yaml | 87 -- .../templates/serviceaccount.yaml | 12 - deploy/loadbalancer/values-production.yaml | 18 - deploy/loadbalancer/values.yaml | 111 -- docker/Dockerfile-loadbalancer | 49 - docker/docker.mk | 28 - .../migration/migration.md | 2 +- internal/cli/cluster/helper.go | 2 +- internal/cli/testing/fake.go | 2 +- internal/cli/types/types.go | 6 +- 76 files changed, 7 insertions(+), 10062 deletions(-) delete mode 100644 .github/workflows/release-image-lb.yml delete mode 100644 cmd/loadbalancer/internal/agent/eni.go delete mode 100644 cmd/loadbalancer/internal/agent/eni_test.go delete mode 100644 cmd/loadbalancer/internal/agent/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/agent/mocks/node_manager_mocks.go delete mode 100644 cmd/loadbalancer/internal/agent/mocks/node_mocks.go delete mode 100644 cmd/loadbalancer/internal/agent/node.go delete mode 100644 cmd/loadbalancer/internal/agent/node_manager.go delete mode 100644 cmd/loadbalancer/internal/agent/node_manager_test.go delete mode 100644 cmd/loadbalancer/internal/agent/node_test.go delete mode 100644 cmd/loadbalancer/internal/agent/suite_test.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/aws_service.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/aws_service_test.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/imds.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/mocks/ec2_mocks.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/suite_test.go delete mode 100644 cmd/loadbalancer/internal/cloud/aws/types.go delete mode 100644 cmd/loadbalancer/internal/cloud/factory/factory.go delete mode 100644 cmd/loadbalancer/internal/cloud/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/cloud/mocks/provider_mocks.go delete mode 100644 cmd/loadbalancer/internal/cloud/types.go delete mode 100644 cmd/loadbalancer/internal/config/config.go delete mode 100644 cmd/loadbalancer/internal/config/suite_test.go delete mode 100644 cmd/loadbalancer/internal/controllers/endpoint_controller.go delete mode 100644 cmd/loadbalancer/internal/controllers/endpoint_controller_test.go delete mode 100644 cmd/loadbalancer/internal/controllers/rbac.go delete mode 100644 cmd/loadbalancer/internal/controllers/service_controller.go delete mode 100644 cmd/loadbalancer/internal/controllers/service_controller_test.go delete mode 100644 cmd/loadbalancer/internal/controllers/suite_test.go delete mode 100644 cmd/loadbalancer/internal/controllers/traffic_policy.go delete mode 100644 cmd/loadbalancer/internal/iptables/iptables.go delete mode 100644 cmd/loadbalancer/internal/iptables/types.go delete mode 100644 cmd/loadbalancer/internal/netlink/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/netlink/mocks/link_mocks.go delete mode 100644 cmd/loadbalancer/internal/netlink/mocks/netlink_mocks.go delete mode 100644 cmd/loadbalancer/internal/netlink/netlink.go delete mode 100644 cmd/loadbalancer/internal/netlink/types.go delete mode 100644 cmd/loadbalancer/internal/network/client.go delete mode 100644 cmd/loadbalancer/internal/network/client_darwin.go delete mode 100644 cmd/loadbalancer/internal/network/client_linux.go delete mode 100644 cmd/loadbalancer/internal/network/client_test.go delete mode 100644 cmd/loadbalancer/internal/network/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/network/mocks/network_mocks.go delete mode 100644 cmd/loadbalancer/internal/network/suite_test.go delete mode 100644 cmd/loadbalancer/internal/network/types.go delete mode 100644 cmd/loadbalancer/internal/procfs/generate_mocks.go delete mode 100644 cmd/loadbalancer/internal/procfs/mocks/procfs_mocks.go delete mode 100644 cmd/loadbalancer/internal/procfs/types.go delete mode 100644 cmd/loadbalancer/internal/protocol/generate.go delete mode 100644 cmd/loadbalancer/internal/protocol/mocks/node_client_mocks.go delete mode 100644 cmd/loadbalancer/internal/protocol/node.pb.go delete mode 100644 cmd/loadbalancer/internal/protocol/node.proto delete mode 100644 cmd/loadbalancer/internal/protocol/node_grpc.pb.go delete mode 100644 cmd/loadbalancer/main.go delete mode 100644 cmd/loadbalancer/proxy.go delete mode 100644 config/loadbalancer/role.yaml delete mode 100644 deploy/loadbalancer/Chart.yaml delete mode 100644 deploy/loadbalancer/README.md delete mode 100644 deploy/loadbalancer/templates/_helpers.tpl delete mode 100644 deploy/loadbalancer/templates/clusterrole.yaml delete mode 100644 deploy/loadbalancer/templates/clusterrolebinding.yaml delete mode 100644 deploy/loadbalancer/templates/daemonset.yaml delete mode 100644 deploy/loadbalancer/templates/deployment.yaml delete mode 100644 deploy/loadbalancer/templates/serviceaccount.yaml delete mode 100644 deploy/loadbalancer/values-production.yaml delete mode 100644 deploy/loadbalancer/values.yaml delete mode 100644 docker/Dockerfile-loadbalancer diff --git a/.github/workflows/release-image-lb.yml b/.github/workflows/release-image-lb.yml deleted file mode 100644 index 56a08e90e..000000000 --- a/.github/workflows/release-image-lb.yml +++ /dev/null @@ -1,20 +0,0 @@ -name: RELEASE-IMAGE-LB - -on: - workflow_dispatch: - inputs: - image_tag: - description: 'image tag' - required: false - default: 'latest' - -jobs: - release-image: - uses: apecloud/apecd/.github/workflows/release-image.yml@v0.2.0 - with: - MAKE_OPS_PRE: "generate" - MAKE_OPS: "push-loadbalancer-image" - IMG: "apecloud/loadbalancer" - VERSION: "${{ inputs.image_tag }}" - GO_VERSION: 1.19 - secrets: inherit diff --git a/Makefile b/Makefile index 723bae660..7911f4fcc 100644 --- a/Makefile +++ b/Makefile @@ -112,7 +112,7 @@ help: ## Display this help. @awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m\033[0m\n"} /^[a-zA-Z_0-9-]+:.*?##/ { printf " \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST) .PHONY: all -all: manager kbcli probe reloader loadbalancer ## Make all cmd binaries. +all: manager kbcli probe reloader ## Make all cmd binaries. ##@ Development @@ -121,7 +121,6 @@ manifests: test-go-generate controller-gen ## Generate WebhookConfiguration, Clu $(CONTROLLER_GEN) rbac:roleName=manager-role crd:generateEmbeddedObjectMeta=true webhook paths="./cmd/manager/...;./apis/...;./controllers/...;./internal/..." output:crd:artifacts:config=config/crd/bases @cp config/crd/bases/* $(CHART_PATH)/crds @cp config/rbac/role.yaml $(CHART_PATH)/config/rbac/role.yaml - $(CONTROLLER_GEN) rbac:roleName=loadbalancer-role paths="./cmd/loadbalancer/..." output:dir=config/loadbalancer .PHONY: preflight-manifests preflight-manifests: generate ## Generate external Preflight API @@ -401,8 +400,6 @@ bump-chart-ver: \ bump-single-chart-ver.chatgpt-retrieval-plugin bump-chart-ver: ## Bump helm chart version. -LOADBALANCER_CHART_VERSION= - .PHONY: helm-package helm-package: bump-chart-ver ## Do helm package. ## it will pull down the latest charts that satisfy the dependencies, and clean up old dependencies. @@ -410,8 +407,6 @@ helm-package: bump-chart-ver ## Do helm package. ## before dependency update. # cd $(CHART_PATH)/charts && ls ../depend-charts/*.tgz | xargs -n1 tar xf #$(HELM) dependency update --skip-refresh $(CHART_PATH) - $(HELM) package deploy/loadbalancer - mv loadbalancer-*.tgz deploy/helm/depend-charts/ $(HELM) package deploy/apecloud-mysql mv apecloud-mysql-*.tgz deploy/helm/depend-charts/ $(HELM) package deploy/postgresql diff --git a/cmd/cmd.mk b/cmd/cmd.mk index 283941a73..62e9324fe 100644 --- a/cmd/cmd.mk +++ b/cmd/cmd.mk @@ -13,23 +13,6 @@ ##@ Sub-commands -## loadbalancer cmd - -.PHONY: loadbalancer-go-generate -loadbalancer-go-generate: ## Run go generate against loadbalancer code. -ifeq ($(SKIP_GO_GEN), false) - $(GO) generate -x ./cmd/loadbalancer/internal/... -endif - -.PHONY: loadbalancer -loadbalancer: loadbalancer-go-generate test-go-generate build-checks ## Build loadbalancer binary. - $(GO) build -ldflags=${LD_FLAGS} -o bin/loadbalancer ./cmd/loadbalancer - - -.PHONY: clean-loadbalancer -clean-loadbalancer: ## Clean bin/loadbalancer. - rm -f bin/loadbalancer - ## reloader cmd RELOADER_LD_FLAGS = "-s -w" diff --git a/cmd/loadbalancer/internal/agent/eni.go b/cmd/loadbalancer/internal/agent/eni.go deleted file mode 100644 index 8b3b5cd44..000000000 --- a/cmd/loadbalancer/internal/agent/eni.go +++ /dev/null @@ -1,375 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "context" - "encoding/json" - "fmt" - "strings" - "time" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - "k8s.io/apimachinery/pkg/util/wait" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" - "github.com/apecloud/kubeblocks/internal/cli/util" -) - -type NodeResource struct { - TotalPrivateIPs int - UsedPrivateIPs int - SubnetIds map[string]map[string]*pb.ENIMetadata - ENIResources map[string]*ENIResource -} - -func (h *NodeResource) GetSparePrivateIPs() int { - return h.TotalPrivateIPs - h.UsedPrivateIPs -} - -type ENIResource struct { - ENIId string - SubnetID string - TotalPrivateIPs int - UsedPrivateIPs int -} - -type eniManager struct { - logger logr.Logger - maxIPsPerENI int - maxENI int - minPrivateIP int - instanceID string - subnetID string - securityGroupIds []string - resource *NodeResource - cp cloud.Provider - nc pb.NodeClient -} - -func newENIManager(logger logr.Logger, ip string, info *pb.InstanceInfo, nc pb.NodeClient, cp cloud.Provider) (*eniManager, error) { - c := &eniManager{ - nc: nc, - cp: cp, - } - - c.instanceID = info.GetInstanceId() - c.subnetID = info.GetSubnetId() - c.securityGroupIds = info.GetSecurityGroupIds() - c.logger = logger.WithValues("ip", ip, "instance id", c.instanceID) - - c.minPrivateIP = config.MinPrivateIP - c.maxIPsPerENI = cp.GetENIIPv4Limit() - - c.maxENI = cp.GetENILimit() - if config.MaxENI > 0 && config.MaxENI < c.maxENI { - c.maxENI = config.MaxENI - } - - return c, c.init() -} - -func (c *eniManager) init() error { - managedENIs, err := c.getManagedENIs() - if err != nil { - return errors.Wrap(err, "ipamd init: failed to retrieve attached ENIs info") - } - hostResource := c.buildHostResource(managedENIs) - c.updateHostResource(hostResource) - - for i := range managedENIs { - eni := managedENIs[i] - c.logger.Info("Discovered managed ENI, trying to set it up", "eni id", eni.EniId) - - options := &util.RetryOptions{MaxRetry: 10, Delay: 1 * time.Second} - if err = util.DoWithRetry(context.Background(), c.logger, func() error { - setupENIRequest := &pb.SetupNetworkForENIRequest{ - RequestId: util.GenRequestID(), - Eni: eni, - } - _, err = c.nc.SetupNetworkForENI(context.Background(), setupENIRequest) - return err - }, options); err != nil { - c.logger.Error(err, "Failed to setup ENI", "eni id", eni.EniId) - } else { - c.logger.Info("ENI set up completed", "eni id", eni.EniId) - } - } - c.logger.Info("Successfully init node") - - return nil -} - -func (c *eniManager) updateHostResource(resource *NodeResource) { - c.resource = resource -} - -func (c *eniManager) buildHostResource(enis []*pb.ENIMetadata) *NodeResource { - result := &NodeResource{ - SubnetIds: make(map[string]map[string]*pb.ENIMetadata), - ENIResources: make(map[string]*ENIResource), - } - for index := range enis { - eni := enis[index] - result.TotalPrivateIPs += c.maxIPsPerENI - result.UsedPrivateIPs += len(eni.Ipv4Addresses) - result.ENIResources[eni.EniId] = &ENIResource{ - ENIId: eni.EniId, - SubnetID: eni.SubnetId, - TotalPrivateIPs: c.maxIPsPerENI, - UsedPrivateIPs: len(eni.Ipv4Addresses), - } - subnetEnis, ok := result.SubnetIds[eni.SubnetId] - if !ok { - subnetEnis = make(map[string]*pb.ENIMetadata) - } - subnetEnis[eni.EniId] = eni - result.SubnetIds[eni.SubnetId] = subnetEnis - } - return result -} - -func (c *eniManager) start(stop chan struct{}, reconcileInterval time.Duration, cleanLeakedInterval time.Duration) error { - if err := c.modifyPrimaryENISourceDestCheck(true); err != nil { - return errors.Wrap(err, "Failed to modify primary eni source/dest check") - } - - f1 := func() { - if err := c.ensureENI(); err != nil { - c.logger.Error(err, "Failed to ensure eni") - } - } - go wait.Until(f1, reconcileInterval, stop) - - f2 := func() { - if err := c.cleanLeakedENIs(); err != nil { - c.logger.Error(err, "Failed to clean leaked enis") - } - } - go wait.Until(f2, cleanLeakedInterval, stop) - - return nil -} - -func (c *eniManager) modifyPrimaryENISourceDestCheck(enabled bool) error { - describeENIRequest := &pb.DescribeAllENIsRequest{RequestId: util.GenRequestID()} - describeENIResponse, err := c.nc.DescribeAllENIs(context.Background(), describeENIRequest) - if err != nil { - return errors.Wrap(err, "Failed to get all enis, retry later") - } - - var primaryENI *pb.ENIMetadata - for _, eni := range describeENIResponse.Enis { - if eni.DeviceNumber == 0 { - primaryENI = eni - break - } - } - if primaryENI == nil { - return errors.Wrap(err, "Failed to find primary eni") - } - // TODO enable check when we can ensure traffic comes in and out from same interface - if err := c.cp.ModifySourceDestCheck(primaryENI.GetEniId(), enabled); err != nil { - return errors.Wrap(err, "Failed to disable primary eni source/destination check") - } - c.logger.Info("Successfully disable primary eni source/destination check") - return nil -} - -func (c *eniManager) getManagedENIs() ([]*pb.ENIMetadata, error) { - describeENIRequest := &pb.DescribeAllENIsRequest{RequestId: util.GenRequestID()} - describeENIResponse, err := c.nc.DescribeAllENIs(context.Background(), describeENIRequest) - if err != nil { - return nil, errors.Wrap(err, "ipamd init: failed to retrieve attached ENIs info") - } - - return c.filterManagedENIs(describeENIResponse.GetEnis()), nil -} - -func (c *eniManager) filterManagedENIs(enis map[string]*pb.ENIMetadata) []*pb.ENIMetadata { - var ( - managedENIList []*pb.ENIMetadata - ) - for eniID, eni := range enis { - if _, found := eni.Tags[cloud.TagENIKubeBlocksManaged]; !found { - continue - } - - // ignore primary ENI even if tagged - if eni.DeviceNumber == 0 { - continue - } - - managedENIList = append(managedENIList, enis[eniID]) - } - return managedENIList -} - -func (c *eniManager) ensureENI() error { - describeENIRequest := &pb.DescribeAllENIsRequest{RequestId: util.GenRequestID()} - describeENIResponse, err := c.nc.DescribeAllENIs(context.Background(), describeENIRequest) - if err != nil { - return errors.Wrap(err, "Failed to get all enis, retry later") - } - - var ( - min = c.minPrivateIP - max = c.minPrivateIP + c.maxIPsPerENI - managedENIs = c.filterManagedENIs(describeENIResponse.GetEnis()) - hostResource = c.buildHostResource(managedENIs) - totalSpare = hostResource.TotalPrivateIPs - hostResource.UsedPrivateIPs - ) - - c.updateHostResource(hostResource) - - c.logger.Info("Local private ip buffer status", - "spare private ip", totalSpare, "min spare private ip", min, "max spare private ip", max) - - b, _ := json.Marshal(hostResource) - c.logger.Info("Local private ip buffer status", "info", string(b)) - - if totalSpare < min { - if len(describeENIResponse.Enis) >= c.maxENI { - c.logger.Info("Limit exceed, can not create new eni", "current", len(describeENIResponse.Enis), "max", c.maxENI) - return nil - } - if err = c.tryCreateAndAttachENI(); err != nil { - c.logger.Error(err, "Failed to create and attach new ENI") - } - } else if totalSpare > max { - if err = c.tryDetachAndDeleteENI(managedENIs); err != nil { - c.logger.Error(err, "Failed to detach and delete idle ENI") - } - } - return nil -} - -func (c *eniManager) tryCreateAndAttachENI() error { - c.logger.Info("Try to create and attach new eni") - - // create ENI, use same sg and subnet as primary ENI - eniID, err := c.cp.CreateENI(c.instanceID, c.subnetID, c.securityGroupIds) - if err != nil { - return errors.Wrap(err, "Failed to create ENI, retry later") - } - c.logger.Info("Successfully create new eni", "eni id", eniID) - - if _, err = c.cp.AttachENI(c.instanceID, eniID); err != nil { - if derr := c.cp.DeleteENI(eniID); derr != nil { - c.logger.Error(derr, "Failed to delete newly created untagged ENI!") - } - return errors.Wrap(err, "Failed to attach ENI") - } - c.logger.Info("Successfully attach new eni, waiting for it to take effect", "eni id", eniID) - - // waiting for ENI attached - if err = c.waitForENIAttached(eniID); err != nil { - return errors.Wrap(err, "Unable to discover attached ENI from metadata service") - } - c.logger.Info("Successfully find eni attached", "eni id", eniID) - - // setup ENI networking stack - setupENIRequest := &pb.SetupNetworkForENIRequest{ - RequestId: util.GenRequestID(), - Eni: &pb.ENIMetadata{ - EniId: eniID, - }, - } - if _, err = c.nc.SetupNetworkForENI(context.Background(), setupENIRequest); err != nil { - return errors.Wrapf(err, "Failed to set up network for eni %s", eniID) - } - c.logger.Info("Successfully initialized new eni", "eni id", eniID) - return nil -} - -func (c *eniManager) tryDetachAndDeleteENI(enis []*pb.ENIMetadata) error { - c.logger.Info("Try to detach and delete idle eni") - - for _, eni := range enis { - if len(eni.Ipv4Addresses) > 1 { - continue - } - cleanENIRequest := &pb.CleanNetworkForENIRequest{ - RequestId: util.GenRequestID(), - Eni: eni, - } - if _, err := c.nc.CleanNetworkForENI(context.Background(), cleanENIRequest); err != nil { - return errors.Wrapf(err, "Failed to clean network for eni %s", eni.EniId) - } - if err := c.cp.FreeENI(eni.EniId); err != nil { - return errors.Wrapf(err, "Failed to free eni %s", eni.EniId) - } - c.logger.Info("Successfully detach and delete idle eni", "eni id", eni.EniId) - return nil - } - return errors.New("Failed to find a idle eni") -} - -func (c *eniManager) cleanLeakedENIs() error { - c.logger.Info("Start cleaning leaked enis") - - leakedENIs, err := c.cp.FindLeakedENIs(c.instanceID) - if err != nil { - return errors.Wrap(err, "Failed to find leaked enis, skip") - } - if len(leakedENIs) == 0 { - c.logger.Info("No leaked enis found, skip cleaning") - return nil - } - - var errs []string - for _, eni := range leakedENIs { - if err = c.cp.DeleteENI(eni.ID); err != nil { - errs = append(errs, fmt.Sprintf("%s: %s", eni.ID, err.Error())) - continue - } - c.logger.Info("Successfully deleted leaked eni", "eni id", eni.ID) - } - if len(errs) != 0 { - return errors.New(fmt.Sprintf("Failed to delete leaked enis, err: %s", strings.Join(errs, "|"))) - } - return nil -} - -var ( - ErrNoNetworkInterfaces = errors.New("No network interfaces found for ENI") - ErrENINotFound = errors.New("ENI is not found") -) - -func (c *eniManager) waitForENIAttached(eniID string) error { - f := func() error { - describeENIRequest := &pb.DescribeAllENIsRequest{RequestId: util.GenRequestID()} - enis, err := c.nc.DescribeAllENIs(context.Background(), describeENIRequest) - if err != nil { - c.logger.Error(err, "Failed to discover attached ENIs") - return ErrNoNetworkInterfaces - } - for _, eni := range enis.GetEnis() { - if eniID == eni.EniId { - return nil - } - } - return ErrENINotFound - } - if err := util.DoWithRetry(context.Background(), c.logger, f, &util.RetryOptions{MaxRetry: 15, Delay: 3 * time.Second}); err != nil { - return fmt.Errorf("giving up trying to retrieve ENIs from metadata service") - } - return nil -} diff --git a/cmd/loadbalancer/internal/agent/eni_test.go b/cmd/loadbalancer/internal/agent/eni_test.go deleted file mode 100644 index ff69b2494..000000000 --- a/cmd/loadbalancer/internal/agent/eni_test.go +++ /dev/null @@ -1,239 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "context" - "math" - "time" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/golang/mock/gomock" - "google.golang.org/grpc" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - mockcloud "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/mocks" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" - mockprotocol "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol/mocks" - "github.com/apecloud/kubeblocks/internal/cli/util" -) - -const ( - masterHostIP = "172.31.1.2" - nodeIP = "172.31.1.100" - subnet = "172.31.0.0/16" - - instanceID = "i-0000000000000" - securityGroupID = "sec-0000000000000" - eniID1 = "eni-01" - eniMac1 = "00:00:00:00:00:01" - eniIP11 = "172.31.1.10" - eniIP12 = "172.31.1.11" - eniIP13 = "172.31.1.12" - - eniID2 = "eni-02" - eniMac2 = "00:00:00:00:00:02" - eniIP21 = "172.31.2.10" - eniIP22 = "172.31.2.11" - eniIP23 = "172.31.2.12" - - eniID3 = "eni-03" - eniMac3 = "00:00:00:00:00:03" - eniIP31 = "172.31.3.10" - eniIP32 = "172.31.3.11" - - eniID4 = "eni-04" - eniIP41 = "172.31.4.10" - eniID5 = "eni-05" -) - -var getDescribeAllENIResponse = func() *pb.DescribeAllENIsResponse { - return &pb.DescribeAllENIsResponse{ - RequestId: util.GenRequestID(), - Enis: getMockENIs(), - } -} - -var getMockENIs = func() map[string]*pb.ENIMetadata { - return map[string]*pb.ENIMetadata{ - eniID1: { - EniId: eniID1, - Mac: eniMac1, - DeviceNumber: 0, - SubnetIpv4Cidr: subnet, - Ipv4Addresses: []*pb.IPv4Address{ - { - Primary: true, - Address: eniIP11, - }, - { - Primary: true, - Address: eniIP12, - }, - { - Primary: true, - Address: eniIP13, - }, - }, - }, - // busiest ENI - eniID2: { - EniId: eniID2, - Mac: eniMac2, - DeviceNumber: 1, - SubnetIpv4Cidr: subnet, - Tags: map[string]string{ - cloud.TagENIKubeBlocksManaged: "true", - cloud.TagENINode: masterHostIP, - cloud.TagENICreatedAt: time.Now().String(), - }, - Ipv4Addresses: []*pb.IPv4Address{ - { - Primary: true, - Address: eniIP21, - }, - { - Primary: false, - Address: eniIP22, - }, - { - Primary: false, - Address: eniIP23, - }, - }, - }, - eniID3: { - EniId: eniID3, - Mac: eniMac3, - DeviceNumber: 3, - SubnetIpv4Cidr: subnet, - Tags: map[string]string{ - cloud.TagENIKubeBlocksManaged: "true", - cloud.TagENINode: masterHostIP, - cloud.TagENICreatedAt: time.Now().String(), - }, - Ipv4Addresses: []*pb.IPv4Address{ - { - Primary: true, - Address: eniIP31, - }, - { - Primary: false, - Address: eniIP32, - }, - }, - }, - eniID4: { - EniId: eniID4, - DeviceNumber: 4, - Tags: map[string]string{ - cloud.TagENIKubeBlocksManaged: "true", - cloud.TagENINode: masterHostIP, - cloud.TagENICreatedAt: time.Now().String(), - }, - Ipv4Addresses: []*pb.IPv4Address{ - { - Primary: true, - Address: eniIP41, - }, - }, - }, - } -} - -var _ = Describe("Eni", func() { - - setup := func() (*eniManager, *mockcloud.MockProvider, *mockprotocol.MockNodeClient) { - ctrl := gomock.NewController(GinkgoT()) - mockProvider := mockcloud.NewMockProvider(ctrl) - mockNodeClient := mockprotocol.NewMockNodeClient(ctrl) - mockProvider.EXPECT().GetENILimit().Return(math.MaxInt) - mockProvider.EXPECT().GetENIIPv4Limit().Return(6) - mockNodeClient.EXPECT().DescribeAllENIs(gomock.Any(), gomock.Any()).Return(getDescribeAllENIResponse(), nil).AnyTimes() - mockNodeClient.EXPECT().SetupNetworkForENI(gomock.Any(), gomock.Any()).Return(nil, nil).AnyTimes() - info := &pb.InstanceInfo{ - InstanceId: instanceID, - SubnetId: subnet1Id, - SecurityGroupIds: []string{securityGroupID}, - } - manager, err := newENIManager(logger, nodeIP, info, mockNodeClient, mockProvider) - Expect(err).Should(BeNil()) - return manager, mockProvider, mockNodeClient - } - - Context("Test start", func() { - It("", func() { - manager, mockProvider, _ := setup() - mockProvider.EXPECT().ModifySourceDestCheck(eniID1, gomock.Any()).Return(nil) - // we close stop channel to prevent running ensureENI - stop := make(chan struct{}) - close(stop) - - Expect(manager.start(stop, 10*time.Second, 1*time.Minute)).Should(Succeed()) - }) - }) - - Context("Ensure ENI, alloc new ENI", func() { - It("", func() { - manager, mockProvider, mockNodeClient := setup() - manager.minPrivateIP = math.MaxInt - - eni := cloud.ENIMetadata{ID: eniID5} - mockProvider.EXPECT().CreateENI(gomock.Any(), gomock.Any(), gomock.Any()).Return(eni.ID, nil) - mockProvider.EXPECT().AttachENI(gomock.Any(), gomock.Any()).Return(eni.ID, nil) - mockNodeClient.EXPECT().SetupNetworkForENI(gomock.Any(), gomock.Any()).Return(nil, nil).AnyTimes() - response := getDescribeAllENIResponse() - response.Enis[eniID5] = &pb.ENIMetadata{EniId: eniID5} - mockNodeClient.EXPECT().DescribeAllENIs(gomock.Any(), gomock.Any()).Return(response, nil).AnyTimes() - Expect(manager.ensureENI()).Should(Succeed()) - }) - }) - - Context("Ensure ENI, delete spare ENI", func() { - It("", func() { - manager, mockProvider, mockNodeClient := setup() - - var ids []string - recordDeletedENI := func(ctx context.Context, request *pb.CleanNetworkForENIRequest, options ...grpc.CallOption) (*pb.CleanNetworkForENIResponse, error) { - ids = append(ids, request.GetEni().EniId) - return nil, nil - } - mockNodeClient.EXPECT().CleanNetworkForENI(gomock.Any(), gomock.Any()).DoAndReturn(recordDeletedENI).Return(nil, nil).AnyTimes() - mockProvider.EXPECT().FreeENI(gomock.Any()).Return(nil).AnyTimes() - Expect(manager.ensureENI()).Should(Succeed()) - Expect(len(ids)).Should(Equal(1)) - Expect(ids[0]).Should(Equal(eniID4)) - }) - }) - - Context("Clean leaked ENI", func() { - It("", func() { - manager, mockProvider, _ := setup() - enis := []*cloud.ENIMetadata{ - { - ID: eniID1, - DeviceNumber: 0, - }, - } - mockProvider.EXPECT().FindLeakedENIs(gomock.Any()).Return(enis, nil) - mockProvider.EXPECT().DeleteENI(gomock.Any()).Return(nil).AnyTimes() - Expect(manager.cleanLeakedENIs()).Should(Succeed()) - }) - }) -}) diff --git a/cmd/loadbalancer/internal/agent/generate_mocks.go b/cmd/loadbalancer/internal/agent/generate_mocks.go deleted file mode 100644 index f4ec1cb6a..000000000 --- a/cmd/loadbalancer/internal/agent/generate_mocks.go +++ /dev/null @@ -1,20 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/node_manager_mocks.go . NodeManager -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/node_mocks.go . Node diff --git a/cmd/loadbalancer/internal/agent/mocks/node_manager_mocks.go b/cmd/loadbalancer/internal/agent/mocks/node_manager_mocks.go deleted file mode 100644 index a75940640..000000000 --- a/cmd/loadbalancer/internal/agent/mocks/node_manager_mocks.go +++ /dev/null @@ -1,114 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent (interfaces: NodeManager) - -// Package mock_agent is a generated GoMock package. -package mock_agent - -import ( - context "context" - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - - agent "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" -) - -// MockNodeManager is a mock of NodeManager interface. -type MockNodeManager struct { - ctrl *gomock.Controller - recorder *MockNodeManagerMockRecorder -} - -// MockNodeManagerMockRecorder is the mock recorder for MockNodeManager. -type MockNodeManagerMockRecorder struct { - mock *MockNodeManager -} - -// NewMockNodeManager creates a new mock instance. -func NewMockNodeManager(ctrl *gomock.Controller) *MockNodeManager { - mock := &MockNodeManager{ctrl: ctrl} - mock.recorder = &MockNodeManagerMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockNodeManager) EXPECT() *MockNodeManagerMockRecorder { - return m.recorder -} - -// ChooseSpareNode mocks base method. -func (m *MockNodeManager) ChooseSpareNode(arg0 string) (agent.Node, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "ChooseSpareNode", arg0) - ret0, _ := ret[0].(agent.Node) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// ChooseSpareNode indicates an expected call of ChooseSpareNode. -func (mr *MockNodeManagerMockRecorder) ChooseSpareNode(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ChooseSpareNode", reflect.TypeOf((*MockNodeManager)(nil).ChooseSpareNode), arg0) -} - -// GetNode mocks base method. -func (m *MockNodeManager) GetNode(arg0 string) (agent.Node, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetNode", arg0) - ret0, _ := ret[0].(agent.Node) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// GetNode indicates an expected call of GetNode. -func (mr *MockNodeManagerMockRecorder) GetNode(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNode", reflect.TypeOf((*MockNodeManager)(nil).GetNode), arg0) -} - -// GetNodes mocks base method. -func (m *MockNodeManager) GetNodes() ([]agent.Node, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetNodes") - ret0, _ := ret[0].([]agent.Node) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// GetNodes indicates an expected call of GetNodes. -func (mr *MockNodeManagerMockRecorder) GetNodes() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodes", reflect.TypeOf((*MockNodeManager)(nil).GetNodes)) -} - -// Start mocks base method. -func (m *MockNodeManager) Start(arg0 context.Context) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Start", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// Start indicates an expected call of Start. -func (mr *MockNodeManagerMockRecorder) Start(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Start", reflect.TypeOf((*MockNodeManager)(nil).Start), arg0) -} diff --git a/cmd/loadbalancer/internal/agent/mocks/node_mocks.go b/cmd/loadbalancer/internal/agent/mocks/node_mocks.go deleted file mode 100644 index 15c33baa4..000000000 --- a/cmd/loadbalancer/internal/agent/mocks/node_mocks.go +++ /dev/null @@ -1,181 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent (interfaces: Node) - -// Package mock_agent is a generated GoMock package. -package mock_agent - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - - agent "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" - protocol "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -// MockNode is a mock of Node interface. -type MockNode struct { - ctrl *gomock.Controller - recorder *MockNodeMockRecorder -} - -// MockNodeMockRecorder is the mock recorder for MockNode. -type MockNodeMockRecorder struct { - mock *MockNode -} - -// NewMockNode creates a new mock instance. -func NewMockNode(ctrl *gomock.Controller) *MockNode { - mock := &MockNode{ctrl: ctrl} - mock.recorder = &MockNodeMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockNode) EXPECT() *MockNodeMockRecorder { - return m.recorder -} - -// ChooseENI mocks base method. -func (m *MockNode) ChooseENI() (*protocol.ENIMetadata, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "ChooseENI") - ret0, _ := ret[0].(*protocol.ENIMetadata) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// ChooseENI indicates an expected call of ChooseENI. -func (mr *MockNodeMockRecorder) ChooseENI() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ChooseENI", reflect.TypeOf((*MockNode)(nil).ChooseENI)) -} - -// CleanNetworkForService mocks base method. -func (m *MockNode) CleanNetworkForService(arg0 string, arg1 *protocol.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "CleanNetworkForService", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// CleanNetworkForService indicates an expected call of CleanNetworkForService. -func (mr *MockNodeMockRecorder) CleanNetworkForService(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanNetworkForService", reflect.TypeOf((*MockNode)(nil).CleanNetworkForService), arg0, arg1) -} - -// GetIP mocks base method. -func (m *MockNode) GetIP() string { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetIP") - ret0, _ := ret[0].(string) - return ret0 -} - -// GetIP indicates an expected call of GetIP. -func (mr *MockNodeMockRecorder) GetIP() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetIP", reflect.TypeOf((*MockNode)(nil).GetIP)) -} - -// GetManagedENIs mocks base method. -func (m *MockNode) GetManagedENIs() ([]*protocol.ENIMetadata, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetManagedENIs") - ret0, _ := ret[0].([]*protocol.ENIMetadata) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// GetManagedENIs indicates an expected call of GetManagedENIs. -func (mr *MockNodeMockRecorder) GetManagedENIs() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetManagedENIs", reflect.TypeOf((*MockNode)(nil).GetManagedENIs)) -} - -// GetNodeInfo mocks base method. -func (m *MockNode) GetNodeInfo() *protocol.InstanceInfo { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetNodeInfo") - ret0, _ := ret[0].(*protocol.InstanceInfo) - return ret0 -} - -// GetNodeInfo indicates an expected call of GetNodeInfo. -func (mr *MockNodeMockRecorder) GetNodeInfo() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetNodeInfo", reflect.TypeOf((*MockNode)(nil).GetNodeInfo)) -} - -// GetResource mocks base method. -func (m *MockNode) GetResource() *agent.NodeResource { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetResource") - ret0, _ := ret[0].(*agent.NodeResource) - return ret0 -} - -// GetResource indicates an expected call of GetResource. -func (mr *MockNodeMockRecorder) GetResource() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetResource", reflect.TypeOf((*MockNode)(nil).GetResource)) -} - -// SetupNetworkForService mocks base method. -func (m *MockNode) SetupNetworkForService(arg0 string, arg1 *protocol.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "SetupNetworkForService", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// SetupNetworkForService indicates an expected call of SetupNetworkForService. -func (mr *MockNodeMockRecorder) SetupNetworkForService(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetupNetworkForService", reflect.TypeOf((*MockNode)(nil).SetupNetworkForService), arg0, arg1) -} - -// Start mocks base method. -func (m *MockNode) Start() error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Start") - ret0, _ := ret[0].(error) - return ret0 -} - -// Start indicates an expected call of Start. -func (mr *MockNodeMockRecorder) Start() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Start", reflect.TypeOf((*MockNode)(nil).Start)) -} - -// Stop mocks base method. -func (m *MockNode) Stop() { - m.ctrl.T.Helper() - m.ctrl.Call(m, "Stop") -} - -// Stop indicates an expected call of Stop. -func (mr *MockNodeMockRecorder) Stop() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Stop", reflect.TypeOf((*MockNode)(nil).Stop)) -} diff --git a/cmd/loadbalancer/internal/agent/node.go b/cmd/loadbalancer/internal/agent/node.go deleted file mode 100644 index becba0b25..000000000 --- a/cmd/loadbalancer/internal/agent/node.go +++ /dev/null @@ -1,153 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "context" - "sync" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" - "github.com/apecloud/kubeblocks/internal/cli/util" -) - -type Node interface { - Start() error - - Stop() - - GetIP() string - - GetResource() *NodeResource - - ChooseENI() (*pb.ENIMetadata, error) - - GetManagedENIs() ([]*pb.ENIMetadata, error) - - GetNodeInfo() *pb.InstanceInfo - - SetupNetworkForService(floatingIP string, eni *pb.ENIMetadata) error - - CleanNetworkForService(floatingIP string, eni *pb.ENIMetadata) error -} - -type node struct { - ip string - nc pb.NodeClient - cp cloud.Provider - em *eniManager - once sync.Once - info *pb.InstanceInfo - stop chan struct{} - logger logr.Logger -} - -func NewNode(logger logr.Logger, ip string, nc pb.NodeClient, cp cloud.Provider) (*node, error) { - result := &node{ - nc: nc, - ip: ip, - cp: cp, - once: sync.Once{}, - stop: make(chan struct{}), - logger: logger.WithValues("ip", ip), - } - - resp, err := nc.DescribeNodeInfo(context.Background(), &pb.DescribeNodeInfoRequest{RequestId: util.GenRequestID()}) - if err != nil { - return nil, errors.Wrap(err, "Failed to describe node info") - } - result.info = resp.GetInfo() - - em, err := newENIManager(logger, ip, result.info, nc, cp) - if err != nil { - return nil, errors.Wrap(err, "Failed to init eni manager") - } - result.em = em - - return result, nil -} - -func (n *node) Start() error { - return n.em.start(n.stop, config.ENIReconcileInterval, config.CleanLeakedENIInterval) -} - -func (n *node) Stop() { - n.once.Do(func() { - if n.stop != nil { - close(n.stop) - } - }) -} - -func (n *node) GetIP() string { - return n.ip -} - -func (n *node) GetNodeInfo() *pb.InstanceInfo { - return n.info -} - -func (n *node) GetResource() *NodeResource { - // TODO deepcopy - return n.em.resource -} - -func (n *node) GetManagedENIs() ([]*pb.ENIMetadata, error) { - return n.em.getManagedENIs() -} - -func (n *node) ChooseENI() (*pb.ENIMetadata, error) { - managedENIs, err := n.em.getManagedENIs() - if err != nil { - return nil, errors.Wrap(err, "Failed to get managed ENIs") - } - if len(managedENIs) == 0 { - return nil, errors.New("No managed eni found") - } - candidate := managedENIs[0] - for _, eni := range managedENIs { - if len(eni.Ipv4Addresses) > len(candidate.Ipv4Addresses) && len(eni.Ipv4Addresses) < n.em.maxIPsPerENI { - candidate = eni - } - } - n.logger.Info("Found busiest eni", "eni id", candidate.EniId) - return candidate, nil -} - -func (n *node) SetupNetworkForService(floatingIP string, eni *pb.ENIMetadata) error { - request := &pb.SetupNetworkForServiceRequest{ - RequestId: util.GenRequestID(), - PrivateIp: floatingIP, - Eni: eni, - } - _, err := n.nc.SetupNetworkForService(context.Background(), request) - return err -} - -func (n *node) CleanNetworkForService(floatingIP string, eni *pb.ENIMetadata) error { - request := &pb.CleanNetworkForServiceRequest{ - RequestId: util.GenRequestID(), - PrivateIp: floatingIP, - Eni: eni, - } - _, err := n.nc.CleanNetworkForService(context.Background(), request) - return err -} diff --git a/cmd/loadbalancer/internal/agent/node_manager.go b/cmd/loadbalancer/internal/agent/node_manager.go deleted file mode 100644 index 011a9c61c..000000000 --- a/cmd/loadbalancer/internal/agent/node_manager.go +++ /dev/null @@ -1,221 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "context" - "fmt" - "sync" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - "google.golang.org/grpc" - "google.golang.org/grpc/credentials/insecure" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/util/wait" - "sigs.k8s.io/controller-runtime/pkg/client" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -var ( - ErrNodeNotFound = errors.New("Node not found") - - newGRPCConn = func(addr string) (*grpc.ClientConn, error) { - return grpc.Dial(addr, grpc.WithTransportCredentials(insecure.NewCredentials())) - } - - newNode = func(logger logr.Logger, ip string, nc pb.NodeClient, cp cloud.Provider) (Node, error) { - return NewNode(logger, ip, nc, cp) - } -) - -type NodeManager interface { - Start(ctx context.Context) error - - GetNode(ip string) (Node, error) - - GetNodes() ([]Node, error) - - ChooseSpareNode(subnet string) (Node, error) -} - -type nodeManager struct { - sync.RWMutex - client.Client - - rpcPort int - cp cloud.Provider - logger logr.Logger - nodes map[string]Node -} - -func NewNodeManager(logger logr.Logger, rpcPort int, cp cloud.Provider, c client.Client) (*nodeManager, error) { - nm := &nodeManager{ - Client: c, - cp: cp, - logger: logger.WithName("NodeManager"), - rpcPort: rpcPort, - nodes: make(map[string]Node), - } - nm.logger.Info("Monitoring nodes", "labels", fmt.Sprintf("%v", config.TrafficNodeLabels)) - return nm, nil -} - -func (nm *nodeManager) Start(ctx context.Context) error { - if err := nm.refreshNodes(); err != nil { - return errors.Wrapf(err, "Failed to refresh nodes") - } - - f := func() { - if err := nm.refreshNodes(); err != nil { - nm.logger.Error(err, "Failed to refresh nodes") - } else { - nm.logger.Info("Successfully refresh nodes") - } - } - go wait.Until(f, config.RefreshNodeInterval, ctx.Done()) - return nil -} - -func (nm *nodeManager) refreshNodes() error { - nodeList := &corev1.NodeList{} - opts := []client.ListOption{client.MatchingLabels(config.TrafficNodeLabels)} - if err := nm.Client.List(context.Background(), nodeList, opts...); err != nil { - return errors.Wrap(err, "Failed to list cluster nodes") - } - nodesLatest := make(map[string]struct{}) - for _, item := range nodeList.Items { - var nodeIP string - for _, addr := range item.Status.Addresses { - if addr.Type != corev1.NodeInternalIP { - continue - } - nodeIP = addr.Address - } - if nodeIP == "" { - nm.logger.Error(fmt.Errorf("invalid cluster node %v", item), "Skip init node") - continue - } - nodesLatest[nodeIP] = struct{}{} - - cachedNode, err := nm.GetNode(nodeIP) - if err == nil && cachedNode != nil { - continue - } - if err != ErrNodeNotFound { - nm.logger.Error(err, "Failed to find node", "ip", nodeIP) - continue - } - cachedNode, err = nm.initNode(nodeIP) - if err != nil { - return errors.Wrapf(err, "Failed to init node %s", nodeIP) - } - nm.SetNode(nodeIP, cachedNode) - } - - nodesCached, err := nm.GetNodes() - if err != nil { - return errors.Wrapf(err, "Failed to get cached nodes") - } - for index := range nodesCached { - cachedNode := nodesCached[index] - if _, ok := nodesLatest[cachedNode.GetIP()]; ok { - continue - } - cachedNode.Stop() - nm.RemoveNode(cachedNode.GetIP()) - nm.logger.Info("Successfully removed node", "ip", cachedNode.GetIP()) - } - return nil -} - -func (nm *nodeManager) initNode(ip string) (Node, error) { - addr := fmt.Sprintf("%s:%d", ip, nm.rpcPort) - conn, err := newGRPCConn(addr) - if err != nil { - return nil, errors.Wrapf(err, "Failed to dial: %v", addr) - } - n, err := newNode(nm.logger, ip, pb.NewNodeClient(conn), nm.cp) - if err != nil { - return nil, errors.Wrapf(err, "Failed to init node") - } - if err = n.Start(); err != nil { - return nil, errors.Wrapf(err, "Failed to start node") - } - return n, nil -} - -func (nm *nodeManager) ChooseSpareNode(subnet string) (Node, error) { - nm.RLock() - defer nm.RUnlock() - var spareNode Node - for _, item := range nm.nodes { - if subnet != "" { - _, ok := item.GetResource().SubnetIds[subnet] - if !ok { - continue - } - } - if spareNode == nil { - spareNode = item - continue - } - if item.GetResource().GetSparePrivateIPs() < spareNode.GetResource().GetSparePrivateIPs() { - continue - } - spareNode = item - } - if spareNode == nil { - return nil, errors.New("Failed to find spare node") - } - return spareNode, nil -} - -func (nm *nodeManager) GetNodes() ([]Node, error) { - nm.RLock() - defer nm.RUnlock() - var result []Node - for _, item := range nm.nodes { - result = append(result, item) - } - return result, nil -} - -func (nm *nodeManager) GetNode(ip string) (Node, error) { - nm.RLock() - defer nm.RUnlock() - result, ok := nm.nodes[ip] - if !ok { - return nil, ErrNodeNotFound - } - return result, nil -} - -func (nm *nodeManager) RemoveNode(ip string) { - nm.Lock() - defer nm.Unlock() - delete(nm.nodes, ip) -} - -func (nm *nodeManager) SetNode(ip string, node Node) { - nm.Lock() - defer nm.Unlock() - nm.nodes[ip] = node -} diff --git a/cmd/loadbalancer/internal/agent/node_manager_test.go b/cmd/loadbalancer/internal/agent/node_manager_test.go deleted file mode 100644 index b3498729b..000000000 --- a/cmd/loadbalancer/internal/agent/node_manager_test.go +++ /dev/null @@ -1,138 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "context" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/go-logr/logr" - "github.com/golang/mock/gomock" - "google.golang.org/grpc" - corev1 "k8s.io/api/core/v1" - "sigs.k8s.io/controller-runtime/pkg/client" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -const ( - node1IP = "172.31.1.10" - subnet1Id = "subnet-001" - node2IP = "172.31.1.11" - subnet2Id = "subnet-002" - node3IP = "172.31.1.11" -) - -var _ = Describe("NodeManager", func() { - setup := func() (*gomock.Controller, *nodeManager) { - ctrl := gomock.NewController(GinkgoT()) - nm := &nodeManager{ - logger: logger, - Client: &mockK8sClient{}, - nodes: map[string]Node{ - node1IP: &node{ - ip: node1IP, - em: &eniManager{resource: &NodeResource{ - TotalPrivateIPs: 6, - UsedPrivateIPs: 1, - SubnetIds: map[string]map[string]*pb.ENIMetadata{ - subnet1Id: {}, - }, - }}, - }, - node2IP: &node{ - ip: node2IP, - em: &eniManager{resource: &NodeResource{ - TotalPrivateIPs: 6, - UsedPrivateIPs: 4, - SubnetIds: map[string]map[string]*pb.ENIMetadata{ - subnet1Id: {}, - subnet2Id: {}, - }, - }}, - }, - }, - } - return ctrl, nm - } - - Context("Refresh nodes", func() { - It("", func() { - _, nm := setup() - newGRPCConn = func(addr string) (*grpc.ClientConn, error) { - return nil, nil - } - newNode = func(logger logr.Logger, ip string, nc pb.NodeClient, cp cloud.Provider) (Node, error) { - return &mockNode{node: &node{stop: make(chan struct{})}}, nil - } - Expect(nm.refreshNodes()).Should(Succeed()) - }) - }) - - Context("Choose spare node", func() { - It("", func() { - _, nm := setup() - node, err := nm.ChooseSpareNode(subnet2Id) - Expect(err).Should(BeNil()) - Expect(node.GetIP()).Should(Equal(node2IP)) - - node, err = nm.ChooseSpareNode(subnet1Id) - Expect(err).Should(BeNil()) - Expect(node.GetIP()).Should(Equal(node1IP)) - - node, err = nm.ChooseSpareNode("") - Expect(err).Should(BeNil()) - Expect(node.GetIP()).Should(Equal(node1IP)) - }) - }) -}) - -type mockNode struct { - *node -} - -func (m *mockNode) Start() error { - return nil -} - -type mockK8sClient struct { - client.Client -} - -func (m *mockK8sClient) List(ctx context.Context, list client.ObjectList, opts ...client.ListOption) error { - result := list.(*corev1.NodeList) - result.Items = []corev1.Node{ - { - Status: corev1.NodeStatus{ - Addresses: []corev1.NodeAddress{ - { - Type: corev1.NodeInternalIP, - Address: node1IP, - }, - { - Type: corev1.NodeInternalIP, - Address: node2IP, - }, - }, - }, - }, - } - return nil -} diff --git a/cmd/loadbalancer/internal/agent/node_test.go b/cmd/loadbalancer/internal/agent/node_test.go deleted file mode 100644 index ee53f9d60..000000000 --- a/cmd/loadbalancer/internal/agent/node_test.go +++ /dev/null @@ -1,60 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "math" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/golang/mock/gomock" - - mockcloud "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/mocks" - mock_protocol "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol/mocks" -) - -var _ = Describe("Node", func() { - setup := func() (*node, *eniManager, *mock_protocol.MockNodeClient, *mockcloud.MockProvider) { - ctrl := gomock.NewController(GinkgoT()) - mockNodeClient := mock_protocol.NewMockNodeClient(ctrl) - mockProvider := mockcloud.NewMockProvider(ctrl) - em := &eniManager{ - maxIPsPerENI: math.MaxInt, - cp: mockProvider, - nc: mockNodeClient, - } - node := &node{ - em: em, - nc: mockNodeClient, - cp: mockProvider, - logger: logger, - } - return node, em, mockNodeClient, mockProvider - } - - Context("Choose ENI", func() { - It("", func() { - - node, _, mockNodeClient, _ := setup() - mockNodeClient.EXPECT().DescribeAllENIs(gomock.Any(), gomock.Any()).Return(getDescribeAllENIResponse(), nil) - eni, err := node.ChooseENI() - Expect(err).Should(BeNil()) - Expect(eni.EniId).Should(Equal(eniID2)) - }) - }) -}) diff --git a/cmd/loadbalancer/internal/agent/suite_test.go b/cmd/loadbalancer/internal/agent/suite_test.go deleted file mode 100644 index d80d1e4ec..000000000 --- a/cmd/loadbalancer/internal/agent/suite_test.go +++ /dev/null @@ -1,51 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package agent - -import ( - "testing" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/spf13/viper" - "go.uber.org/zap/zapcore" - "sigs.k8s.io/controller-runtime/pkg/log/zap" -) - -func init() { - viper.AutomaticEnv() -} - -var ( - logger = zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true), func(o *zap.Options) { - o.TimeEncoder = zapcore.ISO8601TimeEncoder - }) -) - -func TestNetwork(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecs(t, "Agent Test Suite") -} - -var _ = BeforeSuite(func() { -}) - -var _ = AfterSuite(func() { - -}) diff --git a/cmd/loadbalancer/internal/cloud/aws/aws_service.go b/cmd/loadbalancer/internal/cloud/aws/aws_service.go deleted file mode 100644 index f1301e3aa..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/aws_service.go +++ /dev/null @@ -1,776 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -import ( - "context" - "fmt" - "math" - "net/http" - "strings" - "time" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/awserr" - "github.com/aws/aws-sdk-go/aws/ec2metadata" - "github.com/aws/aws-sdk-go/aws/endpoints" - "github.com/aws/aws-sdk-go/aws/session" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/go-logr/logr" - "github.com/pkg/errors" - "k8s.io/apimachinery/pkg/util/sets" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/internal/cli/util" -) - -const ( - // ENINoManageTagKey is the tag that may be set on an ENI to indicate aws vpc cni should not manage it in any form. - ENINoManageTagKey = "node.k8s.amazonaws.com/no_manage" - - ErrCodeENINotFound = "InvalidNetworkInterfaceID.NotFound" - - // MinENILifeTime is the minimum lifetime for ENI being garbage collected - MinENILifeTime = 10 * time.Minute -) - -var ( - // ErrENINotFound is an error when ENI is not found. - ErrENINotFound = errors.New("ENI is not found") - - // ErrNoNetworkInterfaces occurs when DescribeNetworkInterfaces(eniID) returns no network interfaces - ErrNoNetworkInterfaces = errors.New("No network interfaces found for ENI") -) - -type awsService struct { - ec2Svc EC2 - imdsSvc imdsService - securityGroups []string - instanceID string - subnetID string - localIPv4 string - instanceType string - primaryENI string - primaryENImac string - availabilityZone string - eniLimit int - eniIPv4Limit int - logger logr.Logger -} - -func NewAwsService(logger logr.Logger) (*awsService, error) { - svc := &awsService{ - logger: logger, - } - awsCfg := aws.Config{ - MaxRetries: aws.Int(2), - HTTPClient: &http.Client{ - Timeout: 5 * time.Second, - }, - STSRegionalEndpoint: endpoints.RegionalSTSEndpoint, - } - sess := session.Must(session.NewSession(&awsCfg)) - svc.imdsSvc = imdsService{IMDS: ec2metadata.New(sess)} - - region, err := svc.imdsSvc.Region() - if err != nil { - return nil, errors.Wrap(err, "Failed to get region") - } - awsCfg.WithRegion(region).WithDisableSSL(true) - svc.ec2Svc = ec2.New(sess.Copy(&awsCfg)) - - if err := svc.initWithEC2Metadata(context.Background()); err != nil { - return nil, errors.Wrap(err, "Failed to init ec2 metadata") - } - - if err = svc.initInstanceTypeLimits(); err != nil { - return nil, errors.Wrap(err, "Failed to init instance limits") - } - - return svc, nil -} - -func (c *awsService) initWithEC2Metadata(ctx context.Context) error { - var kvs []interface{} - defer func() { - if len(kvs) > 0 { - c.logger.Info("Init Instance metadata", kvs...) - } - }() - - var err error - // retrieve availability-zone - c.availabilityZone, err = c.imdsSvc.getAZ(ctx) - if err != nil { - return err - } - kvs = append(kvs, "az", c.availabilityZone) - - // retrieve eth0 local-ipv4 - c.localIPv4, err = c.imdsSvc.getLocalIPv4(ctx) - if err != nil { - return err - } - kvs = append(kvs, "local ip", c.localIPv4) - - // retrieve instance-id - c.instanceID, err = c.imdsSvc.getInstanceID(ctx) - if err != nil { - return err - } - kvs = append(kvs, "instance id", c.instanceID) - - // retrieve instance-type - c.instanceType, err = c.imdsSvc.getInstanceType(ctx) - if err != nil { - return err - } - kvs = append(kvs, "instance type", c.instanceType) - - // retrieve primary interface's mac - c.primaryENImac, err = c.imdsSvc.getPrimaryMAC(ctx) - if err != nil { - return err - } - kvs = append(kvs, "primary eni mac", c.primaryENImac) - - c.primaryENI, err = c.imdsSvc.getInterfaceIDByMAC(ctx, c.primaryENImac) - if err != nil { - return err - } - kvs = append(kvs, "primary eni id", c.primaryENI) - - // retrieve sub-id - c.subnetID, err = c.imdsSvc.getSubnetID(ctx, c.primaryENImac) - if err != nil { - return err - } - kvs = append(kvs, "subnet id", c.subnetID) - - c.securityGroups, err = c.imdsSvc.getSecurityGroupIds(ctx, c.primaryENImac) - if err != nil { - return err - } - kvs = append(kvs, "security groups", c.securityGroups) - - return nil -} - -// GetInstanceInfo return EC2 instance info -func (c *awsService) GetInstanceInfo() *cloud.InstanceInfo { - return &cloud.InstanceInfo{ - InstanceID: c.instanceID, - SubnetID: c.subnetID, - SecurityGroupIDs: c.securityGroups, - } -} - -// GetInstanceType return EC2 instance type -func (c *awsService) GetInstanceType() string { - return c.instanceType -} - -// GetENIIPv4Limit return IP address limit per ENI based on EC2 instance type -func (c *awsService) GetENIIPv4Limit() int { - return c.eniIPv4Limit -} - -// GetENILimit returns the number of ENIs can be attached to an instance -func (c *awsService) GetENILimit() int { - return c.eniLimit -} - -func (c *awsService) initInstanceTypeLimits() error { - describeInstanceTypesInput := &ec2.DescribeInstanceTypesInput{InstanceTypes: []*string{aws.String(c.instanceType)}} - output, err := c.ec2Svc.DescribeInstanceTypesWithContext(context.Background(), describeInstanceTypesInput) - if err != nil || len(output.InstanceTypes) != 1 { - return errors.New(fmt.Sprintf("Failed calling DescribeInstanceTypes for `%s`: %v", c.instanceType, err)) - } - info := output.InstanceTypes[0] - - c.eniLimit = int(aws.Int64Value(info.NetworkInfo.MaximumNetworkInterfaces)) - c.eniIPv4Limit = int(aws.Int64Value(info.NetworkInfo.Ipv4AddressesPerInterface)) - - // Not checking for empty hypervisorType since have seen certain instances not getting this filled. - if aws.StringValue(info.InstanceType) != "" && c.eniLimit > 0 && c.eniIPv4Limit > 0 { - return nil - } - return errors.New(fmt.Sprintf("Unknown instance type %s", c.instanceType)) -} - -func (c *awsService) DescribeAllENIs() (map[string]*cloud.ENIMetadata, error) { - attachedENIList, err := c.GetAttachedENIs() - if err != nil { - return nil, errors.Wrap(err, "Failed to get local ENI metadata") - } - - attachedENIMap := make(map[string]cloud.ENIMetadata, len(attachedENIList)) - for _, eni := range attachedENIList { - attachedENIMap[eni.ID] = eni - } - - input := &ec2.DescribeNetworkInterfacesInput{ - Filters: []*ec2.Filter{{ - Name: aws.String("attachment.instance-id"), - Values: []*string{ - aws.String(c.instanceID), - }, - }}, - } - response, err := c.ec2Svc.DescribeNetworkInterfacesWithContext(context.Background(), input) - if err != nil { - return nil, errors.Wrap(err, "Failed to query attached network interface") - } - - var ( - result = make(map[string]*cloud.ENIMetadata, len(response.NetworkInterfaces)) - ) - for _, item := range response.NetworkInterfaces { - - c.logger.Info("Found eni", "card index", aws.Int64Value(item.Attachment.NetworkCardIndex), - "device index", aws.Int64Value(item.Attachment.DeviceIndex), - "eni id", aws.StringValue(item.NetworkInterfaceId)) - - eniID := aws.StringValue(item.NetworkInterfaceId) - - eniMetadata := attachedENIMap[eniID] - eniMetadata.SubnetID = aws.StringValue(item.SubnetId) - eniMetadata.Tags = convertSDKTagsToTags(item.TagSet) - - result[eniID] = &eniMetadata - - // Check IPv4 addresses - c.checkOutOfSyncState(eniID, eniMetadata.IPv4Addresses, item.PrivateIpAddresses) - } - return result, nil -} - -func (c *awsService) FindLeakedENIs(instanceID string) ([]*cloud.ENIMetadata, error) { - input := &ec2.DescribeNetworkInterfacesInput{ - Filters: []*ec2.Filter{ - { - Name: aws.String("tag-key"), - Values: []*string{ - aws.String(cloud.TagENINode), - }, - }, - { - Name: aws.String("status"), - Values: []*string{ - aws.String(ec2.NetworkInterfaceStatusAvailable), - }, - }, - }, - MaxResults: aws.Int64(1000), - } - - needClean := func(eni *ec2.NetworkInterface) bool { - ctxLog := c.logger.WithValues("eni id", eni.NetworkInterfaceId) - - var ( - tags = convertSDKTagsToTags(eni.TagSet) - eniID = aws.StringValue(eni.NetworkInterfaceId) - ) - node, ok := tags[cloud.TagENINode] - if !ok || node != instanceID { - return true - } - - if eni.Attachment != nil { - ctxLog.Info("ENI is attached, skip it", "attachment id", eni.Attachment.AttachmentId) - return false - } - - retagMap := map[string]string{ - cloud.TagENICreatedAt: time.Now().Format(time.RFC3339), - } - createdAt, ok := tags[cloud.TagENICreatedAt] - if !ok { - ctxLog.Info("Timestamp tag not exists, tag it") - if err := c.tagENI(eniID, retagMap); err != nil { - ctxLog.Error(err, "Failed to add tag for eni", "eni id", eniID) - } - return false - } - - t, err := time.Parse(time.RFC3339, createdAt) - if err != nil { - ctxLog.Error(err, "Timestamp tag is wrong, retagging it with current timestamp", "time", createdAt) - if err := c.tagENI(eniID, retagMap); err != nil { - ctxLog.Error(err, "Failed to retagging for eni", "eni id", eniID) - } - return false - } - - if time.Since(t) < MinENILifeTime { - ctxLog.Info("Found an leaked eni created less than 10 minutes ago, skip it") - return false - } - - return true - } - - var leakedENIs []*cloud.ENIMetadata - pageFn := func(output *ec2.DescribeNetworkInterfacesOutput, lastPage bool) bool { - enis := output.NetworkInterfaces - for index := range enis { - eni := enis[index] - if needClean(eni) { - leakedENIs = append(leakedENIs, &cloud.ENIMetadata{ - ID: aws.StringValue(eni.NetworkInterfaceId), - }) - } - } - return true - } - if err := c.ec2Svc.DescribeNetworkInterfacesPagesWithContext(context.Background(), input, pageFn); err != nil { - c.logger.Error(err, "") - return nil, errors.Wrap(err, "Failed to describe leaked enis") - } - - return leakedENIs, nil -} - -func (c *awsService) tagENI(eniID string, tagMap map[string]string) error { - input := &ec2.CreateTagsInput{ - Resources: []*string{ - aws.String(eniID), - }, - Tags: convertTagsToSDKTags(tagMap), - } - _, err := c.ec2Svc.CreateTagsWithContext(context.Background(), input) - return err -} - -func (c *awsService) AssignPrivateIPAddresses(eniID string, privateIP string) error { - input := &ec2.AssignPrivateIpAddressesInput{ - NetworkInterfaceId: aws.String(eniID), - PrivateIpAddresses: []*string{&privateIP}, - } - if _, err := c.ec2Svc.AssignPrivateIpAddressesWithContext(context.Background(), input); err != nil { - return errors.Wrapf(err, "Failed to assign private ip address %s on eni %s", privateIP, eniID) - } - return nil -} - -// Comparing the IMDS IPv4 addresses attached to the ENI with the DescribeNetworkInterfaces AWS API call, which -// technically should be the source of truth and contain the freshest information. Let's just do a quick scan here -// and output some diagnostic messages if we find stale info in the IMDS result. -func (c *awsService) checkOutOfSyncState(eniID string, imdsIPv4s []*cloud.IPv4Address, ec2IPv4s []*ec2.NetworkInterfacePrivateIpAddress) bool { - ctxLog := c.logger.WithName("checkOutOfSyncState").WithValues("eni id", eniID) - - synced := true - - imdsIPv4Set := sets.String{} - imdsPrimaryIP := "" - for _, imdsIPv4 := range imdsIPv4s { - imdsIPv4Set.Insert(imdsIPv4.Address) - if imdsIPv4.Primary { - imdsPrimaryIP = imdsIPv4.Address - } - } - ec2IPv4Set := sets.String{} - ec2IPv4PrimaryIP := "" - for _, privateIPv4 := range ec2IPv4s { - ec2IPv4Set.Insert(aws.StringValue(privateIPv4.PrivateIpAddress)) - if aws.BoolValue(privateIPv4.Primary) { - ec2IPv4PrimaryIP = aws.StringValue(privateIPv4.PrivateIpAddress) - } - } - missingIMDS := ec2IPv4Set.Difference(imdsIPv4Set).List() - missingDNI := imdsIPv4Set.Difference(ec2IPv4Set).List() - if len(missingIMDS) > 0 { - synced = false - strMissing := strings.Join(missingIMDS, ",") - ctxLog.Info("DescribeNetworkInterfaces yielded private IPv4 addresses that were not yet found in IMDS.", "ip list", strMissing) - } - if len(missingDNI) > 0 { - synced = false - strMissing := strings.Join(missingDNI, ",") - ctxLog.Info("IMDS query yielded stale IPv4 addresses that were not found in DescribeNetworkInterfaces.", "ip list", strMissing) - } - if imdsPrimaryIP != ec2IPv4PrimaryIP { - synced = false - ctxLog.Info("Primary IPs do not match", "imds", imdsPrimaryIP, "ec2", ec2IPv4PrimaryIP) - } - return synced -} - -func (c *awsService) GetAttachedENIs() (eniList []cloud.ENIMetadata, err error) { - ctx := context.TODO() - - macs, err := c.imdsSvc.getMACs(ctx) - if err != nil { - return nil, err - } - c.logger.Info("Total number of interfaces found", "count", len(macs)) - - enis := make([]cloud.ENIMetadata, len(macs)) - for i, mac := range macs { - enis[i], err = c.getENIMetadata(mac) - if err != nil { - return nil, errors.Wrapf(err, "Failed to retrieve metadata for ENI: %s", mac) - } - } - return enis, nil -} - -func (c *awsService) getENIMetadata(eniMAC string) (cloud.ENIMetadata, error) { - var ( - err error - deviceNum int - result cloud.ENIMetadata - ctx = context.TODO() - ) - - eniID, err := c.imdsSvc.getInterfaceIDByMAC(ctx, eniMAC) - if err != nil { - return result, err - } - - deviceNum, err = c.imdsSvc.getInterfaceDeviceNumber(ctx, eniMAC) - if err != nil { - return result, err - } - - primaryMAC, err := c.imdsSvc.getPrimaryMAC(ctx) - if err != nil { - return result, err - } - if eniMAC == primaryMAC && deviceNum != 0 { - // Can this even happen? To be backwards compatible, we will always use 0 here and log an error. - c.logger.Error(errors.New(fmt.Sprintf("Device number of primary ENI is %d! Forcing it to be 0 as expected", deviceNum)), "") - deviceNum = 0 - } - - cidr, err := c.imdsSvc.getSubnetIPv4CIDRBlock(ctx, eniMAC) - if err != nil { - return result, err - } - - ips, err := c.imdsSvc.getInterfacePrivateAddresses(ctx, eniMAC) - if err != nil { - return result, err - } - - ec2ip4s := make([]*cloud.IPv4Address, len(ips)) - for i, ip := range ips { - ec2ip4s[i] = &cloud.IPv4Address{ - Primary: i == 0, - Address: ip, - } - } - - return cloud.ENIMetadata{ - ID: eniID, - MAC: eniMAC, - DeviceNumber: deviceNum, - SubnetIPv4CIDR: cidr, - IPv4Addresses: ec2ip4s, - }, nil -} - -func (c *awsService) DeallocIPAddresses(eniID string, ips []string) error { - ctxLog := c.logger.WithValues("eni id", eniID, "ip", ips) - ctxLog.Info("Trying to unassign private ip from ENI") - - if len(ips) == 0 { - return nil - } - ipsInput := aws.StringSlice(ips) - - input := &ec2.UnassignPrivateIpAddressesInput{ - NetworkInterfaceId: aws.String(eniID), - PrivateIpAddresses: ipsInput, - } - - _, err := c.ec2Svc.UnassignPrivateIpAddressesWithContext(context.Background(), input) - if err != nil { - if strings.Contains(err.Error(), "InvalidParameterValue: Some of the specified addresses are not assigned to interface") { - ctxLog.Info("Private ip may has already been unassigned, skip") - return nil - } - return errors.Wrapf(err, "Failed to unassign ip address %s", ips) - } - ctxLog.Info("Successfully unassigned IPs from ENI") - return nil -} - -func (c *awsService) AllocIPAddresses(eniID string) (string, error) { - c.logger.Info("Trying to allocate IP addresses on ENI", "eni id", eniID) - - input := &ec2.AssignPrivateIpAddressesInput{ - NetworkInterfaceId: aws.String(eniID), - SecondaryPrivateIpAddressCount: aws.Int64(int64(1)), - } - - output, err := c.ec2Svc.AssignPrivateIpAddressesWithContext(context.Background(), input) - if err != nil { - c.logger.Error(err, "Failed to allocate private ip on ENI", "eni id", eniID) - return "", err - } - if output != nil { - c.logger.Info("Successfully allocated private IP addresses", "ip list", output.AssignedPrivateIpAddresses) - } - return aws.StringValue(output.AssignedPrivateIpAddresses[0].PrivateIpAddress), nil -} - -func (c *awsService) CreateENI(instanceID, subnetID string, securityGroupIds []string) (string, error) { - ctxLog := c.logger.WithValues("instance id", instanceID) - ctxLog.Info("Trying to create ENI") - - eniDescription := fmt.Sprintf("kubeblocks-lb-%s", instanceID) - tags := map[string]string{ - cloud.TagENINode: instanceID, - cloud.TagENICreatedAt: time.Now().Format(time.RFC3339), - cloud.TagENIKubeBlocksManaged: "true", - ENINoManageTagKey: "true", - } - tagSpec := []*ec2.TagSpecification{ - { - Tags: convertTagsToSDKTags(tags), - ResourceType: aws.String(ec2.ResourceTypeNetworkInterface), - }, - } - - input := &ec2.CreateNetworkInterfaceInput{ - Description: aws.String(eniDescription), - Groups: aws.StringSlice(securityGroupIds), - SubnetId: aws.String(subnetID), - TagSpecifications: tagSpec, - } - - result, err := c.ec2Svc.CreateNetworkInterfaceWithContext(context.Background(), input) - if err != nil { - return "", errors.Wrap(err, "failed to create network interface") - } - ctxLog.Info("Successfully created new ENI", "id", aws.StringValue(result.NetworkInterface.NetworkInterfaceId)) - - return aws.StringValue(result.NetworkInterface.NetworkInterfaceId), nil -} - -func (c *awsService) AttachENI(instanceID string, eniID string) (string, error) { - ctxLog := c.logger.WithValues("instance id", instanceID, "eni id", eniID) - ctxLog.Info("Trying to attach ENI") - - freeDevice, err := c.awsGetFreeDeviceNumber(instanceID) - if err != nil { - return "", errors.Wrap(err, "Failed to get a free device number") - } - - attachInput := &ec2.AttachNetworkInterfaceInput{ - DeviceIndex: aws.Int64(int64(freeDevice)), - InstanceId: aws.String(instanceID), - NetworkInterfaceId: aws.String(eniID), - } - attachOutput, err := c.ec2Svc.AttachNetworkInterfaceWithContext(context.Background(), attachInput) - if err != nil { - return "", errors.Wrap(err, "Failed to attach ENI") - } - - attachmentID := aws.StringValue(attachOutput.AttachmentId) - // Also change the ENI's attribute so that the ENI will be deleted when the instance is deleted. - attributeInput := &ec2.ModifyNetworkInterfaceAttributeInput{ - Attachment: &ec2.NetworkInterfaceAttachmentChanges{ - AttachmentId: aws.String(attachmentID), - DeleteOnTermination: aws.Bool(true), - }, - NetworkInterfaceId: aws.String(eniID), - } - - if _, err := c.ec2Svc.ModifyNetworkInterfaceAttributeWithContext(context.Background(), attributeInput); err != nil { - if err := c.FreeENI(eniID); err != nil { - c.logger.Error(err, "Failed to delete newly created untagged ENI!") - } - return "", errors.Wrap(err, "Failed to change the ENI's attribute") - } - return attachmentID, err -} - -func (c *awsService) awsGetFreeDeviceNumber(instanceID string) (int, error) { - input := &ec2.DescribeInstancesInput{ - InstanceIds: []*string{aws.String(instanceID)}, - } - - result, err := c.ec2Svc.DescribeInstancesWithContext(context.Background(), input) - if err != nil { - return 0, errors.Wrap(err, "Failed to retrieve instance data from EC2 control plane") - } - - if len(result.Reservations) != 1 { - return 0, errors.New(fmt.Sprintf("invalid instance id %s", instanceID)) - } - - // TODO race condition with vpc cni - inst := result.Reservations[0].Instances[0] - device := make(map[int]bool, len(inst.NetworkInterfaces)) - for _, eni := range inst.NetworkInterfaces { - device[int(aws.Int64Value(eni.Attachment.DeviceIndex))] = true - } - - for freeDeviceIndex := 0; freeDeviceIndex < math.MaxInt; freeDeviceIndex++ { - if !device[freeDeviceIndex] { - c.logger.Info("Found a free device number", "index", freeDeviceIndex) - return freeDeviceIndex, nil - } - } - return 0, errors.New("no available device number") -} - -func (c *awsService) DeleteENI(eniID string) error { - ctxLog := c.logger.WithValues("eni id", eniID) - ctxLog.Info("Trying to delete ENI") - - deleteInput := &ec2.DeleteNetworkInterfaceInput{ - NetworkInterfaceId: aws.String(eniID), - } - f := func() error { - if _, err := c.ec2Svc.DeleteNetworkInterfaceWithContext(context.Background(), deleteInput); err != nil { - if awsErr, ok := err.(awserr.Error); ok { - if awsErr.Code() == ErrCodeENINotFound { - ctxLog.Info("ENI has already been deleted") - return nil - } - } - return errors.Wrapf(err, "Failed to delete ENI") - } - ctxLog.Info("Successfully deleted ENI") - return nil - } - return util.DoWithRetry(context.Background(), c.logger, f, &util.RetryOptions{MaxRetry: 10, Delay: 3 * time.Second}) -} - -func (c *awsService) FreeENI(eniID string) error { - return c.freeENI(eniID, 2*time.Second) -} - -func (c *awsService) freeENI(eniID string, sleepDelayAfterDetach time.Duration) error { - ctxLog := c.logger.WithName("freeENI").WithValues("eni id", eniID) - ctxLog.Info("Trying to free eni") - - // Find out attachment - attachID, err := c.getENIAttachmentID(ctxLog, eniID) - if err != nil { - if err == ErrENINotFound { - ctxLog.Info("ENI not found. It seems to be already freed") - return nil - } - ctxLog.Error(err, "Failed to retrieve ENI") - return errors.Wrap(err, "Failed to retrieve ENI's attachment id") - } - ctxLog.Info("Found ENI attachment id", "attachment id", aws.StringValue(attachID)) - - detachInput := &ec2.DetachNetworkInterfaceInput{ - AttachmentId: attachID, - } - f := func() error { - if _, err := c.ec2Svc.DetachNetworkInterfaceWithContext(context.Background(), detachInput); err != nil { - return errors.Wrap(err, "Failed to detach ENI") - } - return nil - } - - // Retry detaching the ENI from the instance - err = util.DoWithRetry(context.Background(), c.logger, f, &util.RetryOptions{MaxRetry: 10, Delay: 3 * time.Second}) - if err != nil { - return err - } - ctxLog.Info("Successfully detached ENI") - - // It does take a while for EC2 to detach ENI from instance, so we wait 2s before trying to delete. - time.Sleep(sleepDelayAfterDetach) - err = c.DeleteENI(eniID) - if err != nil { - return errors.Wrapf(err, "FreeENI: failed to free ENI: %s", eniID) - } - - ctxLog.Info("Successfully freed ENI") - return nil -} - -func (c *awsService) getENIAttachmentID(ctxLog logr.Logger, eniID string) (*string, error) { - ctxLog = ctxLog.WithName("getENIAttachmentID").WithValues("eni id", eniID) - ctxLog.Info("Trying to get ENI attachment id") - - eniIds := make([]*string, 0) - eniIds = append(eniIds, aws.String(eniID)) - input := &ec2.DescribeNetworkInterfacesInput{ - NetworkInterfaceIds: eniIds, - } - - result, err := c.ec2Svc.DescribeNetworkInterfacesWithContext(context.Background(), input) - if err != nil { - if aerr, ok := err.(awserr.Error); ok { - if aerr.Code() == ErrCodeENINotFound { - return nil, ErrENINotFound - } - } - ctxLog.Error(err, "Failed to get ENI information from EC2 control plane") - return nil, errors.Wrap(err, "failed to describe network interface") - } - - // Shouldn't happen, but let's be safe - if len(result.NetworkInterfaces) == 0 { - return nil, ErrNoNetworkInterfaces - } - firstNI := result.NetworkInterfaces[0] - - // We cannot assume that the NetworkInterface.Attachment field is a non-nil - // pointer to a NetworkInterfaceAttachment struct. - // Ref: https://github.com/aws/amazon-vpc-cni-k8s/issues/914 - var attachID *string - if firstNI.Attachment != nil { - attachID = firstNI.Attachment.AttachmentId - } - return attachID, nil -} - -func (c *awsService) ModifySourceDestCheck(eniID string, enabled bool) error { - input := &ec2.ModifyNetworkInterfaceAttributeInput{ - NetworkInterfaceId: aws.String(eniID), - SourceDestCheck: &ec2.AttributeBooleanValue{Value: aws.Bool(enabled)}, - } - _, err := c.ec2Svc.ModifyNetworkInterfaceAttributeWithContext(context.Background(), input) - return err -} - -func convertTagsToSDKTags(tagsMap map[string]string) []*ec2.Tag { - if len(tagsMap) == 0 { - return nil - } - - sdkTags := make([]*ec2.Tag, 0, len(tagsMap)) - for _, key := range sets.StringKeySet(tagsMap).List() { - sdkTags = append(sdkTags, &ec2.Tag{ - Key: aws.String(key), - Value: aws.String(tagsMap[key]), - }) - } - return sdkTags -} - -func convertSDKTagsToTags(sdkTags []*ec2.Tag) map[string]string { - if len(sdkTags) == 0 { - return nil - } - - tagsMap := make(map[string]string, len(sdkTags)) - for _, sdkTag := range sdkTags { - tagsMap[aws.StringValue(sdkTag.Key)] = aws.StringValue(sdkTag.Value) - } - return tagsMap -} diff --git a/cmd/loadbalancer/internal/cloud/aws/aws_service_test.go b/cmd/loadbalancer/internal/cloud/aws/aws_service_test.go deleted file mode 100644 index 1c259c3de..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/aws_service_test.go +++ /dev/null @@ -1,391 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -import ( - "context" - "strings" - "time" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/request" - "github.com/aws/aws-sdk-go/service/ec2" - "github.com/golang/mock/gomock" - "github.com/pkg/errors" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - mockaws "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/aws/mocks" -) - -var _ = Describe("AwsService", func() { - const ( - metadataMACPath = "network/interfaces/macs/" - metadataMAC = "mac" - metadataAZ = "placement/availability-zone" - metadataLocalIP = "local-ipv4" - metadataInstanceID = "instance-id" - metadataInstanceType = "instance-type" - metadataSGs = "/security-group-ids" - metadataSubnetID = "/subnet-id" - metadataVPCcidrs = "/vpc-ipv4-cidr-blocks" - metadataDeviceNum = "/device-number" - metadataInterface = "/interface-id" - metadataSubnetCIDR = "/subnet-ipv4-cidr-block" - metadataIPv4s = "/local-ipv4s" - - maxENI = 3 - maxIPsPerENI = 16 - az = "local" - localIP = "172.31.0.1" - instanceID = "i-0000000000000" - instanceType = "t3.medium" - primaryMAC = "00:00:00:00:00:01" - securityGroupID = "sg-00000000" - subnet = "172.31.0.0/24" - subnetID = "subnet-00000000" - attachmentID = "eni-attach-00000000" - - eniID1 = "eni-01" - eniDeviceIndex1 = "0" - eniMac1 = "00:00:00:00:00:01" - eniIP11 = "172.31.1.10" - eniIP12 = "172.31.1.11" - eniIP13 = "172.31.1.12" - - eniID2 = "eni-02" - eniMac2 = "00:00:00:00:00:02" - eniDeviceIndex2 = "1" - eniIP21 = "172.31.2.10" - eniIP22 = "172.31.2.11" - eniIP23 = "172.31.2.12" - eniIP24 = "172.31.2.14" - - eniID3 = "eni-03" - eniMac3 = "00:00:00:00:00:03" - eniDeviceIndex3 = "2" - eniIP31 = "172.31.3.10" - - eniID4 = "eni-04" - ) - - getImdsService := func(overrides map[string]interface{}) imdsService { - data := map[string]interface{}{ - metadataAZ: az, - metadataLocalIP: localIP, - metadataInstanceID: instanceID, - metadataInstanceType: instanceType, - metadataMAC: primaryMAC, - metadataMACPath: primaryMAC, - metadataMACPath + primaryMAC + metadataDeviceNum: eniDeviceIndex1, - metadataMACPath + primaryMAC + metadataInterface: eniID1, - metadataMACPath + primaryMAC + metadataSGs: securityGroupID, - metadataMACPath + primaryMAC + metadataIPv4s: strings.Join([]string{eniIP11, eniIP12, eniIP13}, " "), - metadataMACPath + primaryMAC + metadataSubnetID: subnetID, - metadataMACPath + primaryMAC + metadataSubnetCIDR: subnet, - metadataMACPath + primaryMAC + metadataVPCcidrs: subnet, - } - - for k, v := range overrides { - data[k] = v - } - return imdsService{fakeIMDS(data)} - } - - getTags := func(t time.Time) map[string]string { - tags := map[string]string{ - cloud.TagENINode: instanceID, - cloud.TagENIKubeBlocksManaged: "true", - } - if !t.IsZero() { - tags[cloud.TagENICreatedAt] = t.Format(time.RFC3339) - } - return tags - } - - setup := func(overrides map[string]interface{}) (*gomock.Controller, *awsService, *mockaws.MockEC2) { - ctrl := gomock.NewController(GinkgoT()) - mockEC2 := mockaws.NewMockEC2(ctrl) - service := &awsService{ - instanceID: instanceID, - logger: logger, - ec2Svc: mockEC2, - imdsSvc: getImdsService(overrides), - } - return ctrl, service, mockEC2 - } - - Context("initWithEC2Metadata", func() { - It("", func() { - _, service, _ := setup(nil) - - Expect(service.initWithEC2Metadata(context.Background())).Should(Succeed()) - Expect(service.securityGroups).Should(Equal([]string{securityGroupID})) - Expect(service.instanceID).Should(Equal(instanceID)) - Expect(service.primaryENImac).Should(Equal(primaryMAC)) - }) - }) - - Context("initInstanceTypeLimits", func() { - It("", func() { - _, service, mockEC2 := setup(nil) - Expect(service.initWithEC2Metadata(context.Background())).Should(Succeed()) - Expect(service.instanceType).Should(Equal(instanceType)) - - describeInstanceTypesInput := &ec2.DescribeInstanceTypesInput{ - InstanceTypes: []*string{ - aws.String(service.instanceType), - }, - } - describeInstanceTypeOutput := &ec2.DescribeInstanceTypesOutput{ - InstanceTypes: []*ec2.InstanceTypeInfo{ - { - InstanceType: aws.String(instanceType), - NetworkInfo: &ec2.NetworkInfo{ - MaximumNetworkInterfaces: aws.Int64(maxENI), - Ipv4AddressesPerInterface: aws.Int64(maxIPsPerENI), - }, - }, - }, - } - mockEC2.EXPECT().DescribeInstanceTypesWithContext(gomock.Any(), describeInstanceTypesInput).Return(describeInstanceTypeOutput, nil) - Expect(service.initInstanceTypeLimits()).Should(Succeed()) - Expect(service.eniLimit).Should(Equal(maxENI)) - Expect(service.eniIPv4Limit).Should(Equal(maxIPsPerENI)) - }) - }) - - Context("DescribeAllENIs", func() { - It("", func() { - overrides := map[string]interface{}{ - metadataMACPath: strings.Join([]string{primaryMAC, eniMac2}, " "), - metadataMACPath + eniMac2 + metadataDeviceNum: eniDeviceIndex2, - metadataMACPath + eniMac2 + metadataInterface: eniID2, - metadataMACPath + eniMac2 + metadataSubnetCIDR: subnet, - metadataMACPath + eniMac2 + metadataIPv4s: strings.Join([]string{eniIP21, eniIP22, eniIP23}, " "), - } - _, service, mockEC2 := setup(overrides) - - enis := &ec2.DescribeNetworkInterfacesOutput{ - NetworkInterfaces: []*ec2.NetworkInterface{ - { - NetworkInterfaceId: aws.String(eniID1), - Attachment: &ec2.NetworkInterfaceAttachment{ - NetworkCardIndex: aws.Int64(0), - DeviceIndex: aws.Int64(0), - }, - PrivateIpAddresses: []*ec2.NetworkInterfacePrivateIpAddress{ - { - Primary: aws.Bool(true), - PrivateIpAddress: aws.String(eniIP11), - }, - { - Primary: aws.Bool(false), - PrivateIpAddress: aws.String(eniIP12), - }, - { - Primary: aws.Bool(false), - PrivateIpAddress: aws.String(eniIP13), - }, - }, - TagSet: convertTagsToSDKTags(getTags(time.Now())), - }, - { - NetworkInterfaceId: aws.String(eniID2), - Attachment: &ec2.NetworkInterfaceAttachment{ - NetworkCardIndex: aws.Int64(0), - DeviceIndex: aws.Int64(1), - }, - PrivateIpAddresses: []*ec2.NetworkInterfacePrivateIpAddress{ - { - Primary: aws.Bool(true), - PrivateIpAddress: aws.String(eniIP21), - }, - { - Primary: aws.Bool(false), - PrivateIpAddress: aws.String(eniIP22), - }, - { - Primary: aws.Bool(false), - PrivateIpAddress: aws.String(eniIP23), - }, - { - Primary: aws.Bool(false), - PrivateIpAddress: aws.String(eniIP24), - }, - }, - TagSet: convertTagsToSDKTags(getTags(time.Now())), - }, - }, - } - mockEC2.EXPECT().DescribeNetworkInterfacesWithContext(gomock.Any(), gomock.Any()).Return(enis, nil).AnyTimes() - - result, err := service.DescribeAllENIs() - Expect(err).Should(BeNil()) - Expect(len(result)).Should(Equal(2)) - Expect(result[eniID2].PrimaryIPv4Address()).Should(Equal(eniIP21)) - - var eni2 cloud.ENIMetadata - imdsENIs, err := service.GetAttachedENIs() - for _, item := range imdsENIs { - if item.ID == eniID2 { - eni2 = item - break - } - } - Expect(err).Should(BeNil()) - Expect(eni2).ShouldNot(BeNil()) - Expect(service.checkOutOfSyncState(eniID2, eni2.IPv4Addresses, enis.NetworkInterfaces[1].PrivateIpAddresses)).Should(BeFalse()) - }) - }) - - Context("Create ENI", func() { - It("", func() { - _, service, mockEC2 := setup(nil) - - createInterfaceOutput := &ec2.CreateNetworkInterfaceOutput{ - NetworkInterface: &ec2.NetworkInterface{ - NetworkInterfaceId: aws.String(eniID3), - }, - } - mockEC2.EXPECT().CreateNetworkInterfaceWithContext(gomock.Any(), gomock.Any()).Return(createInterfaceOutput, nil) - - eniID, err := service.CreateENI(instanceID, subnetID, []string{securityGroupID}) - Expect(err).Should(BeNil()) - Expect(eniID).Should(Equal(eniID3)) - }) - }) - - Context("Free ENI", func() { - It("", func() { - _, service, mockEC2 := setup(nil) - - describeInterfaceInput := &ec2.DescribeNetworkInterfacesInput{ - NetworkInterfaceIds: []*string{aws.String(eniID2)}, - } - describeInterfaceOutput := &ec2.DescribeNetworkInterfacesOutput{ - NetworkInterfaces: []*ec2.NetworkInterface{ - { - NetworkInterfaceId: aws.String(eniID2), - TagSet: convertTagsToSDKTags(getTags(time.Now())), - Attachment: &ec2.NetworkInterfaceAttachment{ - AttachmentId: aws.String(attachmentID), - }, - }, - }, - } - mockEC2.EXPECT().DescribeNetworkInterfacesWithContext(gomock.Any(), describeInterfaceInput).Return(describeInterfaceOutput, nil) - - detachInput := &ec2.DetachNetworkInterfaceInput{ - AttachmentId: aws.String(attachmentID), - } - mockEC2.EXPECT().DetachNetworkInterfaceWithContext(gomock.Any(), detachInput).Return(nil, errors.New("mock detach failed")) - mockEC2.EXPECT().DetachNetworkInterfaceWithContext(gomock.Any(), detachInput).Return(nil, nil) - - deleteInput := &ec2.DeleteNetworkInterfaceInput{ - NetworkInterfaceId: aws.String(eniID2), - } - mockEC2.EXPECT().DeleteNetworkInterfaceWithContext(gomock.Any(), deleteInput).Return(nil, errors.New("mock delete failed")) - mockEC2.EXPECT().DeleteNetworkInterfaceWithContext(gomock.Any(), deleteInput).Return(nil, nil) - Expect(service.FreeENI(eniID2)).Should(Succeed()) - }) - }) - - Context("Clean leaked ENIs", func() { - It("should be success without error", func() { - _, service, mockEC2 := setup(nil) - - store := make(map[string]string) - enis := &ec2.DescribeNetworkInterfacesOutput{ - NetworkInterfaces: []*ec2.NetworkInterface{ - { - NetworkInterfaceId: aws.String(eniID1), - Description: aws.String("just created eni, should not be cleaned"), - TagSet: convertTagsToSDKTags(getTags(time.Now())), - }, - { - NetworkInterfaceId: aws.String(eniID2), - Description: aws.String("expired leaked eni, should be cleaned"), - TagSet: convertTagsToSDKTags(getTags(time.Now().Add(-1 * time.Hour))), - }, - { - Attachment: &ec2.NetworkInterfaceAttachment{ - AttachmentId: aws.String("test"), - }, - NetworkInterfaceId: aws.String(eniID3), - Description: aws.String("eni attached to ec2, should not be cleaned"), - TagSet: convertTagsToSDKTags(getTags(time.Now())), - }, - { - NetworkInterfaceId: aws.String(eniID4), - Description: aws.String("eni without created at tag, should not be cleaned"), - TagSet: convertTagsToSDKTags(getTags(time.Time{})), - }, - }, - } - describeHookFn := func(ctx aws.Context, input *ec2.DescribeNetworkInterfacesInput, fn func(*ec2.DescribeNetworkInterfacesOutput, bool) bool, opts ...request.Option) error { - fn(enis, true) - return nil - } - mockEC2.EXPECT().DescribeNetworkInterfacesPagesWithContext(gomock.Any(), gomock.Any(), gomock.Any()).DoAndReturn(describeHookFn).Return(nil).AnyTimes() - - recordCreatedTagsRequest := func(ctx aws.Context, input *ec2.CreateTagsInput, opts ...request.Option) { - for k, v := range convertSDKTagsToTags(input.Tags) { - store[k] = v - } - } - mockEC2.EXPECT().CreateTagsWithContext(gomock.Any(), gomock.Any()).DoAndReturn(recordCreatedTagsRequest).Return(nil, nil).AnyTimes() - leakedENIs, err := service.FindLeakedENIs(instanceID) - Expect(err).Should(BeNil()) - Expect(len(leakedENIs)).Should(Equal(1)) - Expect(leakedENIs[0].ID).Should(Equal(eniID2)) - _, ok := store[cloud.TagENICreatedAt] - Expect(ok).Should(BeTrue()) - }) - }) - - Context("Alloc and dealloc private IP Addresses", func() { - It("", func() { - - var err error - _, service, mockEC2 := setup(nil) - - assignOutput := &ec2.AssignPrivateIpAddressesOutput{ - AssignedPrivateIpAddresses: []*ec2.AssignedPrivateIpAddress{ - { - PrivateIpAddress: aws.String(eniIP22), - }, - }, - } - mockEC2.EXPECT().AssignPrivateIpAddressesWithContext(gomock.Any(), gomock.Any()).Return(assignOutput, nil) - _, err = service.AllocIPAddresses(eniID2) - Expect(err).Should(BeNil()) - - mockEC2.EXPECT().AssignPrivateIpAddressesWithContext(gomock.Any(), gomock.Any()).Return(nil, errors.New("mock assign failed")) - _, err = service.AllocIPAddresses(eniID2) - Expect(err).ShouldNot(BeNil()) - - mockEC2.EXPECT().UnassignPrivateIpAddressesWithContext(gomock.Any(), gomock.Any()).Return(&ec2.UnassignPrivateIpAddressesOutput{}, nil) - Expect(service.DeallocIPAddresses(eniID2, []string{eniIP22})).Should(Succeed()) - - mockEC2.EXPECT().UnassignPrivateIpAddressesWithContext(gomock.Any(), gomock.Any()).Return(nil, errors.New("mock unassign failed")) - Expect(service.DeallocIPAddresses(eniID2, []string{eniIP22})).ShouldNot(Succeed()) - }) - }) -}) diff --git a/cmd/loadbalancer/internal/cloud/aws/generate_mocks.go b/cmd/loadbalancer/internal/cloud/aws/generate_mocks.go deleted file mode 100644 index 728712d94..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/generate_mocks.go +++ /dev/null @@ -1,19 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../../hack/boilerplate.go.txt -destination mocks/ec2_mocks.go . EC2 diff --git a/cmd/loadbalancer/internal/cloud/aws/imds.go b/cmd/loadbalancer/internal/cloud/aws/imds.go deleted file mode 100644 index 4c45cea46..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/imds.go +++ /dev/null @@ -1,162 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -import ( - "context" - "fmt" - "net/http" - "strconv" - "strings" - - "github.com/aws/aws-sdk-go/aws/awserr" -) - -type imdsService struct { - IMDS -} - -func (i *imdsService) GetMetadataWithContext(ctx context.Context, p string) (string, error) { - return i.IMDS.GetMetadataWithContext(ctx, p) -} - -func (i *imdsService) Region() (string, error) { - return i.IMDS.Region() -} - -func (i *imdsService) getList(ctx context.Context, key string) ([]string, error) { - data, err := i.GetMetadataWithContext(ctx, key) - if err != nil { - return nil, err - } - return strings.Fields(data), nil -} - -func (i *imdsService) getSecurityGroupIds(ctx context.Context, mac string) ([]string, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/security-group-ids", mac) - return i.getList(ctx, key) -} - -func (i *imdsService) getAZ(ctx context.Context) (string, error) { - return i.GetMetadataWithContext(ctx, "placement/availability-zone") -} - -func (i *imdsService) getLocalIPv4(ctx context.Context) (string, error) { - return i.GetMetadataWithContext(ctx, "local-ipv4") -} - -func (i *imdsService) getInstanceID(ctx context.Context) (string, error) { - return i.GetMetadataWithContext(ctx, "instance-id") -} - -func (i *imdsService) getInstanceType(ctx context.Context) (string, error) { - return i.GetMetadataWithContext(ctx, "instance-type") -} - -func (i *imdsService) getPrimaryMAC(ctx context.Context) (string, error) { - return i.GetMetadataWithContext(ctx, "mac") -} - -func (i *imdsService) getInterfaceIDByMAC(ctx context.Context, mac string) (string, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/interface-id", mac) - return i.GetMetadataWithContext(ctx, key) -} - -func (i *imdsService) getSubnetID(ctx context.Context, mac string) (string, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/subnet-id", mac) - return i.GetMetadataWithContext(ctx, key) -} - -func (i *imdsService) getMACs(ctx context.Context) ([]string, error) { - macs, err := i.getList(ctx, "network/interfaces/macs") - if err != nil { - return nil, err - } - for index, item := range macs { - macs[index] = strings.TrimSuffix(item, "/") - } - return macs, nil -} - -func (i *imdsService) getInterfaceDeviceNumber(ctx context.Context, mac string) (int, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/device-number", mac) - data, err := i.GetMetadataWithContext(ctx, key) - if err != nil { - return 0, err - } - n, err := strconv.Atoi(data) - if err != nil { - return 0, err - } - return n, nil -} - -func (i *imdsService) getSubnetIPv4CIDRBlock(ctx context.Context, mac string) (string, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/subnet-ipv4-cidr-block", mac) - return i.GetMetadataWithContext(ctx, key) -} - -func (i *imdsService) getInterfacePrivateAddresses(ctx context.Context, mac string) ([]string, error) { - key := fmt.Sprintf("network/interfaces/macs/%s/local-ipv4s", mac) - return i.getList(ctx, key) -} - -// imdsRequestError to provide the caller on the request status -type imdsRequestError struct { - requestKey string - err error -} - -func (e *imdsRequestError) Error() string { - return fmt.Sprintf("failed to retrieve %s from instance metadata %v", e.requestKey, e.err) -} - -func newIMDSRequestError(requestKey string, err error) *imdsRequestError { - return &imdsRequestError{ - requestKey: requestKey, - err: err, - } -} - -var _ error = &imdsRequestError{} - -// fakeIMDS is a trivial implementation of EC2MetadataIface using an in-memory map - for testing. -type fakeIMDS map[string]interface{} - -func (f fakeIMDS) Region() (string, error) { - return "mock", nil -} - -// GetMetadataWithContext implements the EC2MetadataIface interface. -func (f fakeIMDS) GetMetadataWithContext(ctx context.Context, p string) (string, error) { - result, ok := f[p] - if !ok { - result, ok = f[p+"/"] // Metadata API treats foo/ as foo - } - if !ok { - notFoundErr := awserr.NewRequestFailure(awserr.New("NotFound", "not found", nil), http.StatusNotFound, "dummy-reqid") - return "", newIMDSRequestError(p, notFoundErr) - } - switch v := result.(type) { - case string: - return v, nil - case error: - return "", v - default: - panic(fmt.Sprintf("unknown test metadata value type %T for %s", result, p)) - } -} diff --git a/cmd/loadbalancer/internal/cloud/aws/mocks/ec2_mocks.go b/cmd/loadbalancer/internal/cloud/aws/mocks/ec2_mocks.go deleted file mode 100644 index b1dcf0db3..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/mocks/ec2_mocks.go +++ /dev/null @@ -1,294 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/aws (interfaces: EC2) - -// Package mock_aws is a generated GoMock package. -package mock_aws - -import ( - context "context" - reflect "reflect" - - request "github.com/aws/aws-sdk-go/aws/request" - ec2 "github.com/aws/aws-sdk-go/service/ec2" - gomock "github.com/golang/mock/gomock" -) - -// MockEC2 is a mock of EC2 interface. -type MockEC2 struct { - ctrl *gomock.Controller - recorder *MockEC2MockRecorder -} - -// MockEC2MockRecorder is the mock recorder for MockEC2. -type MockEC2MockRecorder struct { - mock *MockEC2 -} - -// NewMockEC2 creates a new mock instance. -func NewMockEC2(ctrl *gomock.Controller) *MockEC2 { - mock := &MockEC2{ctrl: ctrl} - mock.recorder = &MockEC2MockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockEC2) EXPECT() *MockEC2MockRecorder { - return m.recorder -} - -// AssignPrivateIpAddressesWithContext mocks base method. -func (m *MockEC2) AssignPrivateIpAddressesWithContext(arg0 context.Context, arg1 *ec2.AssignPrivateIpAddressesInput, arg2 ...request.Option) (*ec2.AssignPrivateIpAddressesOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "AssignPrivateIpAddressesWithContext", varargs...) - ret0, _ := ret[0].(*ec2.AssignPrivateIpAddressesOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// AssignPrivateIpAddressesWithContext indicates an expected call of AssignPrivateIpAddressesWithContext. -func (mr *MockEC2MockRecorder) AssignPrivateIpAddressesWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AssignPrivateIpAddressesWithContext", reflect.TypeOf((*MockEC2)(nil).AssignPrivateIpAddressesWithContext), varargs...) -} - -// AttachNetworkInterfaceWithContext mocks base method. -func (m *MockEC2) AttachNetworkInterfaceWithContext(arg0 context.Context, arg1 *ec2.AttachNetworkInterfaceInput, arg2 ...request.Option) (*ec2.AttachNetworkInterfaceOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "AttachNetworkInterfaceWithContext", varargs...) - ret0, _ := ret[0].(*ec2.AttachNetworkInterfaceOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// AttachNetworkInterfaceWithContext indicates an expected call of AttachNetworkInterfaceWithContext. -func (mr *MockEC2MockRecorder) AttachNetworkInterfaceWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AttachNetworkInterfaceWithContext", reflect.TypeOf((*MockEC2)(nil).AttachNetworkInterfaceWithContext), varargs...) -} - -// CreateNetworkInterfaceWithContext mocks base method. -func (m *MockEC2) CreateNetworkInterfaceWithContext(arg0 context.Context, arg1 *ec2.CreateNetworkInterfaceInput, arg2 ...request.Option) (*ec2.CreateNetworkInterfaceOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "CreateNetworkInterfaceWithContext", varargs...) - ret0, _ := ret[0].(*ec2.CreateNetworkInterfaceOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// CreateNetworkInterfaceWithContext indicates an expected call of CreateNetworkInterfaceWithContext. -func (mr *MockEC2MockRecorder) CreateNetworkInterfaceWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateNetworkInterfaceWithContext", reflect.TypeOf((*MockEC2)(nil).CreateNetworkInterfaceWithContext), varargs...) -} - -// CreateTagsWithContext mocks base method. -func (m *MockEC2) CreateTagsWithContext(arg0 context.Context, arg1 *ec2.CreateTagsInput, arg2 ...request.Option) (*ec2.CreateTagsOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "CreateTagsWithContext", varargs...) - ret0, _ := ret[0].(*ec2.CreateTagsOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// CreateTagsWithContext indicates an expected call of CreateTagsWithContext. -func (mr *MockEC2MockRecorder) CreateTagsWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateTagsWithContext", reflect.TypeOf((*MockEC2)(nil).CreateTagsWithContext), varargs...) -} - -// DeleteNetworkInterfaceWithContext mocks base method. -func (m *MockEC2) DeleteNetworkInterfaceWithContext(arg0 context.Context, arg1 *ec2.DeleteNetworkInterfaceInput, arg2 ...request.Option) (*ec2.DeleteNetworkInterfaceOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DeleteNetworkInterfaceWithContext", varargs...) - ret0, _ := ret[0].(*ec2.DeleteNetworkInterfaceOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DeleteNetworkInterfaceWithContext indicates an expected call of DeleteNetworkInterfaceWithContext. -func (mr *MockEC2MockRecorder) DeleteNetworkInterfaceWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteNetworkInterfaceWithContext", reflect.TypeOf((*MockEC2)(nil).DeleteNetworkInterfaceWithContext), varargs...) -} - -// DescribeInstanceTypesWithContext mocks base method. -func (m *MockEC2) DescribeInstanceTypesWithContext(arg0 context.Context, arg1 *ec2.DescribeInstanceTypesInput, arg2 ...request.Option) (*ec2.DescribeInstanceTypesOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeInstanceTypesWithContext", varargs...) - ret0, _ := ret[0].(*ec2.DescribeInstanceTypesOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeInstanceTypesWithContext indicates an expected call of DescribeInstanceTypesWithContext. -func (mr *MockEC2MockRecorder) DescribeInstanceTypesWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeInstanceTypesWithContext", reflect.TypeOf((*MockEC2)(nil).DescribeInstanceTypesWithContext), varargs...) -} - -// DescribeInstancesWithContext mocks base method. -func (m *MockEC2) DescribeInstancesWithContext(arg0 context.Context, arg1 *ec2.DescribeInstancesInput, arg2 ...request.Option) (*ec2.DescribeInstancesOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeInstancesWithContext", varargs...) - ret0, _ := ret[0].(*ec2.DescribeInstancesOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeInstancesWithContext indicates an expected call of DescribeInstancesWithContext. -func (mr *MockEC2MockRecorder) DescribeInstancesWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeInstancesWithContext", reflect.TypeOf((*MockEC2)(nil).DescribeInstancesWithContext), varargs...) -} - -// DescribeNetworkInterfacesPagesWithContext mocks base method. -func (m *MockEC2) DescribeNetworkInterfacesPagesWithContext(arg0 context.Context, arg1 *ec2.DescribeNetworkInterfacesInput, arg2 func(*ec2.DescribeNetworkInterfacesOutput, bool) bool, arg3 ...request.Option) error { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1, arg2} - for _, a := range arg3 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeNetworkInterfacesPagesWithContext", varargs...) - ret0, _ := ret[0].(error) - return ret0 -} - -// DescribeNetworkInterfacesPagesWithContext indicates an expected call of DescribeNetworkInterfacesPagesWithContext. -func (mr *MockEC2MockRecorder) DescribeNetworkInterfacesPagesWithContext(arg0, arg1, arg2 interface{}, arg3 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1, arg2}, arg3...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeNetworkInterfacesPagesWithContext", reflect.TypeOf((*MockEC2)(nil).DescribeNetworkInterfacesPagesWithContext), varargs...) -} - -// DescribeNetworkInterfacesWithContext mocks base method. -func (m *MockEC2) DescribeNetworkInterfacesWithContext(arg0 context.Context, arg1 *ec2.DescribeNetworkInterfacesInput, arg2 ...request.Option) (*ec2.DescribeNetworkInterfacesOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeNetworkInterfacesWithContext", varargs...) - ret0, _ := ret[0].(*ec2.DescribeNetworkInterfacesOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeNetworkInterfacesWithContext indicates an expected call of DescribeNetworkInterfacesWithContext. -func (mr *MockEC2MockRecorder) DescribeNetworkInterfacesWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeNetworkInterfacesWithContext", reflect.TypeOf((*MockEC2)(nil).DescribeNetworkInterfacesWithContext), varargs...) -} - -// DetachNetworkInterfaceWithContext mocks base method. -func (m *MockEC2) DetachNetworkInterfaceWithContext(arg0 context.Context, arg1 *ec2.DetachNetworkInterfaceInput, arg2 ...request.Option) (*ec2.DetachNetworkInterfaceOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DetachNetworkInterfaceWithContext", varargs...) - ret0, _ := ret[0].(*ec2.DetachNetworkInterfaceOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DetachNetworkInterfaceWithContext indicates an expected call of DetachNetworkInterfaceWithContext. -func (mr *MockEC2MockRecorder) DetachNetworkInterfaceWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DetachNetworkInterfaceWithContext", reflect.TypeOf((*MockEC2)(nil).DetachNetworkInterfaceWithContext), varargs...) -} - -// ModifyNetworkInterfaceAttributeWithContext mocks base method. -func (m *MockEC2) ModifyNetworkInterfaceAttributeWithContext(arg0 context.Context, arg1 *ec2.ModifyNetworkInterfaceAttributeInput, arg2 ...request.Option) (*ec2.ModifyNetworkInterfaceAttributeOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "ModifyNetworkInterfaceAttributeWithContext", varargs...) - ret0, _ := ret[0].(*ec2.ModifyNetworkInterfaceAttributeOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// ModifyNetworkInterfaceAttributeWithContext indicates an expected call of ModifyNetworkInterfaceAttributeWithContext. -func (mr *MockEC2MockRecorder) ModifyNetworkInterfaceAttributeWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ModifyNetworkInterfaceAttributeWithContext", reflect.TypeOf((*MockEC2)(nil).ModifyNetworkInterfaceAttributeWithContext), varargs...) -} - -// UnassignPrivateIpAddressesWithContext mocks base method. -func (m *MockEC2) UnassignPrivateIpAddressesWithContext(arg0 context.Context, arg1 *ec2.UnassignPrivateIpAddressesInput, arg2 ...request.Option) (*ec2.UnassignPrivateIpAddressesOutput, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "UnassignPrivateIpAddressesWithContext", varargs...) - ret0, _ := ret[0].(*ec2.UnassignPrivateIpAddressesOutput) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// UnassignPrivateIpAddressesWithContext indicates an expected call of UnassignPrivateIpAddressesWithContext. -func (mr *MockEC2MockRecorder) UnassignPrivateIpAddressesWithContext(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "UnassignPrivateIpAddressesWithContext", reflect.TypeOf((*MockEC2)(nil).UnassignPrivateIpAddressesWithContext), varargs...) -} diff --git a/cmd/loadbalancer/internal/cloud/aws/suite_test.go b/cmd/loadbalancer/internal/cloud/aws/suite_test.go deleted file mode 100644 index 8bb82a036..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/suite_test.go +++ /dev/null @@ -1,52 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -import ( - "testing" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/spf13/viper" - "go.uber.org/zap/zapcore" - "sigs.k8s.io/controller-runtime/pkg/log/zap" -) - -func init() { - viper.AutomaticEnv() -} - -var ( - logger = zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true), func(o *zap.Options) { - o.TimeEncoder = zapcore.ISO8601TimeEncoder - }) -) - -func TestAws(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecs(t, "AWS Cloud Provider Test Suite") -} - -var _ = BeforeSuite(func() { - -}) - -var _ = AfterSuite(func() { - -}) diff --git a/cmd/loadbalancer/internal/cloud/aws/types.go b/cmd/loadbalancer/internal/cloud/aws/types.go deleted file mode 100644 index 2c99a41f2..000000000 --- a/cmd/loadbalancer/internal/cloud/aws/types.go +++ /dev/null @@ -1,57 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package aws - -import ( - "context" - - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/aws/request" - "github.com/aws/aws-sdk-go/service/ec2" -) - -type EC2 interface { - CreateNetworkInterfaceWithContext(ctx aws.Context, input *ec2.CreateNetworkInterfaceInput, opts ...request.Option) (*ec2.CreateNetworkInterfaceOutput, error) - - DescribeInstancesWithContext(ctx aws.Context, input *ec2.DescribeInstancesInput, opts ...request.Option) (*ec2.DescribeInstancesOutput, error) - - DescribeInstanceTypesWithContext(ctx aws.Context, input *ec2.DescribeInstanceTypesInput, opts ...request.Option) (*ec2.DescribeInstanceTypesOutput, error) - - AttachNetworkInterfaceWithContext(ctx aws.Context, input *ec2.AttachNetworkInterfaceInput, opts ...request.Option) (*ec2.AttachNetworkInterfaceOutput, error) - - DeleteNetworkInterfaceWithContext(ctx aws.Context, input *ec2.DeleteNetworkInterfaceInput, opts ...request.Option) (*ec2.DeleteNetworkInterfaceOutput, error) - - DetachNetworkInterfaceWithContext(ctx aws.Context, input *ec2.DetachNetworkInterfaceInput, opts ...request.Option) (*ec2.DetachNetworkInterfaceOutput, error) - - AssignPrivateIpAddressesWithContext(ctx aws.Context, input *ec2.AssignPrivateIpAddressesInput, opts ...request.Option) (*ec2.AssignPrivateIpAddressesOutput, error) - - UnassignPrivateIpAddressesWithContext(ctx aws.Context, input *ec2.UnassignPrivateIpAddressesInput, opts ...request.Option) (*ec2.UnassignPrivateIpAddressesOutput, error) - - DescribeNetworkInterfacesWithContext(ctx aws.Context, input *ec2.DescribeNetworkInterfacesInput, opts ...request.Option) (*ec2.DescribeNetworkInterfacesOutput, error) - - ModifyNetworkInterfaceAttributeWithContext(ctx aws.Context, input *ec2.ModifyNetworkInterfaceAttributeInput, opts ...request.Option) (*ec2.ModifyNetworkInterfaceAttributeOutput, error) - - CreateTagsWithContext(ctx aws.Context, input *ec2.CreateTagsInput, opts ...request.Option) (*ec2.CreateTagsOutput, error) - - DescribeNetworkInterfacesPagesWithContext(ctx aws.Context, input *ec2.DescribeNetworkInterfacesInput, fn func(*ec2.DescribeNetworkInterfacesOutput, bool) bool, opts ...request.Option) error -} - -type IMDS interface { - Region() (string, error) - - GetMetadataWithContext(ctx context.Context, p string) (string, error) -} diff --git a/cmd/loadbalancer/internal/cloud/factory/factory.go b/cmd/loadbalancer/internal/cloud/factory/factory.go deleted file mode 100644 index a64c51af6..000000000 --- a/cmd/loadbalancer/internal/cloud/factory/factory.go +++ /dev/null @@ -1,60 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package factory - -import ( - "fmt" - "sync" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/aws" -) - -type newFunc func(...interface{}) (cloud.Provider, error) - -var ( - lock sync.RWMutex - providers = make(map[string]newFunc) -) - -func init() { - RegisterProvider(cloud.ProviderAWS, func(args ...interface{}) (cloud.Provider, error) { - return aws.NewAwsService(args[0].(logr.Logger)) - }) -} - -func NewProvider(name string, logger logr.Logger) (cloud.Provider, error) { - lock.RLock() - defer lock.RUnlock() - f, ok := providers[name] - if !ok { - return nil, errors.New("Unknown cloud provider") - } - return f(logger) -} - -func RegisterProvider(name string, f newFunc) { - lock.Lock() - defer lock.Unlock() - if _, ok := providers[name]; ok { - panic(fmt.Sprintf("Cloud provider %s exists", name)) - } - providers[name] = f -} diff --git a/cmd/loadbalancer/internal/cloud/generate_mocks.go b/cmd/loadbalancer/internal/cloud/generate_mocks.go deleted file mode 100644 index b6dca4a63..000000000 --- a/cmd/loadbalancer/internal/cloud/generate_mocks.go +++ /dev/null @@ -1,19 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package cloud - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/provider_mocks.go . Provider diff --git a/cmd/loadbalancer/internal/cloud/mocks/provider_mocks.go b/cmd/loadbalancer/internal/cloud/mocks/provider_mocks.go deleted file mode 100644 index 840adbb2e..000000000 --- a/cmd/loadbalancer/internal/cloud/mocks/provider_mocks.go +++ /dev/null @@ -1,241 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud (interfaces: Provider) - -// Package mock_cloud is a generated GoMock package. -package mock_cloud - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - - cloud "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" -) - -// MockProvider is a mock of Provider interface. -type MockProvider struct { - ctrl *gomock.Controller - recorder *MockProviderMockRecorder -} - -// MockProviderMockRecorder is the mock recorder for MockProvider. -type MockProviderMockRecorder struct { - mock *MockProvider -} - -// NewMockProvider creates a new mock instance. -func NewMockProvider(ctrl *gomock.Controller) *MockProvider { - mock := &MockProvider{ctrl: ctrl} - mock.recorder = &MockProviderMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockProvider) EXPECT() *MockProviderMockRecorder { - return m.recorder -} - -// AllocIPAddresses mocks base method. -func (m *MockProvider) AllocIPAddresses(arg0 string) (string, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AllocIPAddresses", arg0) - ret0, _ := ret[0].(string) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// AllocIPAddresses indicates an expected call of AllocIPAddresses. -func (mr *MockProviderMockRecorder) AllocIPAddresses(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AllocIPAddresses", reflect.TypeOf((*MockProvider)(nil).AllocIPAddresses), arg0) -} - -// AssignPrivateIPAddresses mocks base method. -func (m *MockProvider) AssignPrivateIPAddresses(arg0, arg1 string) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AssignPrivateIPAddresses", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// AssignPrivateIPAddresses indicates an expected call of AssignPrivateIPAddresses. -func (mr *MockProviderMockRecorder) AssignPrivateIPAddresses(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AssignPrivateIPAddresses", reflect.TypeOf((*MockProvider)(nil).AssignPrivateIPAddresses), arg0, arg1) -} - -// AttachENI mocks base method. -func (m *MockProvider) AttachENI(arg0, arg1 string) (string, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AttachENI", arg0, arg1) - ret0, _ := ret[0].(string) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// AttachENI indicates an expected call of AttachENI. -func (mr *MockProviderMockRecorder) AttachENI(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AttachENI", reflect.TypeOf((*MockProvider)(nil).AttachENI), arg0, arg1) -} - -// CreateENI mocks base method. -func (m *MockProvider) CreateENI(arg0, arg1 string, arg2 []string) (string, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "CreateENI", arg0, arg1, arg2) - ret0, _ := ret[0].(string) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// CreateENI indicates an expected call of CreateENI. -func (mr *MockProviderMockRecorder) CreateENI(arg0, arg1, arg2 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CreateENI", reflect.TypeOf((*MockProvider)(nil).CreateENI), arg0, arg1, arg2) -} - -// DeallocIPAddresses mocks base method. -func (m *MockProvider) DeallocIPAddresses(arg0 string, arg1 []string) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "DeallocIPAddresses", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// DeallocIPAddresses indicates an expected call of DeallocIPAddresses. -func (mr *MockProviderMockRecorder) DeallocIPAddresses(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeallocIPAddresses", reflect.TypeOf((*MockProvider)(nil).DeallocIPAddresses), arg0, arg1) -} - -// DeleteENI mocks base method. -func (m *MockProvider) DeleteENI(arg0 string) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "DeleteENI", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// DeleteENI indicates an expected call of DeleteENI. -func (mr *MockProviderMockRecorder) DeleteENI(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DeleteENI", reflect.TypeOf((*MockProvider)(nil).DeleteENI), arg0) -} - -// DescribeAllENIs mocks base method. -func (m *MockProvider) DescribeAllENIs() (map[string]*cloud.ENIMetadata, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "DescribeAllENIs") - ret0, _ := ret[0].(map[string]*cloud.ENIMetadata) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeAllENIs indicates an expected call of DescribeAllENIs. -func (mr *MockProviderMockRecorder) DescribeAllENIs() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeAllENIs", reflect.TypeOf((*MockProvider)(nil).DescribeAllENIs)) -} - -// FindLeakedENIs mocks base method. -func (m *MockProvider) FindLeakedENIs(arg0 string) ([]*cloud.ENIMetadata, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FindLeakedENIs", arg0) - ret0, _ := ret[0].([]*cloud.ENIMetadata) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// FindLeakedENIs indicates an expected call of FindLeakedENIs. -func (mr *MockProviderMockRecorder) FindLeakedENIs(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FindLeakedENIs", reflect.TypeOf((*MockProvider)(nil).FindLeakedENIs), arg0) -} - -// FreeENI mocks base method. -func (m *MockProvider) FreeENI(arg0 string) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "FreeENI", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// FreeENI indicates an expected call of FreeENI. -func (mr *MockProviderMockRecorder) FreeENI(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "FreeENI", reflect.TypeOf((*MockProvider)(nil).FreeENI), arg0) -} - -// GetENIIPv4Limit mocks base method. -func (m *MockProvider) GetENIIPv4Limit() int { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetENIIPv4Limit") - ret0, _ := ret[0].(int) - return ret0 -} - -// GetENIIPv4Limit indicates an expected call of GetENIIPv4Limit. -func (mr *MockProviderMockRecorder) GetENIIPv4Limit() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetENIIPv4Limit", reflect.TypeOf((*MockProvider)(nil).GetENIIPv4Limit)) -} - -// GetENILimit mocks base method. -func (m *MockProvider) GetENILimit() int { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetENILimit") - ret0, _ := ret[0].(int) - return ret0 -} - -// GetENILimit indicates an expected call of GetENILimit. -func (mr *MockProviderMockRecorder) GetENILimit() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetENILimit", reflect.TypeOf((*MockProvider)(nil).GetENILimit)) -} - -// GetInstanceInfo mocks base method. -func (m *MockProvider) GetInstanceInfo() *cloud.InstanceInfo { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "GetInstanceInfo") - ret0, _ := ret[0].(*cloud.InstanceInfo) - return ret0 -} - -// GetInstanceInfo indicates an expected call of GetInstanceInfo. -func (mr *MockProviderMockRecorder) GetInstanceInfo() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "GetInstanceInfo", reflect.TypeOf((*MockProvider)(nil).GetInstanceInfo)) -} - -// ModifySourceDestCheck mocks base method. -func (m *MockProvider) ModifySourceDestCheck(arg0 string, arg1 bool) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "ModifySourceDestCheck", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// ModifySourceDestCheck indicates an expected call of ModifySourceDestCheck. -func (mr *MockProviderMockRecorder) ModifySourceDestCheck(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ModifySourceDestCheck", reflect.TypeOf((*MockProvider)(nil).ModifySourceDestCheck), arg0, arg1) -} diff --git a/cmd/loadbalancer/internal/cloud/types.go b/cmd/loadbalancer/internal/cloud/types.go deleted file mode 100644 index f73b84b89..000000000 --- a/cmd/loadbalancer/internal/cloud/types.go +++ /dev/null @@ -1,99 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package cloud - -const ( - ProviderAWS = "aws" - TagENICreatedAt = "kubeblocks.io/created-at" - TagENINode = "kubeblocks.io/instance-id" - TagENIKubeBlocksManaged = "kubeblocks.io/managed" -) - -type Provider interface { - GetENILimit() int - - GetENIIPv4Limit() int - - GetInstanceInfo() *InstanceInfo - - CreateENI(instanceID, subnetID string, securityGroupIDs []string) (string, error) - - AttachENI(instanceID string, eniID string) (string, error) - - DeleteENI(eniID string) error - - FreeENI(eniID string) error - - DescribeAllENIs() (map[string]*ENIMetadata, error) - - FindLeakedENIs(instanceID string) ([]*ENIMetadata, error) - - AllocIPAddresses(eniID string) (string, error) - - DeallocIPAddresses(eniID string, ips []string) error - - AssignPrivateIPAddresses(eniID string, ip string) error - - ModifySourceDestCheck(eniID string, enabled bool) error -} - -type InstanceInfo struct { - InstanceID string `json:"instance_id"` - - SubnetID string `json:"subnet_id"` - - SecurityGroupIDs []string `json:"security_group_ids"` -} - -type IPv4Address struct { - Primary bool - - Address string -} - -type ENIMetadata struct { - // ID is the id of network interface - ID string - - // MAC is the mac address of network interface - MAC string - - // DeviceNumber is the device number of network interface - // 0 means it is primary interface - DeviceNumber int - - // SubnetID is the subnet id of network interface - SubnetID string - - // SubnetIPv4CIDR is the IPv4 CIDR of network interface - SubnetIPv4CIDR string - - // The ip addresses allocated for the network interface - IPv4Addresses []*IPv4Address - - // Tags is the tag set of network interface - Tags map[string]string -} - -func (eni ENIMetadata) PrimaryIPv4Address() string { - for _, addr := range eni.IPv4Addresses { - if addr.Primary { - return addr.Address - } - } - return "" -} diff --git a/cmd/loadbalancer/internal/config/config.go b/cmd/loadbalancer/internal/config/config.go deleted file mode 100644 index 0ba7cf8d4..000000000 --- a/cmd/loadbalancer/internal/config/config.go +++ /dev/null @@ -1,135 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package config - -import ( - "fmt" - "runtime" - "strings" - "time" - - "github.com/go-logr/logr" - "github.com/spf13/viper" -) - -var ( - HostIP string - MaxENI int - MinPrivateIP int - EnableDebug bool - RPCPort int - CleanLeakedENIInterval time.Duration - ENIReconcileInterval time.Duration - RefreshNodeInterval time.Duration - TrafficNodeLabels map[string]string - EndpointsLabels map[string]string - ServiceLabels map[string]string - MaxConcurrentReconciles int - CloudProvider string -) - -const ( - EnvHostIP = "HOST_IP" - EnvMaxENI = "MAX_ENI" - EnvMinPrivateIP = "MIN_PRIVATE_IP" - EnvEnableDebug = "ENABLE_DEBUG" - EnvRPCPort = "RPC_PORT" - EnvENIReconcileInterval = "ENI_RECONCILE_INTERVAL" - EnvCleanLeakedENIInterval = "CLEAN_LEAKED_ENI_INTERVAL" - EnvRefreshNodes = "REFRESH_NODES_INTERVAL" - EnvTrafficNodeLabels = "TRAFFIC_NODE_LABELS" - EnvEndpointsLabels = "ENDPOINTS_LABELS" - EnvServiceLabels = "SERVICE_LABELS" - EnvMaxConcurrentReconciles = "MAX_CONCURRENT_RECONCILES" - EnvCloudProvider = "CLOUD_PROVIDER" -) - -func init() { - _ = viper.BindEnv(EnvHostIP) - - _ = viper.BindEnv(EnvMaxENI) - viper.SetDefault(EnvMaxENI, -1) - - _ = viper.BindEnv(EnvMinPrivateIP) - viper.SetDefault(EnvMinPrivateIP, 1) - - _ = viper.BindEnv(EnvENIReconcileInterval) - viper.SetDefault(EnvENIReconcileInterval, 15) - - _ = viper.BindEnv(EnvCleanLeakedENIInterval) - viper.SetDefault(EnvCleanLeakedENIInterval, 60) - - _ = viper.BindEnv(EnvRefreshNodes) - viper.SetDefault(EnvRefreshNodes, 15) - - _ = viper.BindEnv(EnvRPCPort) - viper.SetDefault(EnvRPCPort, 19200) - - _ = viper.BindEnv(EnvEnableDebug) - viper.SetDefault(EnvEnableDebug, false) - - _ = viper.BindEnv(EnvTrafficNodeLabels) - viper.SetDefault(EnvTrafficNodeLabels, "") - - _ = viper.BindEnv(EnvEndpointsLabels) - viper.SetDefault(EnvEndpointsLabels, "") - - _ = viper.BindEnv(EnvServiceLabels) - viper.SetDefault(EnvServiceLabels, "") - - _ = viper.BindEnv(EnvMaxConcurrentReconciles) - viper.SetDefault(EnvMaxConcurrentReconciles, runtime.NumCPU()*2) - - _ = viper.BindEnv(EnvCloudProvider) -} - -func ReadConfig(logger logr.Logger) { - err := viper.ReadInConfig() // Find and read the config file - if err == nil { // Handle errors reading the config file - logger.Info(fmt.Sprintf("config file: %s", viper.GetViper().ConfigFileUsed())) - } - - HostIP = viper.GetString(EnvHostIP) - MaxENI = viper.GetInt(EnvMaxENI) - MinPrivateIP = viper.GetInt(EnvMinPrivateIP) - EnableDebug = viper.GetBool(EnvEnableDebug) - RPCPort = viper.GetInt(EnvRPCPort) - ENIReconcileInterval = time.Duration(viper.GetInt(EnvENIReconcileInterval)) * time.Second - CleanLeakedENIInterval = time.Duration(viper.GetInt(EnvCleanLeakedENIInterval)) * time.Second - RefreshNodeInterval = time.Duration(viper.GetInt(EnvRefreshNodes)) * time.Second - TrafficNodeLabels = ParseLabels(viper.GetString(EnvTrafficNodeLabels)) - EndpointsLabels = ParseLabels(viper.GetString(EnvEndpointsLabels)) - ServiceLabels = ParseLabels(viper.GetString(EnvServiceLabels)) - MaxConcurrentReconciles = viper.GetInt(EnvMaxConcurrentReconciles) - CloudProvider = viper.GetString(EnvCloudProvider) -} - -func ParseLabels(labels string) map[string]string { - result := make(map[string]string) - for _, label := range strings.Split(labels, ",") { - label = strings.TrimSpace(label) - if label == "" { - continue - } - parts := strings.SplitN(label, ":", 2) - if len(parts) != 2 { - continue - } - result[strings.TrimSpace(parts[0])] = strings.TrimSpace(parts[1]) - } - return result -} diff --git a/cmd/loadbalancer/internal/config/suite_test.go b/cmd/loadbalancer/internal/config/suite_test.go deleted file mode 100644 index 3e38e47ab..000000000 --- a/cmd/loadbalancer/internal/config/suite_test.go +++ /dev/null @@ -1,64 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package config - -import ( - "testing" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "github.com/spf13/viper" - "go.uber.org/zap/zapcore" - "sigs.k8s.io/controller-runtime/pkg/log/zap" -) - -func init() { - viper.AutomaticEnv() -} - -var ( - logger = zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true), func(o *zap.Options) { - o.TimeEncoder = zapcore.ISO8601TimeEncoder - }) -) - -func TestConfig(t *testing.T) { - RegisterFailHandler(Fail) - RunSpecs(t, "Config Suite") -} - -var _ = Describe("Config", func() { - Context("", func() { - It("", func() { - cases := []struct { - input string - output map[string]string - }{ - {input: "", output: map[string]string{}}, - {input: "a:b,abcd", output: map[string]string{"a": "b"}}, - {input: "a:b:c:d,c:d", output: map[string]string{"a": "b:c:d", "c": "d"}}, - {input: "a:b,c:d", output: map[string]string{"a": "b", "c": "d"}}, - } - - ReadConfig(logger) - - for _, item := range cases { - Expect(ParseLabels(item.input)).Should(Equal(item.output)) - } - }) - }) -}) diff --git a/cmd/loadbalancer/internal/controllers/endpoint_controller.go b/cmd/loadbalancer/internal/controllers/endpoint_controller.go deleted file mode 100644 index 92a3573b8..000000000 --- a/cmd/loadbalancer/internal/controllers/endpoint_controller.go +++ /dev/null @@ -1,116 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - - "github.com/go-logr/logr" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/client-go/tools/record" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/builder" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/controller" - "sigs.k8s.io/controller-runtime/pkg/predicate" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" -) - -const ( - AnnotationKeyEndpointsVersion = "service.kubernetes.io/kubeblocks-loadbalancer-endpoints-version" -) - -var endpointsFilterPredicate = func(object client.Object) bool { - for k, v := range config.EndpointsLabels { - if object.GetLabels()[k] != v { - return false - } - } - return true -} - -type EndpointController struct { - client.Client - Scheme *runtime.Scheme - Recorder record.EventRecorder - - logger logr.Logger -} - -func NewEndpointController(logger logr.Logger, client client.Client, scheme *runtime.Scheme, recorder record.EventRecorder) (*EndpointController, error) { - return &EndpointController{ - logger: logger, - Client: client, - Scheme: scheme, - Recorder: recorder, - }, nil -} - -func (c *EndpointController) Start(ctx context.Context) error { - return nil -} - -func (c *EndpointController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - ctxLog := c.logger.WithValues("endpoints", req.NamespacedName.String()) - ctxLog.Info("Receive endpoint reconcile event") - - endpoints := &corev1.Endpoints{} - if err := c.Client.Get(ctx, req.NamespacedName, endpoints); err != nil { - return intctrlutil.CheckedRequeueWithError(err, c.logger, "") - } - - listOptions := []client.ListOption{client.InNamespace(endpoints.GetNamespace())} - for k, v := range endpoints.GetObjectMeta().GetLabels() { - listOptions = append(listOptions, client.MatchingLabels{k: v}) - } - - services := &corev1.ServiceList{} - if err := c.Client.List(ctx, services, listOptions...); err != nil { - return intctrlutil.CheckedRequeueWithError(err, c.logger, "") - } - - for i := range services.Items { - service := &services.Items[i] - if _, ok := service.Annotations[AnnotationKeyLoadBalancerType]; !ok { - ctxLog.Info("Ignore unrelated endpoints") - return intctrlutil.Reconciled() - } - - // endpoint changed, trigger service reconcile - annotations := service.Annotations - if annotations == nil { - annotations = make(map[string]string) - } - annotations[AnnotationKeyEndpointsVersion] = endpoints.GetObjectMeta().GetResourceVersion() - service.SetAnnotations(annotations) - if err := c.Client.Update(ctx, service); err != nil { - return intctrlutil.CheckedRequeueWithError(err, c.logger, "") - } - ctxLog.Info("Successfully updated service") - } - - return intctrlutil.Reconciled() -} - -func (c *EndpointController) SetupWithManager(mgr ctrl.Manager) error { - return ctrl.NewControllerManagedBy(mgr).WithOptions(controller.Options{ - MaxConcurrentReconciles: config.MaxConcurrentReconciles, - }).For(&corev1.Endpoints{}, builder.WithPredicates(predicate.NewPredicateFuncs(endpointsFilterPredicate))).Complete(c) -} diff --git a/cmd/loadbalancer/internal/controllers/endpoint_controller_test.go b/cmd/loadbalancer/internal/controllers/endpoint_controller_test.go deleted file mode 100644 index 902acce93..000000000 --- a/cmd/loadbalancer/internal/controllers/endpoint_controller_test.go +++ /dev/null @@ -1,88 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/types" - "sigs.k8s.io/controller-runtime/pkg/client" - - "github.com/apecloud/kubeblocks/internal/constant" - intctrlutil "github.com/apecloud/kubeblocks/internal/generics" - testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" -) - -var newEndpointsObj = func(svc *corev1.Service) (*corev1.Endpoints, types.NamespacedName) { - endpoints := &corev1.Endpoints{ - ObjectMeta: metav1.ObjectMeta{ - Name: svc.GetName(), - Namespace: svc.GetNamespace(), - Labels: map[string]string{ - constant.AppNameLabelKey: svc.GetName(), - }, - }, - } - return endpoints, types.NamespacedName{ - Name: endpoints.GetName(), - Namespace: endpoints.GetNamespace(), - } -} - -var _ = Describe("EndpointController", func() { - cleanEnv := func() { - // must wait until resources deleted and no longer exist before the testcases start, - // otherwise if later it needs to create some new resource objects with the same name, - // in race conditions, it will find the existence of old objects, resulting failure to - // create the new objects. - By("clean resources") - - // delete rest mocked objects - inNS := client.InNamespace(testCtx.DefaultNamespace) - ml := client.HasLabels{testCtx.TestObjLabelKey} - // namespaced - testapps.ClearResources(&testCtx, intctrlutil.ServiceSignature, inNS, ml) - testapps.ClearResources(&testCtx, intctrlutil.EndpointsSignature, inNS, ml) - testapps.ClearResources(&testCtx, intctrlutil.PodSignature, inNS, ml) - } - - BeforeEach(cleanEnv) - - AfterEach(cleanEnv) - - It("Check endpoint event", func() { - svc := newSvcObj(false, node1IP) - svcKey := client.ObjectKey{Namespace: svc.GetNamespace(), Name: svc.GetName()} - ep, epKey := newEndpointsObj(svc) - Expect(testCtx.CheckedCreateObj(context.Background(), svc)).Should(Succeed()) - Expect(testCtx.CheckedCreateObj(context.Background(), ep)).Should(Succeed()) - Eventually(func() bool { - if err := k8sClient.Get(context.Background(), svcKey, svc); err != nil { - return false - } - if err := k8sClient.Get(context.Background(), epKey, ep); err != nil { - return false - } - return svc.Annotations[AnnotationKeyEndpointsVersion] == ep.GetObjectMeta().GetResourceVersion() - }).Should(BeTrue()) - }) -}) diff --git a/cmd/loadbalancer/internal/controllers/rbac.go b/cmd/loadbalancer/internal/controllers/rbac.go deleted file mode 100644 index 490b0a3d0..000000000 --- a/cmd/loadbalancer/internal/controllers/rbac.go +++ /dev/null @@ -1,26 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -// read + update access -// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;update - -// read only + watch access -// +kubebuilder:rbac:groups=core,resources=pods;endpoints,verbs=get;list;watch - -// read only access -// +kubebuilder:rbac:groups=core,resources=nodes,verbs=get;list diff --git a/cmd/loadbalancer/internal/controllers/service_controller.go b/cmd/loadbalancer/internal/controllers/service_controller.go deleted file mode 100644 index b3b90427d..000000000 --- a/cmd/loadbalancer/internal/controllers/service_controller.go +++ /dev/null @@ -1,508 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - "fmt" - "strings" - "sync" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/client-go/tools/record" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/builder" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/controller" - "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" - "sigs.k8s.io/controller-runtime/pkg/predicate" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" - intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" -) - -const ( - FinalizerKey = "service.kubernetes.io/kubeblocks-loadbalancer-finalizer" - - AnnotationKeyENIId = "service.kubernetes.io/kubeblocks-loadbalancer-eni-id" - AnnotationKeyENINodeIP = "service.kubernetes.io/kubeblocks-loadbalancer-eni-node-ip" - AnnotationKeyFloatingIP = "service.kubernetes.io/kubeblocks-loadbalancer-floating-ip" - AnnotationKeySubnetID = "service.kubernetes.io/kubeblocks-loadbalancer-subnet-id" - AnnotationKeyMasterNodeIP = "service.kubernetes.io/kubeblocks-loadbalancer-master-node-ip" - - AnnotationKeyLoadBalancerType = "service.kubernetes.io/kubeblocks-loadbalancer-type" - AnnotationValueLoadBalancerTypePrivateIP = "private-ip" - AnnotationValueLoadBalancerTypeNone = "none" - - AnnotationKeyTrafficPolicy = "service.kubernetes.io/kubeblocks-loadbalancer-traffic-policy" - AnnotationValueClusterTrafficPolicy = "Cluster" - AnnotationValueLocalTrafficPolicy = "Local" - AnnotationValueBestEffortLocalTrafficPolicy = "BestEffortLocal" - DefaultTrafficPolicy = AnnotationValueClusterTrafficPolicy -) - -var serviceFilterPredicate = func(object client.Object) bool { - for k, v := range config.ServiceLabels { - if object.GetLabels()[k] != v { - return false - } - } - return true -} - -type FloatingIP struct { - ip string - subnetID string - nodeIP string - eni *pb.ENIMetadata -} - -type ServiceController struct { - sync.RWMutex - client.Client - Scheme *runtime.Scheme - Recorder record.EventRecorder - logger logr.Logger - cp cloud.Provider - nm agent.NodeManager - tps map[string]TrafficPolicy - cache map[string]*FloatingIP -} - -func NewServiceController(logger logr.Logger, client client.Client, scheme *runtime.Scheme, recorder record.EventRecorder, cp cloud.Provider, nm agent.NodeManager) (*ServiceController, error) { - c := &ServiceController{ - Client: client, - Scheme: scheme, - Recorder: recorder, - logger: logger, - cp: cp, - nm: nm, - cache: make(map[string]*FloatingIP), - } - - c.initTrafficPolicies() - - return c, nil -} - -func (c *ServiceController) Start(ctx context.Context) error { - nodeList, err := c.nm.GetNodes() - if err != nil { - return errors.Wrap(err, "Failed to get cluster nodes") - } - if err := c.initNodes(nodeList); err != nil { - return errors.Wrap(err, "Failed to init nodes") - } - return nil -} - -func (c *ServiceController) initTrafficPolicies() { - var ( - cp = &ClusterTrafficPolicy{nm: c.nm} - lp = &LocalTrafficPolicy{logger: c.logger, nm: c.nm, client: c.Client} - blp = &BestEffortLocalPolicy{LocalTrafficPolicy: *lp, ClusterTrafficPolicy: *cp} - ) - c.tps = map[string]TrafficPolicy{ - AnnotationValueClusterTrafficPolicy: cp, - AnnotationValueLocalTrafficPolicy: lp, - AnnotationValueBestEffortLocalTrafficPolicy: blp, - } -} - -func (c *ServiceController) initNodes(nodeList []agent.Node) error { - for _, item := range nodeList { - if err := c.initNode(item.GetIP()); err != nil { - return errors.Wrapf(err, "Failed to init node %s", item.GetIP()) - } - } - return nil -} - -func (c *ServiceController) initNode(nodeIP string) error { - ctxLog := c.logger.WithValues("node", nodeIP) - - node, err := c.nm.GetNode(nodeIP) - if err != nil { - return errors.Wrapf(err, "Failed to get rpc client for node %s", nodeIP) - } - enis, err := node.GetManagedENIs() - if err != nil { - return errors.Wrapf(err, "Failed to query enis from node %s", nodeIP) - } - for _, eni := range enis { - for _, addr := range eni.Ipv4Addresses { - if err := node.SetupNetworkForService(addr.Address, eni); err != nil { - return errors.Wrapf(err, "Failed to init service private ip %s for node %s", addr.Address, nodeIP) - } - c.setFloatingIP(addr.Address, &FloatingIP{ip: addr.Address, subnetID: eni.SubnetId, nodeIP: nodeIP, eni: eni}) - ctxLog.Info("Successfully init service", "private ip", addr.Address) - } - } - return nil -} - -func (c *ServiceController) getFloatingIP(ip string) *FloatingIP { - c.RLock() - defer c.RUnlock() - fip := c.cache[ip] - if fip != nil { - c.logger.Info("Get floating ip from cache", "ip", ip, "eni id", fip.eni.EniId) - } - return fip -} - -func (c *ServiceController) setFloatingIP(ip string, fip *FloatingIP) { - c.Lock() - defer c.Unlock() - c.cache[ip] = fip - c.logger.Info("Put floating ip to cache", "ip", ip, "eni id", fip.eni.EniId) -} - -func (c *ServiceController) removeFloatingIP(ip string) { - c.Lock() - defer c.Unlock() - fip := c.cache[ip] - delete(c.cache, ip) - if fip != nil { - c.logger.Info("Remove floating ip from cache", "ip", ip, "eni id", fip.eni.EniId) - } -} - -func (c *ServiceController) SetupWithManager(mgr ctrl.Manager) error { - return ctrl.NewControllerManagedBy(mgr).WithOptions(controller.Options{ - MaxConcurrentReconciles: config.MaxConcurrentReconciles, - }).For(&corev1.Service{}, builder.WithPredicates(predicate.NewPredicateFuncs(serviceFilterPredicate))).Complete(c) -} - -func (c *ServiceController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { - ctxLog := c.logger.WithValues("service", req.NamespacedName.String()) - ctxLog.Info("Receive service reconcile event") - - svc := &corev1.Service{} - if err := c.Client.Get(ctx, req.NamespacedName, svc); err != nil { - return intctrlutil.CheckedRequeueWithError(err, c.logger, "") - } - - annotations := svc.GetAnnotations() - if annotations[AnnotationKeyLoadBalancerType] != AnnotationValueLoadBalancerTypePrivateIP { - ctxLog.Info("Ignore unrelated service") - return intctrlutil.Reconciled() - } - - if err := c.ensureFloatingIP(ctx, ctxLog, svc); err != nil { - return intctrlutil.CheckedRequeueWithError(err, ctxLog, "") - } - - return intctrlutil.Reconciled() -} - -func (c *ServiceController) ensureFloatingIP(ctx context.Context, ctxLog logr.Logger, svc *corev1.Service) error { - var ( - err error - annotations = svc.GetAnnotations() - fip = c.getFloatingIP(annotations[AnnotationKeyFloatingIP]) - ) - - // service is in deleting - if !svc.GetDeletionTimestamp().IsZero() { - return c.deleteFloatingIP(ctx, ctxLog, fip, svc) - } - - nodeIP, err := c.chooseTrafficNode(svc) - if err != nil { - return errors.Wrap(err, "Failed to get master nodes") - } - - // service is in creating - if fip == nil { - return c.createFloatingIP(ctx, ctxLog, nodeIP, svc) - } - - return c.migrateFloatingIP(ctx, ctxLog, nodeIP, fip, svc) -} - -func (c *ServiceController) migrateFloatingIP(ctx context.Context, ctxLog logr.Logger, nodeIP string, fip *FloatingIP, svc *corev1.Service) error { - ctxLog = ctxLog.WithName("migrateFloatingIP").WithValues("new node", nodeIP, "old node", fip.nodeIP) - - if fip.nodeIP == nodeIP && fip.eni.EniId == svc.GetAnnotations()[AnnotationKeyENIId] { - ctxLog.Info("Floating ip is in sync, do nothing") - return nil - } - - var ( - wg = sync.WaitGroup{} - errs []error - ) - wg.Add(2) - go func() { - defer wg.Done() - errs = append(errs, c.migrateOnNewMaster(ctx, ctxLog, nodeIP, fip, svc)) - }() - go func() { - defer wg.Done() - errs = append(errs, c.migrateOnOldMaster(ctx, ctxLog, fip)) - }() - wg.Wait() - - var messages []string - for _, err := range errs { - if err == nil { - continue - } - messages = append(messages, err.Error()) - } - if len(messages) != 0 { - return errors.New(fmt.Sprintf("Failed to migrate floating ip, err: %s", strings.Join(messages, " | "))) - } - return nil -} - -func (c *ServiceController) migrateOnNewMaster(ctx context.Context, ctxLog logr.Logger, nodeIP string, fip *FloatingIP, svc *corev1.Service) error { - ctxLog = ctxLog.WithName("migrateOnNewMaster").WithValues("node", nodeIP) - - if err := c.cp.DeallocIPAddresses(fip.eni.EniId, []string{fip.ip}); err != nil { - return errors.Wrapf(err, "Failed to dealloc private ip %s", fip.ip) - } - ctxLog.Info("Successfully released floating ip") - - node, err := c.nm.GetNode(nodeIP) - if err != nil { - return errors.Wrap(err, "Failed to get master node") - } - - newENI, err := c.tryAssignPrivateIP(ctxLog, fip.ip, node) - if err != nil { - return errors.Wrap(err, "Failed to assign private ip") - } - newFip := &FloatingIP{ip: fip.ip, subnetID: fip.subnetID, nodeIP: nodeIP, eni: newENI} - c.setFloatingIP(newFip.ip, newFip) - - if err = node.SetupNetworkForService(newFip.ip, newENI); err != nil { - return errors.Wrap(err, "Failed to setup host network stack for service") - } - ctxLog.Info("Successfully setup service network") - - if err = c.updateService(ctx, ctxLog, svc, newFip, false); err != nil { - return errors.Wrap(err, "Failed to update service") - } - return nil -} - -func (c *ServiceController) migrateOnOldMaster(ctx context.Context, ctxLog logr.Logger, fip *FloatingIP) error { - ctxLog = ctxLog.WithName("migrateOnOldMaster").WithValues("node", fip.nodeIP) - - node, err := c.nm.GetNode(fip.nodeIP) - if err != nil { - return errors.Wrap(err, "Failed to get old master node") - } - if err := node.CleanNetworkForService(fip.ip, fip.eni); err != nil { - return errors.Wrap(err, "Failed to cleanup private ip") - } - ctxLog.Info("Successfully clean service network ") - return nil -} - -func (c *ServiceController) createFloatingIP(ctx context.Context, ctxLog logr.Logger, nodeIP string, svc *corev1.Service) error { - ctxLog = ctxLog.WithName("createFloatingIP").WithValues("node", nodeIP) - - node, err := c.nm.GetNode(nodeIP) - if err != nil { - return errors.Wrap(err, "Failed to get master node") - } - - privateIP, eni, err := c.tryAllocPrivateIP(ctxLog, node) - if err != nil { - return errors.Wrap(err, "Failed to alloc new private ip for service") - } - ctxLog.Info("Successfully alloc private ip") - - fip := &FloatingIP{ - ip: privateIP, - subnetID: eni.SubnetId, - nodeIP: nodeIP, - eni: eni, - } - c.setFloatingIP(privateIP, fip) - - if err = node.SetupNetworkForService(fip.ip, fip.eni); err != nil { - return errors.Wrap(err, "Failed to setup host network stack for service") - } - ctxLog.Info("Successfully setup service host network") - - if err = c.updateService(ctx, ctxLog, svc, fip, false); err != nil { - return errors.Wrap(err, "Failed to update service") - } - - return nil -} - -func (c *ServiceController) deleteFloatingIP(ctx context.Context, ctxLog logr.Logger, fip *FloatingIP, svc *corev1.Service) error { - if fip == nil { - fip = c.buildFIPFromAnnotation(svc) - c.logger.Info("Can not find fip in cache, use annotation", "info", fip) - } - ctxLog = ctxLog.WithName("deleteFloatingIP").WithValues("ip", fip.ip, "node", fip.nodeIP) - - annotations := svc.GetAnnotations() - eniID, ok := annotations[AnnotationKeyENIId] - if !ok { - return errors.New("Invalid service, private ip exists but eni id not found") - } - - ctxLog.Info("Deleting service private ip", "eni id", eniID, "ip", fip.ip) - if err := c.cp.DeallocIPAddresses(eniID, []string{fip.ip}); err != nil { - return errors.Wrapf(err, "Failed to dealloc private ip address %s", fip.ip) - } - ctxLog.Info("Successfully released floating ip") - - node, err := c.nm.GetNode(fip.nodeIP) - if err != nil { - return errors.Wrapf(err, "Failed to get rpc client for node %s", fip.nodeIP) - } - if err = node.CleanNetworkForService(fip.ip, fip.eni); err != nil { - return errors.Wrap(err, "Failed to cleanup private ip") - } - ctxLog.Info("Successfully clean service network on node") - - c.removeFloatingIP(fip.ip) - ctxLog.Info("Successfully remove floating ip from cache") - - if err = c.updateService(ctx, ctxLog, svc, fip, true); err != nil { - return errors.Wrap(err, "Failed to update service") - } - ctxLog.Info("Successfully deleted floating ip and cleaned it's host networking") - return nil - -} - -func (c *ServiceController) updateService(ctx context.Context, logger logr.Logger, svc *corev1.Service, fip *FloatingIP, deleting bool) error { - annotations := svc.GetAnnotations() - if annotations == nil { - annotations = make(map[string]string) - } - if fip.eni.EniId == "" { - return errors.New("Invalid eni id") - } - annotations[AnnotationKeyENIId] = fip.eni.EniId - - if fip.nodeIP == "" { - return errors.New("Invalid node ip") - } - annotations[AnnotationKeyENINodeIP] = fip.nodeIP - - if fip.ip == "" { - return errors.New("Invalid floating ip") - } - annotations[AnnotationKeyFloatingIP] = fip.ip - - if fip.subnetID == "" { - return errors.New("Invalid subnet id") - } - annotations[AnnotationKeySubnetID] = fip.subnetID - - svc.SetAnnotations(annotations) - - if deleting { - controllerutil.RemoveFinalizer(svc, FinalizerKey) - } else { - controllerutil.AddFinalizer(svc, FinalizerKey) - } - - svc.Spec.ExternalIPs = []string{fip.ip} - - if err := c.Client.Update(ctx, svc); err != nil { - return err - } - logger.Info("Successfully update service", "info", svc.String()) - return nil -} - -func (c *ServiceController) tryAllocPrivateIP(ctxLog logr.Logger, node agent.Node) (string, *pb.ENIMetadata, error) { - ctxLog = ctxLog.WithName("tryAllocPrivateIP") - - eni, err := node.ChooseENI() - if err != nil { - return "", nil, errors.Wrap(err, "Failed to choose busiest ENI") - } - ctxLog.Info("Successfully choose busiest eni", "eni id", eni.EniId) - - ip, err := c.cp.AllocIPAddresses(eni.EniId) - if err != nil { - return "", nil, errors.Wrapf(err, "Failed to alloc private ip on eni %s", eni.EniId) - } - ctxLog.Info("Successfully alloc private ip", "ip", ip, "eni id", eni.EniId) - - return ip, eni, nil -} - -func (c *ServiceController) tryAssignPrivateIP(ctxLog logr.Logger, ip string, node agent.Node) (*pb.ENIMetadata, error) { - ctxLog = ctxLog.WithName("tryAssignPrivateIP").WithValues("ip", ip) - - eni, err := node.ChooseENI() - if err != nil { - return nil, errors.Wrap(err, "Failed to choose busiest ENI") - } - - if err := c.cp.AssignPrivateIPAddresses(eni.EniId, ip); err != nil { - return nil, errors.Wrapf(err, "Failed to assign private ip %s on eni %s", ip, eni.EniId) - } - ctxLog.Info("Successfully assign private ip") - return eni, nil -} - -func (c *ServiceController) chooseTrafficNode(svc *corev1.Service) (string, error) { - ctxLog := c.logger.WithValues("svc", getObjectFullName(svc)) - var ( - annotations = svc.GetAnnotations() - ) - masterNodeIP := annotations[AnnotationKeyMasterNodeIP] - if masterNodeIP != "" { - return masterNodeIP, nil - } - - trafficPolicy, ok := annotations[AnnotationKeyTrafficPolicy] - if !ok { - trafficPolicy = DefaultTrafficPolicy - } - - policy, ok := c.tps[trafficPolicy] - if !ok { - return "", fmt.Errorf("unknown traffic policy %s", trafficPolicy) - } - ctxLog.Info("Choosing traffic node", "policy", trafficPolicy) - - return policy.ChooseNode(svc) -} - -func (c *ServiceController) buildFIPFromAnnotation(svc *corev1.Service) *FloatingIP { - annotations := svc.GetAnnotations() - result := &FloatingIP{ - ip: annotations[AnnotationKeyFloatingIP], - subnetID: annotations[AnnotationKeySubnetID], - nodeIP: annotations[AnnotationKeyENINodeIP], - eni: &pb.ENIMetadata{ - EniId: annotations[AnnotationKeyENIId], - }, - } - return result -} diff --git a/cmd/loadbalancer/internal/controllers/service_controller_test.go b/cmd/loadbalancer/internal/controllers/service_controller_test.go deleted file mode 100644 index 579cf66a1..000000000 --- a/cmd/loadbalancer/internal/controllers/service_controller_test.go +++ /dev/null @@ -1,351 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - "fmt" - "time" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "k8s.io/apimachinery/pkg/util/intstr" - - "github.com/golang/mock/gomock" - "github.com/sethvargo/go-password/password" - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "sigs.k8s.io/controller-runtime/pkg/client" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" - mockagent "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent/mocks" - mockcloud "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/mocks" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" - "github.com/apecloud/kubeblocks/internal/constant" - intctrlutil "github.com/apecloud/kubeblocks/internal/generics" - testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" -) - -const ( - namespace = "default" - appName = "service-controller-test" - imageName = "nginx" - - node1IP = "172.31.1.2" - eniID1 = "eni-01" - subnetID1 = "subnet-00000001" - subnetID2 = "subnet-00000002" - eniIP11 = "172.31.1.10" - eniIP12 = "172.31.1.11" - - node2IP = "172.31.1.1" - eniID2 = "eni-02" - eniIP21 = "172.31.2.10" - eniIP22 = "172.31.2.11" - svcPort = 12345 - svcTargetPort = 80 -) - -var newSvcObj = func(managed bool, masterIP string) *corev1.Service { - randomStr, _ := password.Generate(6, 0, 0, true, false) - svcName := fmt.Sprintf("%s-%s", appName, randomStr) - svc := &corev1.Service{ - ObjectMeta: metav1.ObjectMeta{ - Namespace: namespace, - Name: svcName, - Labels: map[string]string{ - "app": svcName, - }, - Annotations: map[string]string{}, - }, - Spec: corev1.ServiceSpec{ - Ports: []corev1.ServicePort{ - { - Port: svcPort, - Protocol: corev1.ProtocolTCP, - TargetPort: intstr.FromInt(svcTargetPort), - }, - }, - Selector: map[string]string{ - constant.AppNameLabelKey: appName, - }, - }, - } - if managed { - svc.Annotations[AnnotationKeyLoadBalancerType] = AnnotationValueLoadBalancerTypePrivateIP - } else { - svc.Annotations[AnnotationKeyLoadBalancerType] = AnnotationValueLoadBalancerTypeNone - } - - if masterIP != "" { - svc.Annotations[AnnotationKeyMasterNodeIP] = masterIP - } - return svc -} - -var _ = Describe("ServiceController", Ordered, func() { - - cleanEnv := func() { - // must wait until resources deleted and no longer exist before the testcases start, - // otherwise if later it needs to create some new resource objects with the same name, - // in race conditions, it will find the existence of old objects, resulting failure to - // create the new objects. - By("clean resources") - - // delete rest mocked objects - inNS := client.InNamespace(testCtx.DefaultNamespace) - ml := client.HasLabels{testCtx.TestObjLabelKey} - // namespaced - testapps.ClearResources(&testCtx, intctrlutil.ServiceSignature, inNS, ml) - testapps.ClearResources(&testCtx, intctrlutil.EndpointsSignature, inNS, ml) - testapps.ClearResources(&testCtx, intctrlutil.PodSignature, inNS, ml) - } - - var ( - ctrl *gomock.Controller - mockProvider *mockcloud.MockProvider - mockNodeManager *mockagent.MockNodeManager - node1 agent.Node - node2 agent.Node - ) - - buildENI := func(eniID string, subnetID string, ips []string) *protocol.ENIMetadata { - if len(ips) == 0 { - panic("can build ENI without private IPs") - } - result := protocol.ENIMetadata{ - EniId: eniID, - SubnetId: subnetID, - Ipv4Addresses: []*protocol.IPv4Address{ - { - Primary: true, - Address: ips[0], - }, - }, - } - for _, ip := range ips { - result.Ipv4Addresses = append(result.Ipv4Addresses, &protocol.IPv4Address{Primary: false, Address: ip}) - } - return &result - } - setupController := func() { - ctrl = gomock.NewController(GinkgoT()) - mockProvider = mockcloud.NewMockProvider(ctrl) - mockNodeManager = mockagent.NewMockNodeManager(ctrl) - - setupNode := func(primaryIP string, subnetID string, eni *protocol.ENIMetadata) agent.Node { - mockNode := mockagent.NewMockNode(ctrl) - mockNode.EXPECT().GetIP().Return(primaryIP).AnyTimes() - mockNode.EXPECT().GetNodeInfo().Return(&protocol.InstanceInfo{SubnetId: subnetID}).AnyTimes() - mockNode.EXPECT().SetupNetworkForService(gomock.Any(), gomock.Any()).Return(nil).AnyTimes() - mockNode.EXPECT().CleanNetworkForService(gomock.Any(), gomock.Any()).Return(nil).AnyTimes() - mockNode.EXPECT().ChooseENI().Return(&protocol.ENIMetadata{EniId: eni.EniId, SubnetId: subnetID}, nil).AnyTimes() - mockNode.EXPECT().GetManagedENIs().Return([]*protocol.ENIMetadata{eni}, nil).AnyTimes() - mockNodeManager.EXPECT().GetNode(primaryIP).Return(mockNode, nil).AnyTimes() - return mockNode - } - node1 = setupNode(node1IP, subnetID1, buildENI(eniID1, subnetID1, []string{eniIP11})) - node2 = setupNode(node2IP, subnetID1, buildENI(eniID2, subnetID1, []string{eniIP21, eniIP22})) - serviceController.cp = mockProvider - serviceController.nm = mockNodeManager - serviceController.initTrafficPolicies() - - } - - setupSpareNode := func(ip string) { - var node agent.Node - switch ip { - case node1.GetIP(): - node = node1 - case node2.GetIP(): - node = node2 - default: - panic("unknown node " + ip) - } - mockNodeManager.EXPECT().ChooseSpareNode(gomock.Any()).Return(node, nil).AnyTimes() - } - - BeforeEach(func() { - cleanEnv() - - setupController() - }) - - AfterEach(cleanEnv) - - It("Init nodes", func() { - sc := &ServiceController{ - Client: k8sClient, - logger: logger, - cp: mockProvider, - nm: mockNodeManager, - cache: make(map[string]*FloatingIP), - } - nodes := []agent.Node{node1, node2} - mockNodeManager.EXPECT().GetNodes().Return(nodes, nil).AnyTimes() - Expect(sc.initNodes(nodes)).Should(Succeed()) - }) - - It("Create and migrate service", func() { - setupSpareNode(node1IP) - - var ( - floatingIP = eniIP12 - oldENIId = eniID1 - newENIId = eniID2 - ) - - By("By creating service") - mockProvider.EXPECT().AllocIPAddresses(oldENIId).Return(floatingIP, nil).AnyTimes() - - svc := newSvcObj(true, node1.GetIP()) - svcKey := client.ObjectKey{Namespace: svc.GetNamespace(), Name: svc.GetName()} - Expect(testCtx.CreateObj(context.Background(), svc)).Should(Succeed()) - - Eventually(func() bool { - if err := k8sClient.Get(context.Background(), svcKey, svc); err != nil { - return false - } - return svc.Annotations[AnnotationKeyFloatingIP] == floatingIP - }).Should(BeTrue()) - - By("By migrating service") - mockProvider.EXPECT().DeallocIPAddresses(oldENIId, gomock.Any()).Return(nil).AnyTimes() - mockProvider.EXPECT().AssignPrivateIPAddresses(newENIId, floatingIP).Return(nil).AnyTimes() - - Expect(testapps.ChangeObj(&testCtx, svc, func() { - svc.GetAnnotations()[AnnotationKeyMasterNodeIP] = node2.GetIP() - })).Should(Succeed()) - Eventually(func() bool { - Expect(k8sClient.Get(context.Background(), svcKey, svc)).Should(Succeed()) - return svc.Annotations[AnnotationKeyENIId] == newENIId - }).Should(BeTrue()) - mockProvider.EXPECT().DeallocIPAddresses(newENIId, gomock.Any()).Return(nil).AnyTimes() - }) - - It("Creating service using cluster traffic policy", func() { - setupSpareNode(node1IP) - - eni := &protocol.ENIMetadata{ - EniId: eniID1, - SubnetId: subnetID1, - } - mockProvider.EXPECT().AllocIPAddresses(eni.EniId).Return(eniIP12, nil) - mockProvider.EXPECT().DeallocIPAddresses(eni.EniId, gomock.Any()).Return(nil).AnyTimes() - - svc := newSvcObj(true, "") - svcKey := client.ObjectKey{Namespace: svc.GetNamespace(), Name: svc.GetName()} - svc.GetAnnotations()[AnnotationKeyTrafficPolicy] = AnnotationValueClusterTrafficPolicy - Expect(testCtx.CreateObj(context.Background(), svc)).Should(Succeed()) - - Eventually(func() bool { - if err := k8sClient.Get(context.Background(), svcKey, svc); err != nil { - return false - } - return svc.Annotations[AnnotationKeyFloatingIP] == eniIP12 - }).Should(BeTrue()) - }) - - It("Creating service using local traffic policy", func() { - setupSpareNode(node1IP) - - mockProvider.EXPECT().DeallocIPAddresses(eniID1, gomock.Any()).Return(nil).AnyTimes() - - eni := &protocol.ENIMetadata{ - EniId: eniID1, - SubnetId: subnetID1, - } - mockProvider.EXPECT().AllocIPAddresses(eni.EniId).Return(eniIP12, nil).AnyTimes() - mockProvider.EXPECT().AssignPrivateIPAddresses(gomock.Any(), gomock.Any()).Return(nil).AnyTimes() - - // mock pods must be created before service, as pod.Status.hostIP is used as traffic node with policy BestEffortLocal - pod := testapps.NewPodFactory(namespace, appName). - AddLabels(constant.AppNameLabelKey, appName). - AddContainer(corev1.Container{Name: appName, Image: imageName}). - Create(&testCtx). - GetObject() - podKey := client.ObjectKey{Namespace: namespace, Name: pod.GetName()} - Expect(testapps.ChangeObjStatus(&testCtx, pod, func() { - pod.Status.HostIP = node1IP - })).Should(Succeed()) - - svc := newSvcObj(true, "") - svcKey := client.ObjectKey{Namespace: namespace, Name: svc.GetName()} - svc.GetAnnotations()[AnnotationKeyTrafficPolicy] = AnnotationValueBestEffortLocalTrafficPolicy - Expect(testCtx.CreateObj(context.Background(), svc)).Should(Succeed()) - - Eventually(func() bool { - if err := k8sClient.Get(context.Background(), podKey, pod); err != nil { - return false - } - if err := k8sClient.Get(context.Background(), svcKey, svc); err != nil { - return false - } - return pod.Status.HostIP != "" && svc.GetAnnotations()[AnnotationKeyENINodeIP] == pod.Status.HostIP - }).Should(BeTrue()) - - Expect(testapps.ChangeObj(&testCtx, svc, func() { - svc.GetAnnotations()[AnnotationKeySubnetID] = subnetID2 - })) - Eventually(func() bool { - if err := k8sClient.Get(context.Background(), podKey, pod); err != nil { - return false - } - if err := k8sClient.Get(context.Background(), svcKey, svc); err != nil { - return false - } - return pod.Status.HostIP != "" && svc.GetAnnotations()[AnnotationKeyENINodeIP] == pod.Status.HostIP - }).Should(BeTrue()) - }) - - It("Choose pod", func() { - t := time.Now() - pods := &corev1.PodList{ - Items: []corev1.Pod{ - { - ObjectMeta: metav1.ObjectMeta{ - CreationTimestamp: metav1.NewTime(t.Add(-20 * time.Second)), - }, - Status: corev1.PodStatus{ - HostIP: node2IP, - Phase: corev1.PodRunning, - }, - }, - { - ObjectMeta: metav1.ObjectMeta{ - CreationTimestamp: metav1.NewTime(t.Add(-30 * time.Second)), - }, - Status: corev1.PodStatus{ - HostIP: node1IP, - Phase: corev1.PodRunning, - }, - }, - { - ObjectMeta: metav1.ObjectMeta{ - CreationTimestamp: metav1.NewTime(t.Add(-10 * time.Second)), - }, - Status: corev1.PodStatus{ - HostIP: node1IP, - Phase: corev1.PodPending, - }, - }, - }, - } - pod := LocalTrafficPolicy{}.choosePod(pods) - Expect(pod.Status.HostIP).Should(Equal(node1IP)) - }) -}) diff --git a/cmd/loadbalancer/internal/controllers/suite_test.go b/cmd/loadbalancer/internal/controllers/suite_test.go deleted file mode 100644 index ca198ea0a..000000000 --- a/cmd/loadbalancer/internal/controllers/suite_test.go +++ /dev/null @@ -1,132 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - "testing" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - "github.com/spf13/viper" - "go.uber.org/zap/zapcore" - corev1 "k8s.io/api/core/v1" - "k8s.io/client-go/kubernetes/scheme" - "k8s.io/client-go/rest" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/envtest" - logf "sigs.k8s.io/controller-runtime/pkg/log" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - - "github.com/apecloud/kubeblocks/internal/testutil" -) - -func init() { - viper.AutomaticEnv() -} - -func TestLoadbalancer(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecs(t, "Loadbalancer Suite") -} - -var ( - cfg *rest.Config - k8sClient client.Client - testEnv *envtest.Environment - ctx context.Context - cancel context.CancelFunc - endpointController *EndpointController - serviceController *ServiceController - logger = zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true), func(o *zap.Options) { - o.TimeEncoder = zapcore.ISO8601TimeEncoder - }) - testCtx testutil.TestContext -) - -var _ = BeforeSuite(func() { - if viper.GetBool("ENABLE_DEBUG_LOG") { - logf.SetLogger(logger) - } - - var err error - - err = corev1.AddToScheme(scheme.Scheme) - Expect(err).ShouldNot(HaveOccurred()) - - ctx, cancel = context.WithCancel(context.TODO()) - - By("bootstrapping test environment") - testEnv = &envtest.Environment{} - cfg, err = testEnv.Start() - Expect(err).NotTo(HaveOccurred()) - Expect(cfg).NotTo(BeNil()) - - // +kubebuilder:scaffold:scheme - - k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) - Expect(err).NotTo(HaveOccurred()) - Expect(k8sClient).NotTo(BeNil()) - - // run reconcile - k8sManager, err := ctrl.NewManager(cfg, ctrl.Options{ - Scheme: scheme.Scheme, - MetricsBindAddress: "0", - ClientDisableCacheFor: []client.Object{&corev1.Service{}, &corev1.Endpoints{}, &corev1.Pod{}}, - }) - Expect(err).ToNot(HaveOccurred()) - - endpointController = &EndpointController{ - logger: logger, - Client: k8sManager.GetClient(), - Scheme: k8sManager.GetScheme(), - Recorder: k8sManager.GetEventRecorderFor("LoadBalancer"), - } - err = endpointController.SetupWithManager(k8sManager) - Expect(err).ToNot(HaveOccurred()) - Expect(endpointController).NotTo(BeNil()) - - serviceController = &ServiceController{ - logger: logger, - cache: make(map[string]*FloatingIP), - Client: k8sManager.GetClient(), - Scheme: k8sManager.GetScheme(), - Recorder: k8sManager.GetEventRecorderFor("LoadBalancer"), - } - err = serviceController.SetupWithManager(k8sManager) - Expect(err).ToNot(HaveOccurred()) - Expect(serviceController).NotTo(BeNil()) - - testCtx = testutil.NewDefaultTestContext(ctx, k8sClient, testEnv) - - go func() { - defer GinkgoRecover() - err = k8sManager.Start(ctx) - Expect(err).ToNot(HaveOccurred(), "failed to run controller") - }() - - k8sManager.GetCache().WaitForCacheSync(ctx) -}) - -var _ = AfterSuite(func() { - cancel() - By("tearing down the test environment") - err := testEnv.Stop() - Expect(err).NotTo(HaveOccurred()) -}) diff --git a/cmd/loadbalancer/internal/controllers/traffic_policy.go b/cmd/loadbalancer/internal/controllers/traffic_policy.go deleted file mode 100644 index 63b6d0f0d..000000000 --- a/cmd/loadbalancer/internal/controllers/traffic_policy.go +++ /dev/null @@ -1,141 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package loadbalancer - -import ( - "context" - "fmt" - "sort" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "sigs.k8s.io/controller-runtime/pkg/client" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" -) - -var ErrCrossSubnet = errors.New("Cross subnet") - -type TrafficPolicy interface { - ChooseNode(svc *corev1.Service) (string, error) -} - -type ClusterTrafficPolicy struct { - nm agent.NodeManager -} - -func (c ClusterTrafficPolicy) ChooseNode(svc *corev1.Service) (string, error) { - annotations := svc.GetAnnotations() - nodeIP, ok := annotations[AnnotationKeyENINodeIP] - if ok { - return nodeIP, nil - } - - subnetID := annotations[AnnotationKeySubnetID] - node, err := c.nm.ChooseSpareNode(subnetID) - if err != nil { - return "", errors.Wrap(err, "Failed to choose spare node") - } - return node.GetIP(), nil -} - -type LocalTrafficPolicy struct { - logger logr.Logger - client client.Client - nm agent.NodeManager -} - -func (l LocalTrafficPolicy) ChooseNode(svc *corev1.Service) (string, error) { - ctxLog := l.logger.WithValues("svc", getObjectFullName(svc)) - matchLabels := client.MatchingLabels{} - for k, v := range svc.Spec.Selector { - matchLabels[k] = v - } - listOptions := []client.ListOption{ - matchLabels, - client.InNamespace(svc.GetNamespace()), - } - pods := &corev1.PodList{} - if err := l.client.List(context.Background(), pods, listOptions...); err != nil { - return "", errors.Wrap(err, "Failed to list service related pods") - } - if len(pods.Items) == 0 { - return "", errors.New(fmt.Sprintf("Can not find master node for service %s", getObjectFullName(svc))) - } - ctxLog.Info("Found master pods", "count", len(pods.Items)) - - pod := l.choosePod(pods) - if pod == nil { - return "", errors.New("Can not find valid backend pod") - } - - node, err := l.nm.GetNode(pod.Status.HostIP) - if err != nil { - return "", err - } - - var ( - svcSubnetID = svc.GetAnnotations()[AnnotationKeySubnetID] - nodeSubnetID = node.GetNodeInfo().GetSubnetId() - ) - ctxLog.Info("Choose master pod", "name", getObjectFullName(pod), - "svc subnet id", svcSubnetID, "node subnet id", node.GetNodeInfo().GetSubnetId()) - if svcSubnetID == "" || svcSubnetID == nodeSubnetID { - return pod.Status.HostIP, nil - } - - return "", ErrCrossSubnet -} - -func (l LocalTrafficPolicy) choosePod(pods *corev1.PodList) *corev1.Pod { - // latest created pods have high priority - sort.SliceStable(pods.Items, func(i, j int) bool { - return pods.Items[i].CreationTimestamp.After(pods.Items[j].CreationTimestamp.Time) - }) - for index := range pods.Items { - pod := pods.Items[index] - if pod.Status.Phase == corev1.PodPending || pod.Status.Phase == corev1.PodRunning { - return &pod - } - } - return nil -} - -type BestEffortLocalPolicy struct { - LocalTrafficPolicy - ClusterTrafficPolicy -} - -func (b BestEffortLocalPolicy) ChooseNode(svc *corev1.Service) (string, error) { - result, err := b.LocalTrafficPolicy.ChooseNode(svc) - if err == nil { - return result, nil - } - - if err != ErrCrossSubnet { - return "", errors.Wrapf(err, "Failed to choose node using Local traffic policy") - } - - b.logger.Info("Pod cross subnets, degrade to cluster traffic policy", "svc", getObjectFullName(svc)) - return b.ClusterTrafficPolicy.ChooseNode(svc) -} - -func getObjectFullName(obj metav1.Object) string { - return fmt.Sprintf("%s/%s", obj.GetNamespace(), obj.GetName()) -} diff --git a/cmd/loadbalancer/internal/iptables/iptables.go b/cmd/loadbalancer/internal/iptables/iptables.go deleted file mode 100644 index 2d8e7be58..000000000 --- a/cmd/loadbalancer/internal/iptables/iptables.go +++ /dev/null @@ -1,30 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package iptables - -import ( - "github.com/coreos/go-iptables/iptables" -) - -type iptablesWrapper struct { - *iptables.IPTables -} - -func NewIPTables() (IPTables, error) { - ipt, err := iptables.NewWithProtocol(iptables.ProtocolIPv4) - return iptablesWrapper{ipt}, err -} diff --git a/cmd/loadbalancer/internal/iptables/types.go b/cmd/loadbalancer/internal/iptables/types.go deleted file mode 100644 index 33e75d7a3..000000000 --- a/cmd/loadbalancer/internal/iptables/types.go +++ /dev/null @@ -1,25 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package iptables - -type IPTables interface { - Exists(table, chain string, ruleSpec ...string) (bool, error) - - Append(table, chain string, ruleSpec ...string) error - - Delete(table, chain string, ruleSpec ...string) error -} diff --git a/cmd/loadbalancer/internal/netlink/generate_mocks.go b/cmd/loadbalancer/internal/netlink/generate_mocks.go deleted file mode 100644 index db1038ab3..000000000 --- a/cmd/loadbalancer/internal/netlink/generate_mocks.go +++ /dev/null @@ -1,20 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package netlink - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/netlink_mocks.go . NetLink -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/link_mocks.go github.com/vishvananda/netlink Link diff --git a/cmd/loadbalancer/internal/netlink/mocks/link_mocks.go b/cmd/loadbalancer/internal/netlink/mocks/link_mocks.go deleted file mode 100644 index 64f8b0123..000000000 --- a/cmd/loadbalancer/internal/netlink/mocks/link_mocks.go +++ /dev/null @@ -1,81 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/vishvananda/netlink (interfaces: Link) - -// Package mock_netlink is a generated GoMock package. -package mock_netlink - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - netlink "github.com/vishvananda/netlink" -) - -// MockLink is a mock of Link interface. -type MockLink struct { - ctrl *gomock.Controller - recorder *MockLinkMockRecorder -} - -// MockLinkMockRecorder is the mock recorder for MockLink. -type MockLinkMockRecorder struct { - mock *MockLink -} - -// NewMockLink creates a new mock instance. -func NewMockLink(ctrl *gomock.Controller) *MockLink { - mock := &MockLink{ctrl: ctrl} - mock.recorder = &MockLinkMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockLink) EXPECT() *MockLinkMockRecorder { - return m.recorder -} - -// Attrs mocks base method. -func (m *MockLink) Attrs() *netlink.LinkAttrs { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Attrs") - ret0, _ := ret[0].(*netlink.LinkAttrs) - return ret0 -} - -// Attrs indicates an expected call of Attrs. -func (mr *MockLinkMockRecorder) Attrs() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Attrs", reflect.TypeOf((*MockLink)(nil).Attrs)) -} - -// Type mocks base method. -func (m *MockLink) Type() string { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Type") - ret0, _ := ret[0].(string) - return ret0 -} - -// Type indicates an expected call of Type. -func (mr *MockLinkMockRecorder) Type() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Type", reflect.TypeOf((*MockLink)(nil).Type)) -} diff --git a/cmd/loadbalancer/internal/netlink/mocks/netlink_mocks.go b/cmd/loadbalancer/internal/netlink/mocks/netlink_mocks.go deleted file mode 100644 index b8e124c5b..000000000 --- a/cmd/loadbalancer/internal/netlink/mocks/netlink_mocks.go +++ /dev/null @@ -1,239 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink (interfaces: NetLink) - -// Package mock_netlink is a generated GoMock package. -package mock_netlink - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - netlink "github.com/vishvananda/netlink" -) - -// MockNetLink is a mock of NetLink interface. -type MockNetLink struct { - ctrl *gomock.Controller - recorder *MockNetLinkMockRecorder -} - -// MockNetLinkMockRecorder is the mock recorder for MockNetLink. -type MockNetLinkMockRecorder struct { - mock *MockNetLink -} - -// NewMockNetLink creates a new mock instance. -func NewMockNetLink(ctrl *gomock.Controller) *MockNetLink { - mock := &MockNetLink{ctrl: ctrl} - mock.recorder = &MockNetLinkMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockNetLink) EXPECT() *MockNetLinkMockRecorder { - return m.recorder -} - -// AddrAdd mocks base method. -func (m *MockNetLink) AddrAdd(arg0 netlink.Link, arg1 *netlink.Addr) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AddrAdd", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// AddrAdd indicates an expected call of AddrAdd. -func (mr *MockNetLinkMockRecorder) AddrAdd(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AddrAdd", reflect.TypeOf((*MockNetLink)(nil).AddrAdd), arg0, arg1) -} - -// AddrDel mocks base method. -func (m *MockNetLink) AddrDel(arg0 netlink.Link, arg1 *netlink.Addr) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AddrDel", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// AddrDel indicates an expected call of AddrDel. -func (mr *MockNetLinkMockRecorder) AddrDel(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AddrDel", reflect.TypeOf((*MockNetLink)(nil).AddrDel), arg0, arg1) -} - -// AddrList mocks base method. -func (m *MockNetLink) AddrList(arg0 netlink.Link, arg1 int) ([]netlink.Addr, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "AddrList", arg0, arg1) - ret0, _ := ret[0].([]netlink.Addr) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// AddrList indicates an expected call of AddrList. -func (mr *MockNetLinkMockRecorder) AddrList(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "AddrList", reflect.TypeOf((*MockNetLink)(nil).AddrList), arg0, arg1) -} - -// LinkList mocks base method. -func (m *MockNetLink) LinkList() ([]netlink.Link, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "LinkList") - ret0, _ := ret[0].([]netlink.Link) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// LinkList indicates an expected call of LinkList. -func (mr *MockNetLinkMockRecorder) LinkList() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "LinkList", reflect.TypeOf((*MockNetLink)(nil).LinkList)) -} - -// LinkSetMTU mocks base method. -func (m *MockNetLink) LinkSetMTU(arg0 netlink.Link, arg1 int) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "LinkSetMTU", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// LinkSetMTU indicates an expected call of LinkSetMTU. -func (mr *MockNetLinkMockRecorder) LinkSetMTU(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "LinkSetMTU", reflect.TypeOf((*MockNetLink)(nil).LinkSetMTU), arg0, arg1) -} - -// LinkSetUp mocks base method. -func (m *MockNetLink) LinkSetUp(arg0 netlink.Link) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "LinkSetUp", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// LinkSetUp indicates an expected call of LinkSetUp. -func (mr *MockNetLinkMockRecorder) LinkSetUp(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "LinkSetUp", reflect.TypeOf((*MockNetLink)(nil).LinkSetUp), arg0) -} - -// NewRule mocks base method. -func (m *MockNetLink) NewRule() *netlink.Rule { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "NewRule") - ret0, _ := ret[0].(*netlink.Rule) - return ret0 -} - -// NewRule indicates an expected call of NewRule. -func (mr *MockNetLinkMockRecorder) NewRule() *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NewRule", reflect.TypeOf((*MockNetLink)(nil).NewRule)) -} - -// RouteDel mocks base method. -func (m *MockNetLink) RouteDel(arg0 *netlink.Route) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RouteDel", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// RouteDel indicates an expected call of RouteDel. -func (mr *MockNetLinkMockRecorder) RouteDel(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RouteDel", reflect.TypeOf((*MockNetLink)(nil).RouteDel), arg0) -} - -// RouteList mocks base method. -func (m *MockNetLink) RouteList(arg0 netlink.Link, arg1 int) ([]netlink.Route, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RouteList", arg0, arg1) - ret0, _ := ret[0].([]netlink.Route) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// RouteList indicates an expected call of RouteList. -func (mr *MockNetLinkMockRecorder) RouteList(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RouteList", reflect.TypeOf((*MockNetLink)(nil).RouteList), arg0, arg1) -} - -// RouteReplace mocks base method. -func (m *MockNetLink) RouteReplace(arg0 *netlink.Route) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RouteReplace", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// RouteReplace indicates an expected call of RouteReplace. -func (mr *MockNetLinkMockRecorder) RouteReplace(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RouteReplace", reflect.TypeOf((*MockNetLink)(nil).RouteReplace), arg0) -} - -// RuleAdd mocks base method. -func (m *MockNetLink) RuleAdd(arg0 *netlink.Rule) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RuleAdd", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// RuleAdd indicates an expected call of RuleAdd. -func (mr *MockNetLinkMockRecorder) RuleAdd(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RuleAdd", reflect.TypeOf((*MockNetLink)(nil).RuleAdd), arg0) -} - -// RuleDel mocks base method. -func (m *MockNetLink) RuleDel(arg0 *netlink.Rule) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RuleDel", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// RuleDel indicates an expected call of RuleDel. -func (mr *MockNetLinkMockRecorder) RuleDel(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RuleDel", reflect.TypeOf((*MockNetLink)(nil).RuleDel), arg0) -} - -// RuleList mocks base method. -func (m *MockNetLink) RuleList(arg0 int) ([]netlink.Rule, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "RuleList", arg0) - ret0, _ := ret[0].([]netlink.Rule) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// RuleList indicates an expected call of RuleList. -func (mr *MockNetLinkMockRecorder) RuleList(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "RuleList", reflect.TypeOf((*MockNetLink)(nil).RuleList), arg0) -} diff --git a/cmd/loadbalancer/internal/netlink/netlink.go b/cmd/loadbalancer/internal/netlink/netlink.go deleted file mode 100644 index 8e1396d23..000000000 --- a/cmd/loadbalancer/internal/netlink/netlink.go +++ /dev/null @@ -1,32 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package netlink - -import "github.com/vishvananda/netlink" - -type netLink struct { - *netlink.Handle -} - -func NewNetLink() NetLink { - nl, _ := netlink.NewHandle() - return &netLink{nl} -} - -func (n *netLink) NewRule() *netlink.Rule { - return netlink.NewRule() -} diff --git a/cmd/loadbalancer/internal/netlink/types.go b/cmd/loadbalancer/internal/netlink/types.go deleted file mode 100644 index e6b05879d..000000000 --- a/cmd/loadbalancer/internal/netlink/types.go +++ /dev/null @@ -1,47 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package netlink - -import "github.com/vishvananda/netlink" - -type NetLink interface { - NewRule() *netlink.Rule - - RuleAdd(rule *netlink.Rule) error - - RuleDel(rule *netlink.Rule) error - - RuleList(family int) ([]netlink.Rule, error) - - LinkSetMTU(link netlink.Link, mtu int) error - - AddrAdd(link netlink.Link, addr *netlink.Addr) error - - AddrDel(link netlink.Link, addr *netlink.Addr) error - - AddrList(link netlink.Link, family int) ([]netlink.Addr, error) - - LinkSetUp(link netlink.Link) error - - LinkList() ([]netlink.Link, error) - - RouteReplace(route *netlink.Route) error - - RouteDel(route *netlink.Route) error - - RouteList(link netlink.Link, family int) ([]netlink.Route, error) -} diff --git a/cmd/loadbalancer/internal/network/client.go b/cmd/loadbalancer/internal/network/client.go deleted file mode 100644 index c642e8808..000000000 --- a/cmd/loadbalancer/internal/network/client.go +++ /dev/null @@ -1,39 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import ( - "github.com/go-logr/logr" - - iptableswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/iptables" - netlinkwrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink" - procfswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs" -) - -const ( - LooseReversePathFilterValue = "2" - MainENIMark = 0x2000 - ConnMarkRulePriority = 1000 - ErrAddressNotExists = "cannot assign requested address" -) - -type networkClient struct { - logger logr.Logger - nl netlinkwrapper.NetLink - ipt iptableswrapper.IPTables - procfs procfswrapper.ProcFS -} diff --git a/cmd/loadbalancer/internal/network/client_darwin.go b/cmd/loadbalancer/internal/network/client_darwin.go deleted file mode 100644 index 244ec3d4c..000000000 --- a/cmd/loadbalancer/internal/network/client_darwin.go +++ /dev/null @@ -1,46 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import ( - "github.com/go-logr/logr" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - iptableswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/iptables" - netlinkwrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink" - procfswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs" -) - -func NewClient(logger logr.Logger, nl netlinkwrapper.NetLink, ipt iptableswrapper.IPTables, procfs procfswrapper.ProcFS) (*networkClient, error) { - return &networkClient{logger: logger, ipt: ipt, procfs: procfs, nl: nl}, nil -} - -func (c *networkClient) SetupNetworkForService(privateIP string, eni *cloud.ENIMetadata) error { - return nil -} - -func (c *networkClient) CleanNetworkForService(privateIP string, eni *cloud.ENIMetadata) error { - return nil -} - -func (c *networkClient) SetupNetworkForENI(eni *cloud.ENIMetadata) error { - return nil -} - -func (c *networkClient) CleanNetworkForENI(eni *cloud.ENIMetadata) error { - return nil -} diff --git a/cmd/loadbalancer/internal/network/client_linux.go b/cmd/loadbalancer/internal/network/client_linux.go deleted file mode 100644 index b6a874566..000000000 --- a/cmd/loadbalancer/internal/network/client_linux.go +++ /dev/null @@ -1,497 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import ( - "context" - "encoding/binary" - "fmt" - "net" - "strings" - "syscall" - "time" - - "github.com/go-logr/logr" - "github.com/pkg/errors" - "github.com/vishvananda/netlink" - "golang.org/x/sys/unix" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - iptableswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/iptables" - netlinkwrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink" - procfswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs" - "github.com/apecloud/kubeblocks/internal/cli/util" -) - -type iptablesRule struct { - name string - table string - chain string - rule []string -} - -func NewClient(logger logr.Logger, nl netlinkwrapper.NetLink, ipt iptableswrapper.IPTables, procfs procfswrapper.ProcFS) (*networkClient, error) { - client := &networkClient{ - nl: nl, - ipt: ipt, - logger: logger, - procfs: procfs, - } - return client, nil -} - -func (c *networkClient) SetupNetworkForService(privateIP string, eni *cloud.ENIMetadata) error { - ctxLog := c.logger.WithValues("eni id", eni.ID, "private ip", privateIP) - ctxLog.Info("Configuring policy routing rules and routes") - - link, err := c.getLinkByMac(c.logger, eni.MAC) - if err != nil { - return errors.Wrapf(err, "Failed to get link by mac %s for private ip %s", eni.MAC, privateIP) - } - - privateIPNet := &net.IPNet{IP: net.ParseIP(privateIP), Mask: net.IPv4Mask(255, 255, 255, 255)} - if err := c.nl.AddrAdd(link, &netlink.Addr{IPNet: privateIPNet}); err != nil { - if !strings.Contains(err.Error(), "file exists") { - return errors.Wrapf(err, fmt.Sprintf("Failed to add private ip %s for link %s", privateIP, link.Attrs().Name)) - } - } - ctxLog.Info("Successfully add address to link") - - // add iptables rules - iptablesRules := c.buildServiceIptablesRules(privateIP, eni) - if err := c.updateIptablesRules(iptablesRules, false); err != nil { - return err - } - ctxLog.Info("Successfully setup iptables for service") - - /* - // add policy routing rule - rule := c.buildServicePolicyRoutingRules(privateIPNet, eni) - if err := c.nl.RuleAdd(rule); err != nil && !isRuleExistsError(err) { - return errors.Wrapf(err, "Failed to add service rule, privateIP=%s, rtTable=%v", privateIP, rule.Table) - } - ctxLog.Info("Successfully setup from private ip rule", "route table", rule.Table) - */ - - return nil -} - -func (c *networkClient) CleanNetworkForService(privateIP string, eni *cloud.ENIMetadata) error { - ctxLog := c.logger.WithValues("private ip", privateIP, "eni id", eni.ID) - ctxLog.Info("Remove policy route rules and routes") - - link, err := c.getLinkByMac(c.logger, eni.MAC) - if err != nil { - return errors.Wrapf(err, "Failed to get link by mac %s for private ip %s", eni.MAC, privateIP) - } - - privateIPNet := &net.IPNet{IP: net.ParseIP(privateIP), Mask: net.IPv4Mask(255, 255, 255, 255)} - if err := c.nl.AddrDel(link, &netlink.Addr{IPNet: privateIPNet}); err != nil { - if !strings.Contains(err.Error(), ErrAddressNotExists) { - return errors.Wrapf(err, "Failed to remove addr for service") - } - ctxLog.Info("Address not exists, skip delete", "address", privateIPNet.String()) - } - - // add iptables rules - iptablesRules := c.buildServiceIptablesRules(privateIP, eni) - if err := c.updateIptablesRules(iptablesRules, true); err != nil { - return err - } - ctxLog.Info("Successfully clean iptables for service") - - /* - routingRules := c.buildServicePolicyRoutingRules(privateIPNet, eni) - if err := c.nl.RuleDel(routingRules); err != nil { - if strings.Contains(err.Error(), "no such file or directory") { - c.logger.Info("Policy rule not exists, skip delete", "rule", routingRules.String()) - return nil - } else { - return errors.Wrapf(err, "Failed to remove service rule, privateIP=%s, rtTable=%v", privateIP, routingRules.Table) - } - } - ctxLog.Info("Successfully remove routes", "route table", routingRules.Table) - */ - return nil -} - -func (c *networkClient) SetupNetworkForENI(eni *cloud.ENIMetadata) error { - ctxLog := c.logger.WithValues("eni id", eni.ID) - - if eni.DeviceNumber == 0 { - return fmt.Errorf("can not setup primary eni %s", eni.ID) - } - - if err := c.looseReversePathFilter(eni); err != nil { - return errors.Wrapf(err, "Failed to loose reverse path filter for interface %s", eni.ID) - } - - link, err := c.getLinkByMac(c.logger, eni.MAC) - if err != nil { - return errors.Wrap(err, "Failed to get link by mac") - } - - if err := c.nl.LinkSetUp(link); err != nil { - return errors.Wrap(err, "Failed to set link up") - } - - _, subnetCIDR, err := net.ParseCIDR(eni.SubnetIPv4CIDR) - if err != nil { - return errors.Wrapf(err, "Failed to parse subnet cidr") - } - - gwIP, err := c.getNetworkGateway(subnetCIDR.IP) - if err != nil { - return errors.Wrap(err, "Failed to calculate gateway ip") - } - - expectedIPMap := make(map[string]struct{}, len(eni.IPv4Addresses)) - for _, ip := range eni.IPv4Addresses { - expectedIPMap[ip.Address] = struct{}{} - } - - addrs, err := c.nl.AddrList(link, unix.AF_INET) - if err != nil { - return errors.Wrap(err, "Failed to list ip address for ENI") - } - - // 1. remove unknown private ip, may be added by user - assignedAddrs := make(map[string]struct{}) - for _, addr := range addrs { - if _, ok := expectedIPMap[addr.IP.String()]; ok { - assignedAddrs[addr.IP.String()] = struct{}{} - continue - } - c.logger.Info("Deleting unknown ip address", "ip", addr.String()) - if err = c.nl.AddrDel(link, &addr); err != nil { - if !strings.Contains(err.Error(), ErrAddressNotExists) { - return errors.Wrapf(err, "Failed to delete ip %s from ENI", addr.IP.String()) - } - ctxLog.Info("Address not exists, skip delete", "address", addr.IP.String()) - } - } - - /* - // 2. assign missing private ip - for _, item := range eni.IPv4Addresses { - ip := aws.StringValue(item.PrivateIpAddress) - if _, ok := assignedAddrs[ip]; ok { - continue - } - ipNet := &net.IPNet{IP: net.ParseIP(ip), Mask: subnetCIDR.Mask} - if err = c.nl.AddrAdd(link, &netlink.Addr{IPNet: ipNet}); err != nil { - return errors.Wrapf(err, "Failed to add private ip %s to link", ipNet.String()) - } - } - */ - - // 2. remove the route that default out to ENI out of main route table - defaultRoute := netlink.Route{ - Dst: subnetCIDR, - Src: net.ParseIP(eni.PrimaryIPv4Address()), - Table: unix.RT_TABLE_MAIN, - Scope: netlink.SCOPE_LINK, - } - - if err := c.nl.RouteDel(&defaultRoute); err != nil { - if errno, ok := err.(syscall.Errno); ok && errno != syscall.ESRCH { - return errors.Wrapf(err, "Unable to delete default route %s for source IP %s", subnetCIDR.String(), eni.PrimaryIPv4Address()) - } - } - ctxLog.Info("Successfully deleted default route for eni primary ip", "route", defaultRoute.String()) - - // 3. add route table for eni, and configure routes - var ( - linkIndex = link.Attrs().Index - routeTableNumber = getENIRouteTable(eni) - ) - ctxLog.Info("Setting up eni default gateway", "gateway", gwIP, "route table", routeTableNumber) - routes := []netlink.Route{ - // Add a direct link route for the host's ENI IP only - { - LinkIndex: linkIndex, - Dst: &net.IPNet{IP: gwIP, Mask: net.CIDRMask(32, 32)}, - Scope: netlink.SCOPE_LINK, - Table: routeTableNumber, - }, - // Route all other traffic via the host's ENI IP - { - LinkIndex: linkIndex, - Dst: &net.IPNet{IP: net.IPv4zero, Mask: net.CIDRMask(0, 32)}, - Scope: netlink.SCOPE_UNIVERSE, - Gw: gwIP, - Table: routeTableNumber, - }, - } - for _, r := range routes { - // RouteReplace must do two times for new created enis - for i := 0; i < 2; i++ { - err := util.DoWithRetry(context.Background(), c.logger, func() error { - _ = c.nl.RouteReplace(&r) - return c.nl.RouteReplace(&r) - }, &util.RetryOptions{MaxRetry: 10, Delay: 1 * time.Second}) - - if err != nil { - return errors.Wrapf(err, "Failed to replace route: %s", r.String()) - } - } - ctxLog.Info("Successfully add route", "route", r.String()) - } - - rule := buildENIPolicyRoutingRule(eni) - if err := c.nl.RuleAdd(rule); err != nil && !isRuleExistsError(err) { - return errors.Wrap(err, "Failed to add connmark policy routing rule") - } - ctxLog.Info("Successfully add eni policy rule") - - iptablesRules := c.buildENIIptablesRules(link.Attrs().Name, eni) - if err := c.updateIptablesRules(iptablesRules, false); err != nil { - return err - } - ctxLog.Info("Successfully update iptables connmark rule", "count", len(iptablesRules)) - - routingRules, _ := c.nl.RuleList(netlink.FAMILY_V4) - ctxLog.Info("Found policy routing rules", "count", len(routingRules)) - for _, rule := range routingRules { - ctxLog.Info("Found policy routing rule", "info", rule.String()) - } - - // TODO show routes from eni route table - routes, _ = c.nl.RouteList(nil, netlink.FAMILY_V4) - ctxLog.Info("Found routes", "count", len(routes)) - for _, route := range routes { - ctxLog.Info("Found route", "info", route.String()) - } - return nil -} - -func (c *networkClient) CleanNetworkForENI(eni *cloud.ENIMetadata) error { - if err := c.CleanNetworkForService(eni.PrimaryIPv4Address(), eni); err != nil { - return errors.Wrap(err, "Failed to clean eni primary ip") - } - - rule := buildENIPolicyRoutingRule(eni) - if err := c.nl.RuleDel(rule); err != nil { - if strings.Contains(err.Error(), "no such file or directory") { - c.logger.Info("Policy rule not exists, skip delete", "rule", rule.String()) - } else { - return errors.Wrapf(err, "Failed to remove eni %s policy routing rule", eni.ID) - } - } - - link, err := c.getLinkByMac(c.logger, eni.MAC) - if err != nil { - return errors.Wrap(err, "Failed to get link by mac") - } - iptablesRules := c.buildENIIptablesRules(link.Attrs().Name, eni) - if err := c.updateIptablesRules(iptablesRules, true); err != nil { - return err - } - - c.logger.Info("Successfully clean eni network", "eni id", eni.ID) - return nil -} - -func (c *networkClient) looseReversePathFilter(eni *cloud.ENIMetadata) error { - var ifaceName string - links, err := c.nl.LinkList() - if err != nil { - return errors.Wrap(err, "Failed to list interfaces") - } - for _, link := range links { - if link.Attrs().HardwareAddr.String() == eni.MAC { - ifaceName = link.Attrs().Name - break - } - } - if ifaceName == "" { - return errors.Errorf("Failed to find local network interface with mac %s", eni.MAC) - } - - procKey := fmt.Sprintf("net/ipv4/conf/%s/rp_filter", ifaceName) - src, err := c.procfs.Get(procKey) - if err != nil { - return errors.Wrapf(err, "Failed to read sysctl config %s", procKey) - } - - if err := c.procfs.Set(procKey, LooseReversePathFilterValue); err != nil { - return errors.Wrapf(err, "Failed to update sysctl config %s", procKey) - } - - c.logger.Info("Successfully loose network interface reverse path filter", - "from", src, "to", LooseReversePathFilterValue, "eni id", eni.ID) - return nil -} - -func (c *networkClient) buildServiceIptablesRules(privateIP string, eni *cloud.ENIMetadata) []iptablesRule { - var ( - rules []iptablesRule - mark = getENIConnMark(eni) - ) - - // handle nat-ed traffic which reply packet comes from other host, restore connmark at PREROUTING chain - rules = append(rules, iptablesRule{ - name: "connmark to fwmark copy", - table: "mangle", - chain: "PREROUTING", - rule: []string{ - "-m", "conntrack", "--ctorigdst", privateIP, - "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eni.ID), - "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", mark), - }, - }) - - // handle normal traffic which reply packet comes from local process, restore connmark at OUTPUT chain - rules = append(rules, iptablesRule{ - name: "connmark to fwmark copy", - table: "mangle", - chain: "OUTPUT", - rule: []string{ - "-m", "conntrack", "--ctorigdst", privateIP, - "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eni.ID), - "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", mark), - }, - }) - return rules -} - -func (c *networkClient) updateIptablesRules(iptableRules []iptablesRule, delete bool) error { - for _, rule := range iptableRules { - c.logger.Info("Execute iptable rule", "rule", rule.name) - - exists, err := c.ipt.Exists(rule.table, rule.chain, rule.rule...) - if err != nil { - c.logger.Error(err, "Failed to check existence of iptables rule", "info", rule) - return errors.Wrapf(err, "Failed to check existence of %v", rule) - } - - if !exists && !delete { - err = c.ipt.Append(rule.table, rule.chain, rule.rule...) - if err != nil { - c.logger.Error(err, "Failed to add iptables rule", "info", rule) - return errors.Wrapf(err, "Failed to add %v", rule) - } - } else if exists && delete { - err = c.ipt.Delete(rule.table, rule.chain, rule.rule...) - if err != nil { - c.logger.Error(err, "Failed to delete iptables rule", "info", rule) - return errors.Wrapf(err, "Failed to delete %v", rule) - } - } - } - return nil -} - -func (c *networkClient) buildENIIptablesRules(iface string, eni *cloud.ENIMetadata) []iptablesRule { - var ( - mark = getENIConnMark(eni) - ) - - var rules []iptablesRule - rules = append(rules, iptablesRule{ - name: "connmark rule for non-VPC outbound traffic", - table: "mangle", - chain: "PREROUTING", - rule: []string{ - "-i", iface, "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eni.ID), - "-m", "addrtype", "--dst-type", "LOCAL", "--limit-iface-in", "-j", "CONNMARK", "--set-xmark", fmt.Sprintf("%#x/%#x", mark, mark), - }}) - - /* - rules = append(rules, iptablesRule{ - name: "connmark to fwmark copy", - table: "mangle", - chain: "PREROUTING", - rule: []string{ - "-i", "eni+", "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eni.ID), - "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", mark), - }, - }) - */ - - return rules -} - -// The first four IP addresses and the last IP address in each subnet CIDR block are not available for your use, and they cannot be assigned to a resource, such as an EC2 instance. -// reference: https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-sizing -func (c *networkClient) getNetworkGateway(ip net.IP) (net.IP, error) { - ip4 := ip.To4() - if ip4 == nil { - return nil, fmt.Errorf("%q is not a valid IPv4 Address", ip) - } - intIP := binary.BigEndian.Uint32(ip4) - if intIP == (1<<32 - 1) { - return nil, fmt.Errorf("%q will be overflowed", ip) - } - intIP++ - nextIPv4 := make(net.IP, 4) - binary.BigEndian.PutUint32(nextIPv4, intIP) - return nextIPv4, nil -} - -func (c *networkClient) getLinkByMac(logger logr.Logger, mac string) (netlink.Link, error) { - var result netlink.Link - f := func() error { - links, err := c.nl.LinkList() - if err != nil { - return err - } - - for _, link := range links { - if mac == link.Attrs().HardwareAddr.String() { - logger.Info("Found ethernet link", "mac", mac, "device index", link.Attrs().Index) - result = link - return nil - } - } - return errors.Errorf("Failed to find network interface with mac address %s", mac) - } - - // The adapter might not be immediately available, so we perform retries - retryOpts := &util.RetryOptions{MaxRetry: 10, Delay: 3 * time.Second} - if err := util.DoWithRetry(context.Background(), logger, f, retryOpts); err != nil { - return nil, err - } - return result, nil -} -func isRuleExistsError(err error) bool { - if errno, ok := err.(syscall.Errno); ok { - return errno == syscall.EEXIST - } - return false -} - -func buildENIPolicyRoutingRule(eni *cloud.ENIMetadata) *netlink.Rule { - var ( - mark = getENIConnMark(eni) - ) - rule := netlink.NewRule() - rule.Mark = mark - rule.Mask = mark - rule.Table = getENIRouteTable(eni) - rule.Priority = ConnMarkRulePriority - rule.Family = unix.AF_INET - return rule -} - -func getENIRouteTable(eni *cloud.ENIMetadata) int { - return eni.DeviceNumber + 10000 -} - -func getENIConnMark(eni *cloud.ENIMetadata) int { - return MainENIMark + eni.DeviceNumber -} diff --git a/cmd/loadbalancer/internal/network/client_test.go b/cmd/loadbalancer/internal/network/client_test.go deleted file mode 100644 index d60b94012..000000000 --- a/cmd/loadbalancer/internal/network/client_test.go +++ /dev/null @@ -1,262 +0,0 @@ -//go:build linux - -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import ( - "fmt" - "net" - "reflect" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" - - "github.com/golang/mock/gomock" - "github.com/pkg/errors" - "github.com/spf13/viper" - "github.com/vishvananda/netlink" - "go.uber.org/zap/zapcore" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - mocknetlink "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink/mocks" - mockprocfs "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs/mocks" -) - -func init() { - viper.AutomaticEnv() -} - -var ( - logger = zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true), func(o *zap.Options) { - o.TimeEncoder = zapcore.ISO8601TimeEncoder - }) -) - -var _ = Describe("Client", func() { - - const ( - loMac = "00:00:00:00:00:01" - eth1Mac = "00:00:00:00:00:02" - svcVIP = "172.31.1.10" - eniID = "eni-01" - extraIP = "1.1.1.1" - subnet = "172.31.0.0/16" - ) - - setup := func() (*gomock.Controller, Client, *mocknetlink.MockNetLink, *memoryIptables, *mockprocfs.MockProcFS) { - ctrl := gomock.NewController(GinkgoT()) - - ipt := NewMemoryIptables() - nl := mocknetlink.NewMockNetLink(ctrl) - procfs := mockprocfs.NewMockProcFS(ctrl) - client, err := NewClient(logger, nl, ipt, procfs) - Expect(err == nil).Should(BeTrue()) - return ctrl, client, nl, ipt, procfs - } - - assertIptablesNotExists := func(ipt *memoryIptables, deletedRules map[string]map[string][][]string) { - for table, chains := range deletedRules { - for chain, rules := range chains { - for _, rule := range rules { - Expect(ipt.Exists(table, chain, rule...)).Should(BeFalse()) - } - } - } - } - - Context("Setup and clean service network", func() { - It("Should success without error", func() { - ctrl, networkClient, mockNetlink, mockIPtables, _ := setup() - - lo := mocknetlink.NewMockLink(ctrl) - eth1 := mocknetlink.NewMockLink(ctrl) - mockNetlink.EXPECT().LinkList().Return([]netlink.Link{lo, eth1}, nil).AnyTimes() - - loHwAddr, err := net.ParseMAC(loMac) - Expect(err).Should(BeNil()) - loAttrs := &netlink.LinkAttrs{ - HardwareAddr: loHwAddr, - } - lo.EXPECT().Attrs().Return(loAttrs).AnyTimes() - - eth1HwAddr, err := net.ParseMAC(eth1Mac) - Expect(err).Should(BeNil()) - eth1Attrs := &netlink.LinkAttrs{ - HardwareAddr: eth1HwAddr, - } - eth1.EXPECT().Attrs().Return(eth1Attrs).AnyTimes() - - privateIPNet := &net.IPNet{IP: net.ParseIP(svcVIP), Mask: net.IPv4Mask(255, 255, 255, 255)} - mockNetlink.EXPECT().AddrAdd(eth1, &netlink.Addr{IPNet: privateIPNet}).Return(errors.New("file exists")) - - eni := &cloud.ENIMetadata{ID: eniID, MAC: eth1Mac, DeviceNumber: 1} - Expect(networkClient.SetupNetworkForService(svcVIP, eni)).Should(Succeed()) - - expectIptables := map[string]map[string][][]string{ - "mangle": { - "PREROUTING": [][]string{ - {"-m", "conntrack", "--ctorigdst", svcVIP, "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eniID), "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", getENIConnMark(eni))}, - }, - "OUTPUT": [][]string{ - {"-m", "conntrack", "--ctorigdst", svcVIP, "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eniID), "-j", "CONNMARK", "--restore-mark", "--mask", fmt.Sprintf("%#x", getENIConnMark(eni))}, - }, - }, - } - Expect(reflect.DeepEqual(expectIptables, mockIPtables.rules)).Should(BeTrue()) - - mockNetlink.EXPECT().AddrDel(eth1, &netlink.Addr{IPNet: privateIPNet}).Return(nil) - Expect(networkClient.CleanNetworkForService(svcVIP, eni)).Should(Succeed()) - - assertIptablesNotExists(mockIPtables, expectIptables) - }) - }) - - Context("Setup and clean eni network", func() { - It("Should success without error", func() { - ctrl, networkClient, mockNetlink, mockIPtables, mockProcfs := setup() - - lo := mocknetlink.NewMockLink(ctrl) - eth1 := mocknetlink.NewMockLink(ctrl) - mockNetlink.EXPECT().LinkList().Return([]netlink.Link{lo, eth1}, nil).AnyTimes() - mockNetlink.EXPECT().LinkSetUp(eth1).Return(nil).AnyTimes() - - loHwAddr, err := net.ParseMAC(loMac) - Expect(err).Should(BeNil()) - loAttrs := &netlink.LinkAttrs{ - HardwareAddr: loHwAddr, - } - lo.EXPECT().Attrs().Return(loAttrs).AnyTimes() - - eth1HwAddr, err := net.ParseMAC(eth1Mac) - Expect(err).Should(BeNil()) - eth1Attrs := &netlink.LinkAttrs{ - HardwareAddr: eth1HwAddr, - Name: "eth1", - } - eth1.EXPECT().Attrs().Return(eth1Attrs).AnyTimes() - - addrs := []netlink.Addr{ - { - IPNet: &net.IPNet{ - IP: net.ParseIP(extraIP), - }, - }, - } - eni := &cloud.ENIMetadata{ - ID: eniID, - MAC: eth1Mac, - DeviceNumber: 1, - SubnetIPv4CIDR: subnet, - } - procKey := "net/ipv4/conf/eth1/rp_filter" - mockProcfs.EXPECT().Get(procKey).Return("1", nil) - mockProcfs.EXPECT().Set(procKey, LooseReversePathFilterValue).Return(nil) - mockNetlink.EXPECT().AddrList(eth1, gomock.Any()).Return(addrs, nil) - mockNetlink.EXPECT().AddrDel(gomock.Any(), gomock.Any()).Return(nil).AnyTimes() - mockNetlink.EXPECT().RouteDel(gomock.Any()).Return(nil).AnyTimes() - mockNetlink.EXPECT().RouteReplace(gomock.Any()).Return(nil).AnyTimes() - mockNetlink.EXPECT().RuleAdd(gomock.Any()).Return(nil) - mockNetlink.EXPECT().RuleList(gomock.Any()).Return([]netlink.Rule{}, nil).AnyTimes() - mockNetlink.EXPECT().RouteList(gomock.Any(), gomock.Any()).Return([]netlink.Route{}, nil).AnyTimes() - - Expect(networkClient.SetupNetworkForENI(eni)).Should(Succeed()) - - mark := getENIConnMark(eni) - expectIptables := map[string]map[string][][]string{ - "mangle": { - "PREROUTING": [][]string{ - { - "-i", eth1.Attrs().Name, "-m", "comment", "--comment", fmt.Sprintf("KubeBlocks, %s", eni.ID), - "-m", "addrtype", "--dst-type", "LOCAL", "--limit-iface-in", "-j", "CONNMARK", "--set-xmark", fmt.Sprintf("%#x/%#x", mark, mark), - }, - }, - }, - } - Expect(reflect.DeepEqual(expectIptables, mockIPtables.rules)).Should(BeTrue()) - - mockNetlink.EXPECT().RuleDel(gomock.Any()).Return(nil).AnyTimes() - Expect(networkClient.CleanNetworkForENI(eni)).Should(Succeed()) - assertIptablesNotExists(mockIPtables, expectIptables) - }) - }) -}) - -type memoryIptables struct { - rules map[string]map[string][][]string -} - -func (m *memoryIptables) Exists(table, chain string, ruleSpec ...string) (bool, error) { - if _, ok := m.rules[table]; !ok { - return false, nil - } - rules, ok := m.rules[table][chain] - if !ok { - return false, nil - } - for _, rule := range rules { - if reflect.DeepEqual(ruleSpec, rule) { - return true, nil - } - } - return false, nil -} - -func (m *memoryIptables) Insert(table, chain string, ruleSpec ...string) error { - if _, ok := m.rules[table]; !ok { - m.rules[table] = make(map[string][][]string) - } - m.rules[table][chain] = append([][]string{ruleSpec}, m.rules[table][chain]...) - return nil -} - -func (m *memoryIptables) Append(table, chain string, ruleSpec ...string) error { - if _, ok := m.rules[table]; !ok { - m.rules[table] = make(map[string][][]string) - } - m.rules[table][chain] = append(m.rules[table][chain], ruleSpec) - return nil -} - -func (m *memoryIptables) Delete(table, chain string, ruleSpec ...string) error { - if _, ok := m.rules[table]; !ok { - return errors.Errorf("Can not find table %s", table) - } - rules, ok := m.rules[table][chain] - if !ok { - return errors.Errorf("Can not find chain %s", chain) - } - idx := -1 - for i, rule := range rules { - if reflect.DeepEqual(rule, ruleSpec) { - idx = i - break - } - } - if idx < 0 { - return errors.Errorf("Can not find rule to delete") - } - m.rules[table][chain] = append(rules[:idx], rules[idx+1:]...) - return nil -} - -func NewMemoryIptables() *memoryIptables { - return &memoryIptables{ - rules: make(map[string]map[string][][]string), - } -} diff --git a/cmd/loadbalancer/internal/network/generate_mocks.go b/cmd/loadbalancer/internal/network/generate_mocks.go deleted file mode 100644 index 2874f022b..000000000 --- a/cmd/loadbalancer/internal/network/generate_mocks.go +++ /dev/null @@ -1,19 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/network_mocks.go . Client diff --git a/cmd/loadbalancer/internal/network/mocks/network_mocks.go b/cmd/loadbalancer/internal/network/mocks/network_mocks.go deleted file mode 100644 index aacb6bd25..000000000 --- a/cmd/loadbalancer/internal/network/mocks/network_mocks.go +++ /dev/null @@ -1,110 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/network (interfaces: Client) - -// Package mock_network is a generated GoMock package. -package mock_network - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - - cloud "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" -) - -// MockClient is a mock of Client interface. -type MockClient struct { - ctrl *gomock.Controller - recorder *MockClientMockRecorder -} - -// MockClientMockRecorder is the mock recorder for MockClient. -type MockClientMockRecorder struct { - mock *MockClient -} - -// NewMockClient creates a new mock instance. -func NewMockClient(ctrl *gomock.Controller) *MockClient { - mock := &MockClient{ctrl: ctrl} - mock.recorder = &MockClientMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockClient) EXPECT() *MockClientMockRecorder { - return m.recorder -} - -// CleanNetworkForENI mocks base method. -func (m *MockClient) CleanNetworkForENI(arg0 *cloud.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "CleanNetworkForENI", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// CleanNetworkForENI indicates an expected call of CleanNetworkForENI. -func (mr *MockClientMockRecorder) CleanNetworkForENI(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanNetworkForENI", reflect.TypeOf((*MockClient)(nil).CleanNetworkForENI), arg0) -} - -// CleanNetworkForService mocks base method. -func (m *MockClient) CleanNetworkForService(arg0 string, arg1 *cloud.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "CleanNetworkForService", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// CleanNetworkForService indicates an expected call of CleanNetworkForService. -func (mr *MockClientMockRecorder) CleanNetworkForService(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanNetworkForService", reflect.TypeOf((*MockClient)(nil).CleanNetworkForService), arg0, arg1) -} - -// SetupNetworkForENI mocks base method. -func (m *MockClient) SetupNetworkForENI(arg0 *cloud.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "SetupNetworkForENI", arg0) - ret0, _ := ret[0].(error) - return ret0 -} - -// SetupNetworkForENI indicates an expected call of SetupNetworkForENI. -func (mr *MockClientMockRecorder) SetupNetworkForENI(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetupNetworkForENI", reflect.TypeOf((*MockClient)(nil).SetupNetworkForENI), arg0) -} - -// SetupNetworkForService mocks base method. -func (m *MockClient) SetupNetworkForService(arg0 string, arg1 *cloud.ENIMetadata) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "SetupNetworkForService", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// SetupNetworkForService indicates an expected call of SetupNetworkForService. -func (mr *MockClientMockRecorder) SetupNetworkForService(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetupNetworkForService", reflect.TypeOf((*MockClient)(nil).SetupNetworkForService), arg0, arg1) -} diff --git a/cmd/loadbalancer/internal/network/suite_test.go b/cmd/loadbalancer/internal/network/suite_test.go deleted file mode 100644 index 787c4f372..000000000 --- a/cmd/loadbalancer/internal/network/suite_test.go +++ /dev/null @@ -1,37 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import ( - "testing" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" -) - -func TestNetwork(t *testing.T) { - RegisterFailHandler(Fail) - - RunSpecs(t, "Network Test Suite") -} - -var _ = BeforeSuite(func() { -}) - -var _ = AfterSuite(func() { - -}) diff --git a/cmd/loadbalancer/internal/network/types.go b/cmd/loadbalancer/internal/network/types.go deleted file mode 100644 index fff41a147..000000000 --- a/cmd/loadbalancer/internal/network/types.go +++ /dev/null @@ -1,29 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package network - -import "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - -type Client interface { - SetupNetworkForService(privateIP string, eni *cloud.ENIMetadata) error - - CleanNetworkForService(privateIP string, eni *cloud.ENIMetadata) error - - SetupNetworkForENI(eni *cloud.ENIMetadata) error - - CleanNetworkForENI(eni *cloud.ENIMetadata) error -} diff --git a/cmd/loadbalancer/internal/procfs/generate_mocks.go b/cmd/loadbalancer/internal/procfs/generate_mocks.go deleted file mode 100644 index a72667f6c..000000000 --- a/cmd/loadbalancer/internal/procfs/generate_mocks.go +++ /dev/null @@ -1,19 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package procfs - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/procfs_mocks.go . ProcFS diff --git a/cmd/loadbalancer/internal/procfs/mocks/procfs_mocks.go b/cmd/loadbalancer/internal/procfs/mocks/procfs_mocks.go deleted file mode 100644 index a988245e6..000000000 --- a/cmd/loadbalancer/internal/procfs/mocks/procfs_mocks.go +++ /dev/null @@ -1,81 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs (interfaces: ProcFS) - -// Package mock_procfs is a generated GoMock package. -package mock_procfs - -import ( - reflect "reflect" - - gomock "github.com/golang/mock/gomock" -) - -// MockProcFS is a mock of ProcFS interface. -type MockProcFS struct { - ctrl *gomock.Controller - recorder *MockProcFSMockRecorder -} - -// MockProcFSMockRecorder is the mock recorder for MockProcFS. -type MockProcFSMockRecorder struct { - mock *MockProcFS -} - -// NewMockProcFS creates a new mock instance. -func NewMockProcFS(ctrl *gomock.Controller) *MockProcFS { - mock := &MockProcFS{ctrl: ctrl} - mock.recorder = &MockProcFSMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockProcFS) EXPECT() *MockProcFSMockRecorder { - return m.recorder -} - -// Get mocks base method. -func (m *MockProcFS) Get(arg0 string) (string, error) { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Get", arg0) - ret0, _ := ret[0].(string) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// Get indicates an expected call of Get. -func (mr *MockProcFSMockRecorder) Get(arg0 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Get", reflect.TypeOf((*MockProcFS)(nil).Get), arg0) -} - -// Set mocks base method. -func (m *MockProcFS) Set(arg0, arg1 string) error { - m.ctrl.T.Helper() - ret := m.ctrl.Call(m, "Set", arg0, arg1) - ret0, _ := ret[0].(error) - return ret0 -} - -// Set indicates an expected call of Set. -func (mr *MockProcFSMockRecorder) Set(arg0, arg1 interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Set", reflect.TypeOf((*MockProcFS)(nil).Set), arg0, arg1) -} diff --git a/cmd/loadbalancer/internal/procfs/types.go b/cmd/loadbalancer/internal/procfs/types.go deleted file mode 100644 index d6f050888..000000000 --- a/cmd/loadbalancer/internal/procfs/types.go +++ /dev/null @@ -1,48 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package procfs - -import ( - "os" - "path" -) - -type ProcFS interface { - Set(key, value string) error - - Get(key string) (string, error) -} - -type procFs struct { -} - -func (p *procFs) path(key string) string { - return path.Join("/proc/sys", key) -} - -func (p *procFs) Set(key, value string) error { - return os.WriteFile(p.path(key), []byte(value), 0644) -} - -func (p *procFs) Get(key string) (string, error) { - data, err := os.ReadFile(p.path(key)) - return string(data), err -} - -func NewProcFS() ProcFS { - return &procFs{} -} diff --git a/cmd/loadbalancer/internal/protocol/generate.go b/cmd/loadbalancer/internal/protocol/generate.go deleted file mode 100644 index 42c25f551..000000000 --- a/cmd/loadbalancer/internal/protocol/generate.go +++ /dev/null @@ -1,21 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package protocol - -//go:generate protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative node.proto - -//go:generate go run github.com/golang/mock/mockgen -copyright_file ../../../../hack/boilerplate.go.txt -destination mocks/node_client_mocks.go . NodeClient diff --git a/cmd/loadbalancer/internal/protocol/mocks/node_client_mocks.go b/cmd/loadbalancer/internal/protocol/mocks/node_client_mocks.go deleted file mode 100644 index 464dde4c0..000000000 --- a/cmd/loadbalancer/internal/protocol/mocks/node_client_mocks.go +++ /dev/null @@ -1,176 +0,0 @@ -// /* -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// */ -// -// - -// Code generated by MockGen. DO NOT EDIT. -// Source: github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol (interfaces: NodeClient) - -// Package mock_protocol is a generated GoMock package. -package mock_protocol - -import ( - context "context" - reflect "reflect" - - gomock "github.com/golang/mock/gomock" - grpc "google.golang.org/grpc" - - protocol "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -// MockNodeClient is a mock of NodeClient interface. -type MockNodeClient struct { - ctrl *gomock.Controller - recorder *MockNodeClientMockRecorder -} - -// MockNodeClientMockRecorder is the mock recorder for MockNodeClient. -type MockNodeClientMockRecorder struct { - mock *MockNodeClient -} - -// NewMockNodeClient creates a new mock instance. -func NewMockNodeClient(ctrl *gomock.Controller) *MockNodeClient { - mock := &MockNodeClient{ctrl: ctrl} - mock.recorder = &MockNodeClientMockRecorder{mock} - return mock -} - -// EXPECT returns an object that allows the caller to indicate expected use. -func (m *MockNodeClient) EXPECT() *MockNodeClientMockRecorder { - return m.recorder -} - -// CleanNetworkForENI mocks base method. -func (m *MockNodeClient) CleanNetworkForENI(arg0 context.Context, arg1 *protocol.CleanNetworkForENIRequest, arg2 ...grpc.CallOption) (*protocol.CleanNetworkForENIResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "CleanNetworkForENI", varargs...) - ret0, _ := ret[0].(*protocol.CleanNetworkForENIResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// CleanNetworkForENI indicates an expected call of CleanNetworkForENI. -func (mr *MockNodeClientMockRecorder) CleanNetworkForENI(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanNetworkForENI", reflect.TypeOf((*MockNodeClient)(nil).CleanNetworkForENI), varargs...) -} - -// CleanNetworkForService mocks base method. -func (m *MockNodeClient) CleanNetworkForService(arg0 context.Context, arg1 *protocol.CleanNetworkForServiceRequest, arg2 ...grpc.CallOption) (*protocol.CleanNetworkForServiceResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "CleanNetworkForService", varargs...) - ret0, _ := ret[0].(*protocol.CleanNetworkForServiceResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// CleanNetworkForService indicates an expected call of CleanNetworkForService. -func (mr *MockNodeClientMockRecorder) CleanNetworkForService(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "CleanNetworkForService", reflect.TypeOf((*MockNodeClient)(nil).CleanNetworkForService), varargs...) -} - -// DescribeAllENIs mocks base method. -func (m *MockNodeClient) DescribeAllENIs(arg0 context.Context, arg1 *protocol.DescribeAllENIsRequest, arg2 ...grpc.CallOption) (*protocol.DescribeAllENIsResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeAllENIs", varargs...) - ret0, _ := ret[0].(*protocol.DescribeAllENIsResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeAllENIs indicates an expected call of DescribeAllENIs. -func (mr *MockNodeClientMockRecorder) DescribeAllENIs(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeAllENIs", reflect.TypeOf((*MockNodeClient)(nil).DescribeAllENIs), varargs...) -} - -// DescribeNodeInfo mocks base method. -func (m *MockNodeClient) DescribeNodeInfo(arg0 context.Context, arg1 *protocol.DescribeNodeInfoRequest, arg2 ...grpc.CallOption) (*protocol.DescribeNodeInfoResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "DescribeNodeInfo", varargs...) - ret0, _ := ret[0].(*protocol.DescribeNodeInfoResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// DescribeNodeInfo indicates an expected call of DescribeNodeInfo. -func (mr *MockNodeClientMockRecorder) DescribeNodeInfo(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "DescribeNodeInfo", reflect.TypeOf((*MockNodeClient)(nil).DescribeNodeInfo), varargs...) -} - -// SetupNetworkForENI mocks base method. -func (m *MockNodeClient) SetupNetworkForENI(arg0 context.Context, arg1 *protocol.SetupNetworkForENIRequest, arg2 ...grpc.CallOption) (*protocol.SetupNetworkForENIResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "SetupNetworkForENI", varargs...) - ret0, _ := ret[0].(*protocol.SetupNetworkForENIResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// SetupNetworkForENI indicates an expected call of SetupNetworkForENI. -func (mr *MockNodeClientMockRecorder) SetupNetworkForENI(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetupNetworkForENI", reflect.TypeOf((*MockNodeClient)(nil).SetupNetworkForENI), varargs...) -} - -// SetupNetworkForService mocks base method. -func (m *MockNodeClient) SetupNetworkForService(arg0 context.Context, arg1 *protocol.SetupNetworkForServiceRequest, arg2 ...grpc.CallOption) (*protocol.SetupNetworkForServiceResponse, error) { - m.ctrl.T.Helper() - varargs := []interface{}{arg0, arg1} - for _, a := range arg2 { - varargs = append(varargs, a) - } - ret := m.ctrl.Call(m, "SetupNetworkForService", varargs...) - ret0, _ := ret[0].(*protocol.SetupNetworkForServiceResponse) - ret1, _ := ret[1].(error) - return ret0, ret1 -} - -// SetupNetworkForService indicates an expected call of SetupNetworkForService. -func (mr *MockNodeClientMockRecorder) SetupNetworkForService(arg0, arg1 interface{}, arg2 ...interface{}) *gomock.Call { - mr.mock.ctrl.T.Helper() - varargs := append([]interface{}{arg0, arg1}, arg2...) - return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "SetupNetworkForService", reflect.TypeOf((*MockNodeClient)(nil).SetupNetworkForService), varargs...) -} diff --git a/cmd/loadbalancer/internal/protocol/node.pb.go b/cmd/loadbalancer/internal/protocol/node.pb.go deleted file mode 100644 index 82bb2d9de..000000000 --- a/cmd/loadbalancer/internal/protocol/node.pb.go +++ /dev/null @@ -1,1352 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// versions: -// protoc-gen-go v1.26.0-rc.1 -// protoc v3.21.12 -// source: node.proto - -package protocol - -import ( - reflect "reflect" - sync "sync" - - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" -) - -const ( - // Verify that this generated code is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) - // Verify that runtime/protoimpl is sufficiently up-to-date. - _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) -) - -type DescribeNodeInfoRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` -} - -func (x *DescribeNodeInfoRequest) Reset() { - *x = DescribeNodeInfoRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[0] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *DescribeNodeInfoRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DescribeNodeInfoRequest) ProtoMessage() {} - -func (x *DescribeNodeInfoRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[0] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DescribeNodeInfoRequest.ProtoReflect.Descriptor instead. -func (*DescribeNodeInfoRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{0} -} - -func (x *DescribeNodeInfoRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -type DescribeNodeInfoResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Info *InstanceInfo `protobuf:"bytes,2,opt,name=info,proto3" json:"info,omitempty"` -} - -func (x *DescribeNodeInfoResponse) Reset() { - *x = DescribeNodeInfoResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[1] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *DescribeNodeInfoResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DescribeNodeInfoResponse) ProtoMessage() {} - -func (x *DescribeNodeInfoResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[1] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DescribeNodeInfoResponse.ProtoReflect.Descriptor instead. -func (*DescribeNodeInfoResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{1} -} - -func (x *DescribeNodeInfoResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *DescribeNodeInfoResponse) GetInfo() *InstanceInfo { - if x != nil { - return x.Info - } - return nil -} - -type DescribeAllENIsRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` -} - -func (x *DescribeAllENIsRequest) Reset() { - *x = DescribeAllENIsRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[2] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *DescribeAllENIsRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DescribeAllENIsRequest) ProtoMessage() {} - -func (x *DescribeAllENIsRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[2] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DescribeAllENIsRequest.ProtoReflect.Descriptor instead. -func (*DescribeAllENIsRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{2} -} - -func (x *DescribeAllENIsRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -type DescribeAllENIsResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Enis map[string]*ENIMetadata `protobuf:"bytes,2,rep,name=enis,proto3" json:"enis,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` -} - -func (x *DescribeAllENIsResponse) Reset() { - *x = DescribeAllENIsResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[3] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *DescribeAllENIsResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*DescribeAllENIsResponse) ProtoMessage() {} - -func (x *DescribeAllENIsResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[3] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use DescribeAllENIsResponse.ProtoReflect.Descriptor instead. -func (*DescribeAllENIsResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{3} -} - -func (x *DescribeAllENIsResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *DescribeAllENIsResponse) GetEnis() map[string]*ENIMetadata { - if x != nil { - return x.Enis - } - return nil -} - -type SetupNetworkForServiceRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - PrivateIp string `protobuf:"bytes,2,opt,name=private_ip,json=privateIp,proto3" json:"private_ip,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,3,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *SetupNetworkForServiceRequest) Reset() { - *x = SetupNetworkForServiceRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[4] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *SetupNetworkForServiceRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetupNetworkForServiceRequest) ProtoMessage() {} - -func (x *SetupNetworkForServiceRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[4] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetupNetworkForServiceRequest.ProtoReflect.Descriptor instead. -func (*SetupNetworkForServiceRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{4} -} - -func (x *SetupNetworkForServiceRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *SetupNetworkForServiceRequest) GetPrivateIp() string { - if x != nil { - return x.PrivateIp - } - return "" -} - -func (x *SetupNetworkForServiceRequest) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type SetupNetworkForServiceResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - PrivateIp string `protobuf:"bytes,2,opt,name=private_ip,json=privateIp,proto3" json:"private_ip,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,3,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *SetupNetworkForServiceResponse) Reset() { - *x = SetupNetworkForServiceResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[5] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *SetupNetworkForServiceResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetupNetworkForServiceResponse) ProtoMessage() {} - -func (x *SetupNetworkForServiceResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[5] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetupNetworkForServiceResponse.ProtoReflect.Descriptor instead. -func (*SetupNetworkForServiceResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{5} -} - -func (x *SetupNetworkForServiceResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *SetupNetworkForServiceResponse) GetPrivateIp() string { - if x != nil { - return x.PrivateIp - } - return "" -} - -func (x *SetupNetworkForServiceResponse) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type CleanNetworkForServiceRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - PrivateIp string `protobuf:"bytes,2,opt,name=private_ip,json=privateIp,proto3" json:"private_ip,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,3,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *CleanNetworkForServiceRequest) Reset() { - *x = CleanNetworkForServiceRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[6] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *CleanNetworkForServiceRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*CleanNetworkForServiceRequest) ProtoMessage() {} - -func (x *CleanNetworkForServiceRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[6] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use CleanNetworkForServiceRequest.ProtoReflect.Descriptor instead. -func (*CleanNetworkForServiceRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{6} -} - -func (x *CleanNetworkForServiceRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *CleanNetworkForServiceRequest) GetPrivateIp() string { - if x != nil { - return x.PrivateIp - } - return "" -} - -func (x *CleanNetworkForServiceRequest) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type CleanNetworkForServiceResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - PrivateIp string `protobuf:"bytes,2,opt,name=private_ip,json=privateIp,proto3" json:"private_ip,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,3,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *CleanNetworkForServiceResponse) Reset() { - *x = CleanNetworkForServiceResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[7] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *CleanNetworkForServiceResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*CleanNetworkForServiceResponse) ProtoMessage() {} - -func (x *CleanNetworkForServiceResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[7] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use CleanNetworkForServiceResponse.ProtoReflect.Descriptor instead. -func (*CleanNetworkForServiceResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{7} -} - -func (x *CleanNetworkForServiceResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *CleanNetworkForServiceResponse) GetPrivateIp() string { - if x != nil { - return x.PrivateIp - } - return "" -} - -func (x *CleanNetworkForServiceResponse) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type SetupNetworkForENIRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,2,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *SetupNetworkForENIRequest) Reset() { - *x = SetupNetworkForENIRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[8] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *SetupNetworkForENIRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetupNetworkForENIRequest) ProtoMessage() {} - -func (x *SetupNetworkForENIRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[8] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetupNetworkForENIRequest.ProtoReflect.Descriptor instead. -func (*SetupNetworkForENIRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{8} -} - -func (x *SetupNetworkForENIRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *SetupNetworkForENIRequest) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type SetupNetworkForENIResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,2,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *SetupNetworkForENIResponse) Reset() { - *x = SetupNetworkForENIResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[9] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *SetupNetworkForENIResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*SetupNetworkForENIResponse) ProtoMessage() {} - -func (x *SetupNetworkForENIResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[9] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use SetupNetworkForENIResponse.ProtoReflect.Descriptor instead. -func (*SetupNetworkForENIResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{9} -} - -func (x *SetupNetworkForENIResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *SetupNetworkForENIResponse) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type CleanNetworkForENIRequest struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,2,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *CleanNetworkForENIRequest) Reset() { - *x = CleanNetworkForENIRequest{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[10] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *CleanNetworkForENIRequest) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*CleanNetworkForENIRequest) ProtoMessage() {} - -func (x *CleanNetworkForENIRequest) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[10] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use CleanNetworkForENIRequest.ProtoReflect.Descriptor instead. -func (*CleanNetworkForENIRequest) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{10} -} - -func (x *CleanNetworkForENIRequest) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *CleanNetworkForENIRequest) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type CleanNetworkForENIResponse struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - RequestId string `protobuf:"bytes,1,opt,name=request_id,json=requestId,proto3" json:"request_id,omitempty"` - Eni *ENIMetadata `protobuf:"bytes,2,opt,name=eni,proto3" json:"eni,omitempty"` -} - -func (x *CleanNetworkForENIResponse) Reset() { - *x = CleanNetworkForENIResponse{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[11] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *CleanNetworkForENIResponse) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*CleanNetworkForENIResponse) ProtoMessage() {} - -func (x *CleanNetworkForENIResponse) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[11] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use CleanNetworkForENIResponse.ProtoReflect.Descriptor instead. -func (*CleanNetworkForENIResponse) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{11} -} - -func (x *CleanNetworkForENIResponse) GetRequestId() string { - if x != nil { - return x.RequestId - } - return "" -} - -func (x *CleanNetworkForENIResponse) GetEni() *ENIMetadata { - if x != nil { - return x.Eni - } - return nil -} - -type InstanceInfo struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - InstanceId string `protobuf:"bytes,1,opt,name=instance_id,json=instanceId,proto3" json:"instance_id,omitempty"` - SubnetId string `protobuf:"bytes,2,opt,name=subnet_id,json=subnetId,proto3" json:"subnet_id,omitempty"` - SecurityGroupIds []string `protobuf:"bytes,3,rep,name=security_group_ids,json=securityGroupIds,proto3" json:"security_group_ids,omitempty"` -} - -func (x *InstanceInfo) Reset() { - *x = InstanceInfo{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[12] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *InstanceInfo) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*InstanceInfo) ProtoMessage() {} - -func (x *InstanceInfo) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[12] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use InstanceInfo.ProtoReflect.Descriptor instead. -func (*InstanceInfo) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{12} -} - -func (x *InstanceInfo) GetInstanceId() string { - if x != nil { - return x.InstanceId - } - return "" -} - -func (x *InstanceInfo) GetSubnetId() string { - if x != nil { - return x.SubnetId - } - return "" -} - -func (x *InstanceInfo) GetSecurityGroupIds() []string { - if x != nil { - return x.SecurityGroupIds - } - return nil -} - -type ENIMetadata struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - EniId string `protobuf:"bytes,1,opt,name=eni_id,json=eniId,proto3" json:"eni_id,omitempty"` - Mac string `protobuf:"bytes,2,opt,name=mac,proto3" json:"mac,omitempty"` - DeviceNumber int32 `protobuf:"varint,3,opt,name=device_number,json=deviceNumber,proto3" json:"device_number,omitempty"` - SubnetId string `protobuf:"bytes,4,opt,name=subnet_id,json=subnetId,proto3" json:"subnet_id,omitempty"` - SubnetIpv4Cidr string `protobuf:"bytes,5,opt,name=subnet_ipv4_cidr,json=subnetIpv4Cidr,proto3" json:"subnet_ipv4_cidr,omitempty"` - Ipv4Addresses []*IPv4Address `protobuf:"bytes,6,rep,name=ipv4_addresses,json=ipv4Addresses,proto3" json:"ipv4_addresses,omitempty"` - Tags map[string]string `protobuf:"bytes,7,rep,name=tags,proto3" json:"tags,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` -} - -func (x *ENIMetadata) Reset() { - *x = ENIMetadata{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[13] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *ENIMetadata) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*ENIMetadata) ProtoMessage() {} - -func (x *ENIMetadata) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[13] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use ENIMetadata.ProtoReflect.Descriptor instead. -func (*ENIMetadata) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{13} -} - -func (x *ENIMetadata) GetEniId() string { - if x != nil { - return x.EniId - } - return "" -} - -func (x *ENIMetadata) GetMac() string { - if x != nil { - return x.Mac - } - return "" -} - -func (x *ENIMetadata) GetDeviceNumber() int32 { - if x != nil { - return x.DeviceNumber - } - return 0 -} - -func (x *ENIMetadata) GetSubnetId() string { - if x != nil { - return x.SubnetId - } - return "" -} - -func (x *ENIMetadata) GetSubnetIpv4Cidr() string { - if x != nil { - return x.SubnetIpv4Cidr - } - return "" -} - -func (x *ENIMetadata) GetIpv4Addresses() []*IPv4Address { - if x != nil { - return x.Ipv4Addresses - } - return nil -} - -func (x *ENIMetadata) GetTags() map[string]string { - if x != nil { - return x.Tags - } - return nil -} - -type IPv4Address struct { - state protoimpl.MessageState - sizeCache protoimpl.SizeCache - unknownFields protoimpl.UnknownFields - - Primary bool `protobuf:"varint,1,opt,name=primary,proto3" json:"primary,omitempty"` - Address string `protobuf:"bytes,2,opt,name=address,proto3" json:"address,omitempty"` -} - -func (x *IPv4Address) Reset() { - *x = IPv4Address{} - if protoimpl.UnsafeEnabled { - mi := &file_node_proto_msgTypes[14] - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - ms.StoreMessageInfo(mi) - } -} - -func (x *IPv4Address) String() string { - return protoimpl.X.MessageStringOf(x) -} - -func (*IPv4Address) ProtoMessage() {} - -func (x *IPv4Address) ProtoReflect() protoreflect.Message { - mi := &file_node_proto_msgTypes[14] - if protoimpl.UnsafeEnabled && x != nil { - ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) - if ms.LoadMessageInfo() == nil { - ms.StoreMessageInfo(mi) - } - return ms - } - return mi.MessageOf(x) -} - -// Deprecated: Use IPv4Address.ProtoReflect.Descriptor instead. -func (*IPv4Address) Descriptor() ([]byte, []int) { - return file_node_proto_rawDescGZIP(), []int{14} -} - -func (x *IPv4Address) GetPrimary() bool { - if x != nil { - return x.Primary - } - return false -} - -func (x *IPv4Address) GetAddress() string { - if x != nil { - return x.Address - } - return "" -} - -var File_node_proto protoreflect.FileDescriptor - -var file_node_proto_rawDesc = []byte{ - 0x0a, 0x0a, 0x6e, 0x6f, 0x64, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x08, 0x70, 0x72, - 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x22, 0x38, 0x0a, 0x17, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, - 0x62, 0x65, 0x4e, 0x6f, 0x64, 0x65, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, - 0x22, 0x65, 0x0a, 0x18, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x4e, 0x6f, 0x64, 0x65, - 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1d, 0x0a, 0x0a, - 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x2a, 0x0a, 0x04, 0x69, - 0x6e, 0x66, 0x6f, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x49, 0x6e, 0x73, 0x74, 0x61, 0x6e, 0x63, 0x65, 0x49, 0x6e, 0x66, - 0x6f, 0x52, 0x04, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x37, 0x0a, 0x16, 0x44, 0x65, 0x73, 0x63, 0x72, - 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, 0x45, 0x4e, 0x49, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, - 0x22, 0xc9, 0x01, 0x0a, 0x17, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, - 0x45, 0x4e, 0x49, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1d, 0x0a, 0x0a, - 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x3f, 0x0a, 0x04, 0x65, - 0x6e, 0x69, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x2b, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, - 0x45, 0x4e, 0x49, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x45, 0x6e, 0x69, - 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x04, 0x65, 0x6e, 0x69, 0x73, 0x1a, 0x4e, 0x0a, 0x09, - 0x45, 0x6e, 0x69, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, - 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x2b, 0x0a, 0x05, 0x76, - 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, - 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, - 0x61, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x86, 0x01, 0x0a, - 0x1d, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, - 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1d, - 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x1d, 0x0a, - 0x0a, 0x70, 0x72, 0x69, 0x76, 0x61, 0x74, 0x65, 0x5f, 0x69, 0x70, 0x18, 0x02, 0x20, 0x01, 0x28, - 0x09, 0x52, 0x09, 0x70, 0x72, 0x69, 0x76, 0x61, 0x74, 0x65, 0x49, 0x70, 0x12, 0x27, 0x0a, 0x03, - 0x65, 0x6e, 0x69, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, - 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, 0x87, 0x01, 0x0a, 0x1e, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, - 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x1d, 0x0a, 0x0a, 0x70, 0x72, 0x69, 0x76, 0x61, - 0x74, 0x65, 0x5f, 0x69, 0x70, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x69, - 0x76, 0x61, 0x74, 0x65, 0x49, 0x70, 0x12, 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, 0x18, 0x03, 0x20, - 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, - 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, - 0x86, 0x01, 0x0a, 0x1d, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, - 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, - 0x12, 0x1d, 0x0a, 0x0a, 0x70, 0x72, 0x69, 0x76, 0x61, 0x74, 0x65, 0x5f, 0x69, 0x70, 0x18, 0x02, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x69, 0x76, 0x61, 0x74, 0x65, 0x49, 0x70, 0x12, - 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, - 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, 0x87, 0x01, 0x0a, 0x1e, 0x43, 0x6c, 0x65, - 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, - 0x69, 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x72, - 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x1d, 0x0a, 0x0a, 0x70, 0x72, - 0x69, 0x76, 0x61, 0x74, 0x65, 0x5f, 0x69, 0x70, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, - 0x70, 0x72, 0x69, 0x76, 0x61, 0x74, 0x65, 0x49, 0x70, 0x12, 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, - 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, - 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, - 0x6e, 0x69, 0x22, 0x63, 0x0a, 0x19, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, - 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, - 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x27, - 0x0a, 0x03, 0x65, 0x6e, 0x69, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, - 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, - 0x74, 0x61, 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, 0x64, 0x0a, 0x1a, 0x53, 0x65, 0x74, 0x75, 0x70, - 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x52, 0x65, 0x73, - 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, - 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, 0x18, 0x02, 0x20, 0x01, 0x28, - 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, - 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, 0x63, 0x0a, - 0x19, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, - 0x45, 0x4e, 0x49, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, - 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, - 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, - 0x6e, 0x69, 0x22, 0x64, 0x0a, 0x1a, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, - 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, - 0x12, 0x1d, 0x0a, 0x0a, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x72, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x49, 0x64, 0x12, - 0x27, 0x0a, 0x03, 0x65, 0x6e, 0x69, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, - 0x61, 0x74, 0x61, 0x52, 0x03, 0x65, 0x6e, 0x69, 0x22, 0x7a, 0x0a, 0x0c, 0x49, 0x6e, 0x73, 0x74, - 0x61, 0x6e, 0x63, 0x65, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x1f, 0x0a, 0x0b, 0x69, 0x6e, 0x73, 0x74, - 0x61, 0x6e, 0x63, 0x65, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, 0x69, - 0x6e, 0x73, 0x74, 0x61, 0x6e, 0x63, 0x65, 0x49, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x75, 0x62, - 0x6e, 0x65, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x75, - 0x62, 0x6e, 0x65, 0x74, 0x49, 0x64, 0x12, 0x2c, 0x0a, 0x12, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, - 0x74, 0x79, 0x5f, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x69, 0x64, 0x73, 0x18, 0x03, 0x20, 0x03, - 0x28, 0x09, 0x52, 0x10, 0x73, 0x65, 0x63, 0x75, 0x72, 0x69, 0x74, 0x79, 0x47, 0x72, 0x6f, 0x75, - 0x70, 0x49, 0x64, 0x73, 0x22, 0xce, 0x02, 0x0a, 0x0b, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, - 0x64, 0x61, 0x74, 0x61, 0x12, 0x15, 0x0a, 0x06, 0x65, 0x6e, 0x69, 0x5f, 0x69, 0x64, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x6e, 0x69, 0x49, 0x64, 0x12, 0x10, 0x0a, 0x03, 0x6d, - 0x61, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6d, 0x61, 0x63, 0x12, 0x23, 0x0a, - 0x0d, 0x64, 0x65, 0x76, 0x69, 0x63, 0x65, 0x5f, 0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x03, - 0x20, 0x01, 0x28, 0x05, 0x52, 0x0c, 0x64, 0x65, 0x76, 0x69, 0x63, 0x65, 0x4e, 0x75, 0x6d, 0x62, - 0x65, 0x72, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x75, 0x62, 0x6e, 0x65, 0x74, 0x5f, 0x69, 0x64, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x75, 0x62, 0x6e, 0x65, 0x74, 0x49, 0x64, 0x12, - 0x28, 0x0a, 0x10, 0x73, 0x75, 0x62, 0x6e, 0x65, 0x74, 0x5f, 0x69, 0x70, 0x76, 0x34, 0x5f, 0x63, - 0x69, 0x64, 0x72, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x73, 0x75, 0x62, 0x6e, 0x65, - 0x74, 0x49, 0x70, 0x76, 0x34, 0x43, 0x69, 0x64, 0x72, 0x12, 0x3c, 0x0a, 0x0e, 0x69, 0x70, 0x76, - 0x34, 0x5f, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x18, 0x06, 0x20, 0x03, 0x28, - 0x0b, 0x32, 0x15, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x49, 0x50, 0x76, - 0x34, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x52, 0x0d, 0x69, 0x70, 0x76, 0x34, 0x41, 0x64, - 0x64, 0x72, 0x65, 0x73, 0x73, 0x65, 0x73, 0x12, 0x33, 0x0a, 0x04, 0x74, 0x61, 0x67, 0x73, 0x18, - 0x07, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, - 0x2e, 0x45, 0x4e, 0x49, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x54, 0x61, 0x67, - 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x04, 0x74, 0x61, 0x67, 0x73, 0x1a, 0x37, 0x0a, 0x09, - 0x54, 0x61, 0x67, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, - 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, - 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, - 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x41, 0x0a, 0x0b, 0x49, 0x50, 0x76, 0x34, 0x41, 0x64, 0x64, - 0x72, 0x65, 0x73, 0x73, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x69, 0x6d, 0x61, 0x72, 0x79, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x08, 0x52, 0x07, 0x70, 0x72, 0x69, 0x6d, 0x61, 0x72, 0x79, 0x12, 0x18, - 0x0a, 0x07, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x07, 0x61, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0x32, 0xe1, 0x04, 0x0a, 0x04, 0x4e, 0x6f, 0x64, - 0x65, 0x12, 0x5b, 0x0a, 0x10, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x4e, 0x6f, 0x64, - 0x65, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x21, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, - 0x2e, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x4e, 0x6f, 0x64, 0x65, 0x49, 0x6e, 0x66, - 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, - 0x63, 0x6f, 0x6c, 0x2e, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x4e, 0x6f, 0x64, 0x65, - 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x58, - 0x0a, 0x0f, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, 0x45, 0x4e, 0x49, - 0x73, 0x12, 0x20, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x44, 0x65, 0x73, - 0x63, 0x72, 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, 0x45, 0x4e, 0x49, 0x73, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x44, - 0x65, 0x73, 0x63, 0x72, 0x69, 0x62, 0x65, 0x41, 0x6c, 0x6c, 0x45, 0x4e, 0x49, 0x73, 0x52, 0x65, - 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x61, 0x0a, 0x12, 0x53, 0x65, 0x74, 0x75, - 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x12, 0x23, - 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, - 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x1a, 0x24, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x53, - 0x65, 0x74, 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, - 0x49, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x61, 0x0a, 0x12, 0x43, - 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, - 0x49, 0x12, 0x23, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x43, 0x6c, 0x65, - 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x45, 0x4e, 0x49, 0x52, - 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x24, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, - 0x6c, 0x2e, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, - 0x72, 0x45, 0x4e, 0x49, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x6d, - 0x0a, 0x16, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, - 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x27, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, - 0x63, 0x6f, 0x6c, 0x2e, 0x53, 0x65, 0x74, 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, - 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x28, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x53, 0x65, 0x74, - 0x75, 0x70, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, - 0x69, 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x6d, 0x0a, - 0x16, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, - 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x27, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, - 0x6f, 0x6c, 0x2e, 0x43, 0x6c, 0x65, 0x61, 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, - 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, - 0x1a, 0x28, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, 0x6c, 0x2e, 0x43, 0x6c, 0x65, 0x61, - 0x6e, 0x4e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x46, 0x6f, 0x72, 0x53, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x42, 0x43, 0x5a, 0x41, - 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x61, 0x70, 0x65, 0x63, 0x6c, - 0x6f, 0x75, 0x64, 0x2f, 0x6b, 0x75, 0x62, 0x65, 0x62, 0x6c, 0x6f, 0x63, 0x6b, 0x73, 0x2f, 0x63, - 0x6d, 0x64, 0x2f, 0x6c, 0x6f, 0x61, 0x64, 0x62, 0x61, 0x6c, 0x61, 0x6e, 0x63, 0x65, 0x72, 0x2f, - 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f, - 0x6c, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, -} - -var ( - file_node_proto_rawDescOnce sync.Once - file_node_proto_rawDescData = file_node_proto_rawDesc -) - -func file_node_proto_rawDescGZIP() []byte { - file_node_proto_rawDescOnce.Do(func() { - file_node_proto_rawDescData = protoimpl.X.CompressGZIP(file_node_proto_rawDescData) - }) - return file_node_proto_rawDescData -} - -var file_node_proto_msgTypes = make([]protoimpl.MessageInfo, 17) -var file_node_proto_goTypes = []interface{}{ - (*DescribeNodeInfoRequest)(nil), // 0: protocol.DescribeNodeInfoRequest - (*DescribeNodeInfoResponse)(nil), // 1: protocol.DescribeNodeInfoResponse - (*DescribeAllENIsRequest)(nil), // 2: protocol.DescribeAllENIsRequest - (*DescribeAllENIsResponse)(nil), // 3: protocol.DescribeAllENIsResponse - (*SetupNetworkForServiceRequest)(nil), // 4: protocol.SetupNetworkForServiceRequest - (*SetupNetworkForServiceResponse)(nil), // 5: protocol.SetupNetworkForServiceResponse - (*CleanNetworkForServiceRequest)(nil), // 6: protocol.CleanNetworkForServiceRequest - (*CleanNetworkForServiceResponse)(nil), // 7: protocol.CleanNetworkForServiceResponse - (*SetupNetworkForENIRequest)(nil), // 8: protocol.SetupNetworkForENIRequest - (*SetupNetworkForENIResponse)(nil), // 9: protocol.SetupNetworkForENIResponse - (*CleanNetworkForENIRequest)(nil), // 10: protocol.CleanNetworkForENIRequest - (*CleanNetworkForENIResponse)(nil), // 11: protocol.CleanNetworkForENIResponse - (*InstanceInfo)(nil), // 12: protocol.InstanceInfo - (*ENIMetadata)(nil), // 13: protocol.ENIMetadata - (*IPv4Address)(nil), // 14: protocol.IPv4Address - nil, // 15: protocol.DescribeAllENIsResponse.EnisEntry - nil, // 16: protocol.ENIMetadata.TagsEntry -} -var file_node_proto_depIdxs = []int32{ - 12, // 0: protocol.DescribeNodeInfoResponse.info:type_name -> protocol.InstanceInfo - 15, // 1: protocol.DescribeAllENIsResponse.enis:type_name -> protocol.DescribeAllENIsResponse.EnisEntry - 13, // 2: protocol.SetupNetworkForServiceRequest.eni:type_name -> protocol.ENIMetadata - 13, // 3: protocol.SetupNetworkForServiceResponse.eni:type_name -> protocol.ENIMetadata - 13, // 4: protocol.CleanNetworkForServiceRequest.eni:type_name -> protocol.ENIMetadata - 13, // 5: protocol.CleanNetworkForServiceResponse.eni:type_name -> protocol.ENIMetadata - 13, // 6: protocol.SetupNetworkForENIRequest.eni:type_name -> protocol.ENIMetadata - 13, // 7: protocol.SetupNetworkForENIResponse.eni:type_name -> protocol.ENIMetadata - 13, // 8: protocol.CleanNetworkForENIRequest.eni:type_name -> protocol.ENIMetadata - 13, // 9: protocol.CleanNetworkForENIResponse.eni:type_name -> protocol.ENIMetadata - 14, // 10: protocol.ENIMetadata.ipv4_addresses:type_name -> protocol.IPv4Address - 16, // 11: protocol.ENIMetadata.tags:type_name -> protocol.ENIMetadata.TagsEntry - 13, // 12: protocol.DescribeAllENIsResponse.EnisEntry.value:type_name -> protocol.ENIMetadata - 0, // 13: protocol.Node.DescribeNodeInfo:input_type -> protocol.DescribeNodeInfoRequest - 2, // 14: protocol.Node.DescribeAllENIs:input_type -> protocol.DescribeAllENIsRequest - 8, // 15: protocol.Node.SetupNetworkForENI:input_type -> protocol.SetupNetworkForENIRequest - 10, // 16: protocol.Node.CleanNetworkForENI:input_type -> protocol.CleanNetworkForENIRequest - 4, // 17: protocol.Node.SetupNetworkForService:input_type -> protocol.SetupNetworkForServiceRequest - 6, // 18: protocol.Node.CleanNetworkForService:input_type -> protocol.CleanNetworkForServiceRequest - 1, // 19: protocol.Node.DescribeNodeInfo:output_type -> protocol.DescribeNodeInfoResponse - 3, // 20: protocol.Node.DescribeAllENIs:output_type -> protocol.DescribeAllENIsResponse - 9, // 21: protocol.Node.SetupNetworkForENI:output_type -> protocol.SetupNetworkForENIResponse - 11, // 22: protocol.Node.CleanNetworkForENI:output_type -> protocol.CleanNetworkForENIResponse - 5, // 23: protocol.Node.SetupNetworkForService:output_type -> protocol.SetupNetworkForServiceResponse - 7, // 24: protocol.Node.CleanNetworkForService:output_type -> protocol.CleanNetworkForServiceResponse - 19, // [19:25] is the sub-list for method output_type - 13, // [13:19] is the sub-list for method input_type - 13, // [13:13] is the sub-list for extension type_name - 13, // [13:13] is the sub-list for extension extendee - 0, // [0:13] is the sub-list for field type_name -} - -func init() { file_node_proto_init() } -func file_node_proto_init() { - if File_node_proto != nil { - return - } - if !protoimpl.UnsafeEnabled { - file_node_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DescribeNodeInfoRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DescribeNodeInfoResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DescribeAllENIsRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*DescribeAllENIsResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*SetupNetworkForServiceRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*SetupNetworkForServiceResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*CleanNetworkForServiceRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*CleanNetworkForServiceResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*SetupNetworkForENIRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*SetupNetworkForENIResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*CleanNetworkForENIRequest); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*CleanNetworkForENIResponse); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[12].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*InstanceInfo); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*ENIMetadata); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - file_node_proto_msgTypes[14].Exporter = func(v interface{}, i int) interface{} { - switch v := v.(*IPv4Address); i { - case 0: - return &v.state - case 1: - return &v.sizeCache - case 2: - return &v.unknownFields - default: - return nil - } - } - } - type x struct{} - out := protoimpl.TypeBuilder{ - File: protoimpl.DescBuilder{ - GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: file_node_proto_rawDesc, - NumEnums: 0, - NumMessages: 17, - NumExtensions: 0, - NumServices: 1, - }, - GoTypes: file_node_proto_goTypes, - DependencyIndexes: file_node_proto_depIdxs, - MessageInfos: file_node_proto_msgTypes, - }.Build() - File_node_proto = out.File - file_node_proto_rawDesc = nil - file_node_proto_goTypes = nil - file_node_proto_depIdxs = nil -} diff --git a/cmd/loadbalancer/internal/protocol/node.proto b/cmd/loadbalancer/internal/protocol/node.proto deleted file mode 100644 index c65b0bb64..000000000 --- a/cmd/loadbalancer/internal/protocol/node.proto +++ /dev/null @@ -1,111 +0,0 @@ -syntax = 'proto3'; - -package protocol; - -option go_package = "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol"; - -service Node { - rpc DescribeNodeInfo(DescribeNodeInfoRequest) returns (DescribeNodeInfoResponse) {} - - rpc DescribeAllENIs(DescribeAllENIsRequest) returns (DescribeAllENIsResponse) {} - - rpc SetupNetworkForENI(SetupNetworkForENIRequest) returns(SetupNetworkForENIResponse) {} - - rpc CleanNetworkForENI(CleanNetworkForENIRequest) returns(CleanNetworkForENIResponse) {} - - rpc SetupNetworkForService(SetupNetworkForServiceRequest) returns (SetupNetworkForServiceResponse) {} - - rpc CleanNetworkForService(CleanNetworkForServiceRequest) returns (CleanNetworkForServiceResponse) {} -} - -message DescribeNodeInfoRequest { - string request_id = 1; -} - -message DescribeNodeInfoResponse { - string request_id = 1; - InstanceInfo info = 2; -} - -message DescribeAllENIsRequest { - string request_id = 1; -} - -message DescribeAllENIsResponse { - string request_id = 1; - map enis = 2; -} - -message SetupNetworkForServiceRequest { - string request_id = 1; - string private_ip = 2; - ENIMetadata eni = 3; -} - -message SetupNetworkForServiceResponse { - string request_id = 1; - string private_ip = 2; - ENIMetadata eni = 3; -} - -message CleanNetworkForServiceRequest { - string request_id = 1; - string private_ip = 2; - ENIMetadata eni = 3; -} - -message CleanNetworkForServiceResponse { - string request_id = 1; - string private_ip = 2; - ENIMetadata eni = 3; -} - -message SetupNetworkForENIRequest { - string request_id = 1; - ENIMetadata eni = 2; -} - -message SetupNetworkForENIResponse { - string request_id = 1; - ENIMetadata eni = 2; -} - -message CleanNetworkForENIRequest { - string request_id = 1; - ENIMetadata eni = 2; -} - -message CleanNetworkForENIResponse { - string request_id = 1; - ENIMetadata eni = 2; -} - -message InstanceInfo { - string instance_id = 1; - - string subnet_id = 2; - - repeated string security_group_ids = 3; -} - -message ENIMetadata { - string eni_id = 1; - - string mac = 2; - - int32 device_number = 3; - - string subnet_id = 4; - - string subnet_ipv4_cidr = 5; - - repeated IPv4Address ipv4_addresses = 6; - - map tags = 7; -} - -message IPv4Address { - bool primary = 1; - - string address = 2; -} \ No newline at end of file diff --git a/cmd/loadbalancer/internal/protocol/node_grpc.pb.go b/cmd/loadbalancer/internal/protocol/node_grpc.pb.go deleted file mode 100644 index 4f61d77b8..000000000 --- a/cmd/loadbalancer/internal/protocol/node_grpc.pb.go +++ /dev/null @@ -1,286 +0,0 @@ -// Code generated by protoc-gen-go-grpc. DO NOT EDIT. -// versions: -// - protoc-gen-go-grpc v1.2.0 -// - protoc v3.21.12 -// source: node.proto - -package protocol - -import ( - context "context" - - grpc "google.golang.org/grpc" - codes "google.golang.org/grpc/codes" - status "google.golang.org/grpc/status" -) - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the grpc package it is being compiled against. -// Requires gRPC-Go v1.32.0 or later. -const _ = grpc.SupportPackageIsVersion7 - -// NodeClient is the client API for Node service. -// -// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. -type NodeClient interface { - DescribeNodeInfo(ctx context.Context, in *DescribeNodeInfoRequest, opts ...grpc.CallOption) (*DescribeNodeInfoResponse, error) - DescribeAllENIs(ctx context.Context, in *DescribeAllENIsRequest, opts ...grpc.CallOption) (*DescribeAllENIsResponse, error) - SetupNetworkForENI(ctx context.Context, in *SetupNetworkForENIRequest, opts ...grpc.CallOption) (*SetupNetworkForENIResponse, error) - CleanNetworkForENI(ctx context.Context, in *CleanNetworkForENIRequest, opts ...grpc.CallOption) (*CleanNetworkForENIResponse, error) - SetupNetworkForService(ctx context.Context, in *SetupNetworkForServiceRequest, opts ...grpc.CallOption) (*SetupNetworkForServiceResponse, error) - CleanNetworkForService(ctx context.Context, in *CleanNetworkForServiceRequest, opts ...grpc.CallOption) (*CleanNetworkForServiceResponse, error) -} - -type nodeClient struct { - cc grpc.ClientConnInterface -} - -func NewNodeClient(cc grpc.ClientConnInterface) NodeClient { - return &nodeClient{cc} -} - -func (c *nodeClient) DescribeNodeInfo(ctx context.Context, in *DescribeNodeInfoRequest, opts ...grpc.CallOption) (*DescribeNodeInfoResponse, error) { - out := new(DescribeNodeInfoResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/DescribeNodeInfo", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *nodeClient) DescribeAllENIs(ctx context.Context, in *DescribeAllENIsRequest, opts ...grpc.CallOption) (*DescribeAllENIsResponse, error) { - out := new(DescribeAllENIsResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/DescribeAllENIs", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *nodeClient) SetupNetworkForENI(ctx context.Context, in *SetupNetworkForENIRequest, opts ...grpc.CallOption) (*SetupNetworkForENIResponse, error) { - out := new(SetupNetworkForENIResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/SetupNetworkForENI", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *nodeClient) CleanNetworkForENI(ctx context.Context, in *CleanNetworkForENIRequest, opts ...grpc.CallOption) (*CleanNetworkForENIResponse, error) { - out := new(CleanNetworkForENIResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/CleanNetworkForENI", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *nodeClient) SetupNetworkForService(ctx context.Context, in *SetupNetworkForServiceRequest, opts ...grpc.CallOption) (*SetupNetworkForServiceResponse, error) { - out := new(SetupNetworkForServiceResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/SetupNetworkForService", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *nodeClient) CleanNetworkForService(ctx context.Context, in *CleanNetworkForServiceRequest, opts ...grpc.CallOption) (*CleanNetworkForServiceResponse, error) { - out := new(CleanNetworkForServiceResponse) - err := c.cc.Invoke(ctx, "/protocol.Node/CleanNetworkForService", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -// NodeServer is the server API for Node service. -// All implementations must embed UnimplementedNodeServer -// for forward compatibility -type NodeServer interface { - DescribeNodeInfo(context.Context, *DescribeNodeInfoRequest) (*DescribeNodeInfoResponse, error) - DescribeAllENIs(context.Context, *DescribeAllENIsRequest) (*DescribeAllENIsResponse, error) - SetupNetworkForENI(context.Context, *SetupNetworkForENIRequest) (*SetupNetworkForENIResponse, error) - CleanNetworkForENI(context.Context, *CleanNetworkForENIRequest) (*CleanNetworkForENIResponse, error) - SetupNetworkForService(context.Context, *SetupNetworkForServiceRequest) (*SetupNetworkForServiceResponse, error) - CleanNetworkForService(context.Context, *CleanNetworkForServiceRequest) (*CleanNetworkForServiceResponse, error) - mustEmbedUnimplementedNodeServer() -} - -// UnimplementedNodeServer must be embedded to have forward compatible implementations. -type UnimplementedNodeServer struct { -} - -func (UnimplementedNodeServer) DescribeNodeInfo(context.Context, *DescribeNodeInfoRequest) (*DescribeNodeInfoResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method DescribeNodeInfo not implemented") -} -func (UnimplementedNodeServer) DescribeAllENIs(context.Context, *DescribeAllENIsRequest) (*DescribeAllENIsResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method DescribeAllENIs not implemented") -} -func (UnimplementedNodeServer) SetupNetworkForENI(context.Context, *SetupNetworkForENIRequest) (*SetupNetworkForENIResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method SetupNetworkForENI not implemented") -} -func (UnimplementedNodeServer) CleanNetworkForENI(context.Context, *CleanNetworkForENIRequest) (*CleanNetworkForENIResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method CleanNetworkForENI not implemented") -} -func (UnimplementedNodeServer) SetupNetworkForService(context.Context, *SetupNetworkForServiceRequest) (*SetupNetworkForServiceResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method SetupNetworkForService not implemented") -} -func (UnimplementedNodeServer) CleanNetworkForService(context.Context, *CleanNetworkForServiceRequest) (*CleanNetworkForServiceResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method CleanNetworkForService not implemented") -} -func (UnimplementedNodeServer) mustEmbedUnimplementedNodeServer() {} - -// UnsafeNodeServer may be embedded to opt out of forward compatibility for this service. -// Use of this interface is not recommended, as added methods to NodeServer will -// result in compilation errors. -type UnsafeNodeServer interface { - mustEmbedUnimplementedNodeServer() -} - -func RegisterNodeServer(s grpc.ServiceRegistrar, srv NodeServer) { - s.RegisterService(&Node_ServiceDesc, srv) -} - -func _Node_DescribeNodeInfo_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(DescribeNodeInfoRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).DescribeNodeInfo(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/DescribeNodeInfo", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).DescribeNodeInfo(ctx, req.(*DescribeNodeInfoRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Node_DescribeAllENIs_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(DescribeAllENIsRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).DescribeAllENIs(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/DescribeAllENIs", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).DescribeAllENIs(ctx, req.(*DescribeAllENIsRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Node_SetupNetworkForENI_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(SetupNetworkForENIRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).SetupNetworkForENI(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/SetupNetworkForENI", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).SetupNetworkForENI(ctx, req.(*SetupNetworkForENIRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Node_CleanNetworkForENI_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(CleanNetworkForENIRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).CleanNetworkForENI(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/CleanNetworkForENI", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).CleanNetworkForENI(ctx, req.(*CleanNetworkForENIRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Node_SetupNetworkForService_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(SetupNetworkForServiceRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).SetupNetworkForService(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/SetupNetworkForService", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).SetupNetworkForService(ctx, req.(*SetupNetworkForServiceRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _Node_CleanNetworkForService_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(CleanNetworkForServiceRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(NodeServer).CleanNetworkForService(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/protocol.Node/CleanNetworkForService", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(NodeServer).CleanNetworkForService(ctx, req.(*CleanNetworkForServiceRequest)) - } - return interceptor(ctx, in, info, handler) -} - -// Node_ServiceDesc is the grpc.ServiceDesc for Node service. -// It's only intended for direct use with grpc.RegisterService, -// and not to be introspected or modified (even as a copy) -var Node_ServiceDesc = grpc.ServiceDesc{ - ServiceName: "protocol.Node", - HandlerType: (*NodeServer)(nil), - Methods: []grpc.MethodDesc{ - { - MethodName: "DescribeNodeInfo", - Handler: _Node_DescribeNodeInfo_Handler, - }, - { - MethodName: "DescribeAllENIs", - Handler: _Node_DescribeAllENIs_Handler, - }, - { - MethodName: "SetupNetworkForENI", - Handler: _Node_SetupNetworkForENI_Handler, - }, - { - MethodName: "CleanNetworkForENI", - Handler: _Node_CleanNetworkForENI_Handler, - }, - { - MethodName: "SetupNetworkForService", - Handler: _Node_SetupNetworkForService_Handler, - }, - { - MethodName: "CleanNetworkForService", - Handler: _Node_CleanNetworkForService_Handler, - }, - }, - Streams: []grpc.StreamDesc{}, - Metadata: "node.proto", -} diff --git a/cmd/loadbalancer/main.go b/cmd/loadbalancer/main.go deleted file mode 100644 index 3987ce2f3..000000000 --- a/cmd/loadbalancer/main.go +++ /dev/null @@ -1,310 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -// TODO assign multiple private ip from different subnets -// TODO create subnets instead of use host subnet -// TODO if mask of src address in policy rule little than 32 bit, causing routing problem -// TODO monitor security group / subnet / vpc changes -// TODO reuse pb.ENIMetadata -// TODO enable grpc auth, transport credentials -// TODO replace with k8s built-in grpc liveness/readiness probe when we can ensure k8s version > 1.23.0 -// TODO define FloatingIP CRD -// TODO implement device plugin to report floating ip resources -// TODO move DescribeAllENIs from agent to controller -// TODO delete enis when node is deleted - -package main - -import ( - "context" - "flag" - "fmt" - "net" - "net/http" - _ "net/http/pprof" - "os" - "time" - - "github.com/go-logr/logr" - "github.com/spf13/viper" - zaplogfmt "github.com/sykesm/zap-logfmt" - uzap "go.uber.org/zap" - "go.uber.org/zap/zapcore" - "google.golang.org/grpc" - "google.golang.org/grpc/health/grpc_health_v1" - "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/healthz" - - // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) - // to ensure that exec-entrypoint and run can make use of them. - "k8s.io/apimachinery/pkg/runtime" - utilruntime "k8s.io/apimachinery/pkg/util/runtime" - clientgoscheme "k8s.io/client-go/kubernetes/scheme" - _ "k8s.io/client-go/plugin/pkg/client/auth" - ctrl "sigs.k8s.io/controller-runtime" - "sigs.k8s.io/controller-runtime/pkg/log/zap" - - // +kubebuilder:scaffold:imports - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/agent" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud/factory" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/config" - lb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/controllers" - iptableswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/iptables" - netlinkwrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/netlink" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/network" - procfswrapper "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/procfs" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -// added lease.coordination.k8s.io for leader election -// +kubebuilder:rbac:groups=coordination.k8s.io,resources=leases,verbs=get;list;watch;create;update;patch - -const ( - appName = "loadbalancer" - RFC3339Mills = "2006-01-02T15:04:05.000" -) - -var ( - scheme = runtime.NewScheme() - setupLog = ctrl.Log.WithName("setup") -) -var ( - metricsAddr string - probeAddr string - enableLeaderElection bool - mode string - logger logr.Logger -) - -type OnLeaderAction struct { - sc *lb.ServiceController - ec *lb.EndpointController - nm agent.NodeManager -} - -func (l OnLeaderAction) NeedLeaderElection() bool { - return true -} - -func (l OnLeaderAction) Start(ctx context.Context) error { - if err := l.nm.Start(ctx); err != nil { - return err - } - if err := l.sc.Start(ctx); err != nil { - return err - } - if err := l.ec.Start(ctx); err != nil { - return err - } - return nil -} - -func init() { - utilruntime.Must(clientgoscheme.AddToScheme(scheme)) - // +kubebuilder:scaffold:scheme - - viper.SetConfigName("config") // name of config file (without extension) - viper.SetConfigType("yaml") // REQUIRED if the config file does not have the extension in the name - viper.AddConfigPath(fmt.Sprintf("/etc/%s/", appName)) // path to look for the config file in - viper.AddConfigPath(fmt.Sprintf("$HOME/.%s", appName)) // call multiple times to add many search paths - viper.AddConfigPath(".") // optionally look for config in the working directory - viper.AutomaticEnv() - - viper.SetDefault("CERT_DIR", "/tmp/k8s-webhook-server/serving-certs") - configLog := uzap.NewProductionEncoderConfig() - configLog.EncodeTime = func(ts time.Time, encoder zapcore.PrimitiveArrayEncoder) { - encoder.AppendString(ts.UTC().Format(RFC3339Mills)) - } - logFmtEncoder := zaplogfmt.NewEncoder(configLog) - // NOTES: - // zap is "Blazing fast, structured, leveled logging in Go.", DON'T event try - // to refactor this logging lib to anything else. Check FAQ - https://github.com/uber-go/zap/blob/master/FAQ.md - logger = zap.New(zap.UseDevMode(true), zap.WriteTo(os.Stdout), zap.Encoder(logFmtEncoder)) - - flag.StringVar(&mode, "mode", "agent", "The mode this binary is running as, can be either 'agent' or 'controller'") - flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.") - flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") - flag.BoolVar(&enableLeaderElection, "leader-elect", false, "Enable leader election for controller manager. "+ - "Enabling this will ensure there is only one active controller manager.") -} - -func RunController(logger logr.Logger) { - ctrl.SetLogger(logger) - - // init config - config.ReadConfig(setupLog) - - mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{ - Scheme: scheme, - MetricsBindAddress: metricsAddr, - HealthProbeBindAddress: probeAddr, - CertDir: viper.GetString("cert_dir"), - LeaderElection: enableLeaderElection, - // NOTES: - // following LeaderElectionID is generated via hash/fnv (FNV-1 and FNV-1a), in - // pattern of '{{ hashFNV .Repo }}.{{ .Domain }}', make sure regenerate this ID - // if you have forked from this project template. - LeaderElectionID: "002c317f.kubeblocks.io", - - // NOTES: - // LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily - // when the Manager ends. This requires the binary to immediately end when the - // Manager is stopped, otherwise, this setting is unsafe. Setting this significantly - // speeds up voluntary leader transitions as the new leader don't have to wait - // LeaseDuration time first. - // - // In the default scaffold provided, the program ends immediately after - // the manager stops, so would be fine to enable this option. However, - // if you are doing or is intended to do any operation such as perform cleanups - // after the manager stops then its usage might be unsafe. - LeaderElectionReleaseOnCancel: true, - }) - if err != nil { - setupLog.Error(err, "Failed to start manager") - os.Exit(1) - } - - if config.EnableDebug { - go pprofListening(logger) - } - - cfg, err := rest.InClusterConfig() - if err != nil { - setupLog.Error(err, "Failed to get incluster config") - os.Exit(1) - } - // https://github.com/kubernetes-sigs/controller-runtime/issues/343 - // The controller manager provided client is designed to do the right thing for controllers by default (which is to read from caches, meaning that it's not strongly consistent) - // We must use raw client to talk with apiserver - c, err := client.New(cfg, client.Options{Scheme: scheme}) - if err != nil { - setupLog.Error(err, "Failed to init k8s client") - os.Exit(1) - } - - cp, err := factory.NewProvider(config.CloudProvider, logger) - if err != nil { - setupLog.Error(err, "Failed to initialize cloud provider") - os.Exit(1) - } - nm, err := agent.NewNodeManager(logger, config.RPCPort, cp, c) - if err != nil { - setupLog.Error(err, "Failed to init node manager") - os.Exit(1) - } - sc, err := lb.NewServiceController(logger, c, mgr.GetScheme(), mgr.GetEventRecorderFor("LoadBalancer"), cp, nm) - if err != nil { - setupLog.Error(err, "Failed to init service controller") - os.Exit(1) - } - if err := sc.SetupWithManager(mgr); err != nil { - setupLog.Error(err, "Failed to create controller", "controller", "Service") - os.Exit(1) - } - - ec, err := lb.NewEndpointController(logger, c, mgr.GetScheme(), mgr.GetEventRecorderFor("LoadBalancer")) - if err != nil { - setupLog.Error(err, "Failed to init endpoints controller") - os.Exit(1) - } - if err := ec.SetupWithManager(mgr); err != nil { - setupLog.Error(err, "Failed to create controller", "controller", "Endpoints") - os.Exit(1) - } - - // +kubebuilder:scaffold:builder - - if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil { - setupLog.Error(err, "Failed to set up health check") - os.Exit(1) - } - if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil { - setupLog.Error(err, "Failed to set up ready check") - os.Exit(1) - } - - if err := mgr.Add(&OnLeaderAction{sc: sc, ec: ec, nm: nm}); err != nil { - setupLog.Error(err, "Failed to add on leader action") - os.Exit(1) - } - - setupLog.Info("Starting manager") - if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { - setupLog.Error(err, "Problem running manager") - os.Exit(1) - } -} - -func RunAgent(logger logr.Logger) { - flag.Parse() - - // init config - config.ReadConfig(logger) - - ipt, err := iptableswrapper.NewIPTables() - if err != nil { - logger.Error(err, "Failed to init iptables") - os.Exit(1) - } - nc, err := network.NewClient(logger, netlinkwrapper.NewNetLink(), ipt, procfswrapper.NewProcFS()) - if err != nil { - logger.Error(err, "Failed to init network client") - os.Exit(1) - } - - cp, err := factory.NewProvider(config.CloudProvider, logger) - if err != nil { - logger.Error(err, "Failed to initialize cloud provider") - os.Exit(1) - } - - lis, err := net.Listen("tcp", fmt.Sprintf("%s:%d", config.HostIP, config.RPCPort)) - if err != nil { - logger.Error(err, "Failed to listen") - os.Exit(1) - } - - server := grpc.NewServer() - proxy := &Proxy{nc: nc, cp: cp} - pb.RegisterNodeServer(server, proxy) - grpc_health_v1.RegisterHealthServer(server, proxy) - logger.Info("Exit", "err", server.Serve(lis)) -} - -func pprofListening(logger logr.Logger) { - l, err := net.Listen("tcp", "127.0.0.1:0") - if err != nil { - panic(err) - } - logger.Info("Starting pprof", "addr", l.Addr().String()) - _ = http.Serve(l, nil) -} - -func main() { - flag.Parse() - - switch mode { - case "agent": - RunAgent(logger) - case "controller": - RunController(logger) - default: - logger.Error(fmt.Errorf("unknown mode %s", mode), "") - os.Exit(1) - } -} diff --git a/cmd/loadbalancer/proxy.go b/cmd/loadbalancer/proxy.go deleted file mode 100644 index 90a0693ee..000000000 --- a/cmd/loadbalancer/proxy.go +++ /dev/null @@ -1,170 +0,0 @@ -/* -Copyright ApeCloud, Inc. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package main - -import ( - "context" - "fmt" - - "google.golang.org/grpc/codes" - "google.golang.org/grpc/health/grpc_health_v1" - "google.golang.org/grpc/status" - - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/cloud" - "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/network" - pb "github.com/apecloud/kubeblocks/cmd/loadbalancer/internal/protocol" -) - -type Proxy struct { - grpc_health_v1.UnimplementedHealthServer - pb.UnimplementedNodeServer - - cp cloud.Provider - nc network.Client -} - -func (p Proxy) DescribeAllENIs(ctx context.Context, request *pb.DescribeAllENIsRequest) (*pb.DescribeAllENIsResponse, error) { - enis, err := p.cp.DescribeAllENIs() - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("[%s], %s", request.GetRequestId(), err.Error())) - } - resp := &pb.DescribeAllENIsResponse{ - Enis: make(map[string]*pb.ENIMetadata), - RequestId: request.GetRequestId(), - } - for index := range enis { - eni := enis[index] - var addrs []*pb.IPv4Address - for i := range eni.IPv4Addresses { - addr := eni.IPv4Addresses[i] - addrs = append(addrs, &pb.IPv4Address{ - Address: addr.Address, - Primary: addr.Primary, - }) - } - resp.Enis[enis[index].ID] = &pb.ENIMetadata{ - EniId: eni.ID, - Mac: eni.MAC, - SubnetId: eni.SubnetID, - DeviceNumber: int32(eni.DeviceNumber), - SubnetIpv4Cidr: eni.SubnetIPv4CIDR, - Tags: eni.Tags, - Ipv4Addresses: addrs, - } - } - return resp, nil -} - -func (p Proxy) DescribeNodeInfo(ctx context.Context, request *pb.DescribeNodeInfoRequest) (*pb.DescribeNodeInfoResponse, error) { - info := p.cp.GetInstanceInfo() - resp := &pb.DescribeNodeInfoResponse{ - RequestId: request.GetRequestId(), - Info: &pb.InstanceInfo{ - InstanceId: info.InstanceID, - SubnetId: info.SubnetID, - SecurityGroupIds: info.SecurityGroupIDs, - }, - } - return resp, nil -} - -func (p Proxy) SetupNetworkForService(ctx context.Context, request *pb.SetupNetworkForServiceRequest) (*pb.SetupNetworkForServiceResponse, error) { - eni := cloud.ENIMetadata{ - ID: request.GetEni().GetEniId(), - MAC: request.GetEni().GetMac(), - SubnetID: request.GetEni().GetSubnetId(), - DeviceNumber: int(request.GetEni().GetDeviceNumber()), - SubnetIPv4CIDR: request.GetEni().GetSubnetIpv4Cidr(), - Tags: request.GetEni().GetTags(), - } - err := p.nc.SetupNetworkForService(request.PrivateIp, &eni) - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("[%s], %s", request.GetRequestId(), err.Error())) - } - resp := &pb.SetupNetworkForServiceResponse{ - RequestId: request.GetRequestId(), - PrivateIp: request.GetPrivateIp(), - Eni: request.GetEni(), - } - return resp, nil -} - -func (p Proxy) CleanNetworkForService(ctx context.Context, request *pb.CleanNetworkForServiceRequest) (*pb.CleanNetworkForServiceResponse, error) { - eni := cloud.ENIMetadata{ - ID: request.GetEni().GetEniId(), - MAC: request.GetEni().GetMac(), - SubnetID: request.GetEni().GetSubnetId(), - DeviceNumber: int(request.GetEni().GetDeviceNumber()), - SubnetIPv4CIDR: request.GetEni().GetSubnetIpv4Cidr(), - Tags: request.GetEni().GetTags(), - } - err := p.nc.CleanNetworkForService(request.PrivateIp, &eni) - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("[%s], %s", request.GetRequestId(), err.Error())) - } - resp := &pb.CleanNetworkForServiceResponse{ - RequestId: request.GetRequestId(), - PrivateIp: request.GetPrivateIp(), - Eni: request.GetEni(), - } - return resp, nil -} - -func (p Proxy) SetupNetworkForENI(ctx context.Context, request *pb.SetupNetworkForENIRequest) (*pb.SetupNetworkForENIResponse, error) { - eni := cloud.ENIMetadata{ - ID: request.GetEni().GetEniId(), - MAC: request.GetEni().GetMac(), - SubnetID: request.GetEni().GetSubnetId(), - DeviceNumber: int(request.GetEni().GetDeviceNumber()), - SubnetIPv4CIDR: request.GetEni().GetSubnetIpv4Cidr(), - Tags: request.GetEni().GetTags(), - } - err := p.nc.SetupNetworkForENI(&eni) - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("[%s], %s", request.GetRequestId(), err.Error())) - } - resp := &pb.SetupNetworkForENIResponse{ - RequestId: request.GetRequestId(), - Eni: request.GetEni(), - } - return resp, nil - -} - -func (p Proxy) CleanNetworkForENI(ctx context.Context, request *pb.CleanNetworkForENIRequest) (*pb.CleanNetworkForENIResponse, error) { - eni := cloud.ENIMetadata{ - ID: request.GetEni().GetEniId(), - MAC: request.GetEni().GetMac(), - DeviceNumber: int(request.GetEni().GetDeviceNumber()), - SubnetIPv4CIDR: request.GetEni().GetSubnetIpv4Cidr(), - Tags: request.GetEni().GetTags(), - } - err := p.nc.CleanNetworkForENI(&eni) - if err != nil { - return nil, status.Error(codes.Internal, fmt.Sprintf("[%s], %s", request.GetRequestId(), err.Error())) - } - resp := &pb.CleanNetworkForENIResponse{ - RequestId: request.GetRequestId(), - Eni: request.GetEni(), - } - return resp, nil - -} - -func (p Proxy) Check(context.Context, *grpc_health_v1.HealthCheckRequest) (*grpc_health_v1.HealthCheckResponse, error) { - return &grpc_health_v1.HealthCheckResponse{Status: grpc_health_v1.HealthCheckResponse_SERVING}, nil -} diff --git a/config/loadbalancer/role.yaml b/config/loadbalancer/role.yaml deleted file mode 100644 index 26a1eae6c..000000000 --- a/config/loadbalancer/role.yaml +++ /dev/null @@ -1,43 +0,0 @@ ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - creationTimestamp: null - name: loadbalancer-role -rules: -- apiGroups: - - coordination.k8s.io - resources: - - leases - verbs: - - create - - get - - list - - patch - - update - - watch -- apiGroups: - - "" - resources: - - endpoints - - pods - verbs: - - get - - list - - watch -- apiGroups: - - "" - resources: - - nodes - verbs: - - get - - list -- apiGroups: - - "" - resources: - - services - verbs: - - get - - list - - update - - watch diff --git a/deploy/helm/values.yaml b/deploy/helm/values.yaml index 8bd9f177b..75de7ea2c 100644 --- a/deploy/helm/values.yaml +++ b/deploy/helm/values.yaml @@ -1524,10 +1524,6 @@ grafana: appProtocol: "" - -loadbalancer: - enabled: false - ### snapshot-controller settings ### ref: https://artifacthub.io/packages/helm/piraeus-charts/snapshot-controller#configuration ### diff --git a/deploy/loadbalancer/Chart.yaml b/deploy/loadbalancer/Chart.yaml deleted file mode 100644 index 152c5b10e..000000000 --- a/deploy/loadbalancer/Chart.yaml +++ /dev/null @@ -1,37 +0,0 @@ -apiVersion: v2 -name: loadbalancer -description: A loadbalancer Helm chart for Kubernetes - -# A chart can be either an 'application' or a 'library' chart. -# -# Application charts are a collection of templates that can be packaged into versioned archives -# to be deployed. -# -# Library charts provide useful utilities or functions for the chart developer. They're included as -# a dependency of application charts to inject those utilities and functions into the rendering -# pipeline. Library charts do not define any templates and therefore cannot be deployed. -type: application - -# This is the chart version. This version number should be incremented each time you make changes -# to the chart and its templates, including the app version. -# Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.1.2 - -# This is the version number of the application being deployed. This version number should be -# incremented each time you make changes to the application. Versions are not expected to -# follow Semantic Versioning. They should reflect the version the application is using. -# It is recommended to use it with quotes. -appVersion: "0.1.2" - -kubeVersion: '>=1.20.0-0' - - -home: https://www.kubeblocks.io/ -icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png -keywords: - - kubernetes - - database - - paas - - -sources: - - https://github.com/apecloud/kubeblocks/ diff --git a/deploy/loadbalancer/README.md b/deploy/loadbalancer/README.md deleted file mode 100644 index a968df1e2..000000000 --- a/deploy/loadbalancer/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Installation guide - -## Test/Dev - -```bash -helm install lb . -``` - -## Production - -Use `values-production.yaml` to override default resources. - -```bash -helm install lb --values values.yaml --values values-production.yaml . -``` \ No newline at end of file diff --git a/deploy/loadbalancer/templates/_helpers.tpl b/deploy/loadbalancer/templates/_helpers.tpl deleted file mode 100644 index 5befe9cea..000000000 --- a/deploy/loadbalancer/templates/_helpers.tpl +++ /dev/null @@ -1,93 +0,0 @@ -{{/* -Expand the name of the chart. -*/}} -{{- define "loadbalancer.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "loadbalancer.fullname" -}} -{{- if .Values.fullnameOverride }} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- $name := default .Chart.Name .Values.nameOverride }} -{{- if contains $name .Release.Name }} -{{- .Release.Name | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} -{{- end }} -{{- end }} -{{- end }} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "loadbalancer.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Common labels -*/}} -{{- define "loadbalancer.labels" -}} -helm.sh/chart: {{ include "loadbalancer.chart" . }} -{{ include "loadbalancer.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "loadbalancer.selectorLabels" -}} -app.kubernetes.io/name: {{ include "loadbalancer.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - -{{/* -Create the name of the service account to use -*/}} -{{- define "loadbalancer.serviceAccountName" -}} -{{- if .Values.serviceAccount.create }} -{{- default (include "loadbalancer.fullname" .) .Values.serviceAccount.name }} -{{- else }} -{{- default "default" .Values.serviceAccount.name }} -{{- end }} -{{- end }} - -{{/* Create the default PodDisruptionBudget to use */}} -{{- define "podDisruptionBudget.spec" -}} -{{- if and .Values.podDisruptionBudget.minAvailable .Values.podDisruptionBudget.maxUnavailable }} -{{- fail "Cannot set both .Values.podDisruptionBudget.minAvailable and .Values.podDisruptionBudget.maxUnavailable" -}} -{{- end }} -{{- if not .Values.podDisruptionBudget.maxUnavailable }} -minAvailable: {{ default 1 .Values.podDisruptionBudget.minAvailable }} -{{- end }} -{{- if .Values.podDisruptionBudget.maxUnavailable }} -maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }} -{{- end }} -{{- end }} - -{{- define "loadbalancer.nodeLabels" -}} -{{- if .Values.nodeLabelList }} -{{- join "," .Values.nodeLabelList }} -{{- end }} -{{- end }} - -{{- define "loadbalancer.endpointLabels" -}} -{{- if .Values.endpointLabelList }} -{{- join "," .Values.endpointLabelList }} -{{- end }} -{{- end }} - -{{- define "loadbalancer.serviceLabels" -}} -{{- if .Values.serviceLabelList }} -{{- join "," .Values.serviceLabelList }} -{{- end }} -{{- end }} diff --git a/deploy/loadbalancer/templates/clusterrole.yaml b/deploy/loadbalancer/templates/clusterrole.yaml deleted file mode 100644 index 13714b889..000000000 --- a/deploy/loadbalancer/templates/clusterrole.yaml +++ /dev/null @@ -1,44 +0,0 @@ ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: {{ include "loadbalancer.fullname" . }}-role - labels: - {{- include "loadbalancer.labels" . | nindent 4 }} -rules: - - apiGroups: - - coordination.k8s.io - resources: - - leases - verbs: - - create - - get - - list - - patch - - update - - watch - - apiGroups: - - "" - resources: - - endpoints - - pods - verbs: - - get - - list - - watch - - apiGroups: - - "" - resources: - - nodes - verbs: - - get - - list - - apiGroups: - - "" - resources: - - services - verbs: - - get - - list - - update - - watch \ No newline at end of file diff --git a/deploy/loadbalancer/templates/clusterrolebinding.yaml b/deploy/loadbalancer/templates/clusterrolebinding.yaml deleted file mode 100644 index 24b0c66a1..000000000 --- a/deploy/loadbalancer/templates/clusterrolebinding.yaml +++ /dev/null @@ -1,14 +0,0 @@ -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: {{ include "loadbalancer.fullname" . }}-rolebinding - labels: - {{- include "loadbalancer.labels" . | nindent 4 }} -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: {{ include "loadbalancer.fullname" . }}-role -subjects: - - kind: ServiceAccount - name: {{ include "loadbalancer.serviceAccountName" . }} - namespace: {{ .Release.Namespace }} \ No newline at end of file diff --git a/deploy/loadbalancer/templates/daemonset.yaml b/deploy/loadbalancer/templates/daemonset.yaml deleted file mode 100644 index aa35e9092..000000000 --- a/deploy/loadbalancer/templates/daemonset.yaml +++ /dev/null @@ -1,88 +0,0 @@ -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: {{ include "loadbalancer.fullname" . }}-agent - labels: - {{- include "loadbalancer.labels" . | nindent 4 }} -spec: - selector: - matchLabels: - {{- include "loadbalancer.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "loadbalancer.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - containers: - - name: agent - command: - - "/loadbalancer" - args: - - "--mode=agent" - image: "{{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:v{{ .Values.image.tag | default .Chart.AppVersion }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - securityContext: - {{- toYaml .Values.podSecurityContext | nindent 10 }} - ports: - - name: health - containerPort: {{ .Values.agent.rpcPort }} - protocol: TCP - readinessProbe: - tcpSocket: - port: {{ .Values.agent.rpcPort }} - initialDelaySeconds: 5 - timeoutSeconds: 3 - livenessProbe: - tcpSocket: - port: {{ .Values.agent.rpcPort }} - initialDelaySeconds: 10 - timeoutSeconds: 3 - resources: - {{- toYaml .Values.agent.resources | nindent 12 }} - env: - - name: HOST_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.hostIP - - name: RPC_PORT - value: "{{ .Values.agent.rpcPort }}" - - name: CLOUD_PROVIDER - {{- if regexMatch ".*-eks-.*" .Capabilities.KubeVersion.GitVersion }} - value: "aws" - {{- else }} - {{ fail "loadbalancer is unsupported on this kubernetes provider" }} - {{- end }} - volumeMounts: - - mountPath: /run/xtables.lock - name: xtables-lock - dnsPolicy: ClusterFirstWithHostNet - hostNetwork: true - priorityClassName: system-node-critical - restartPolicy: Always - terminationGracePeriodSeconds: 15 - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: - - hostPath: - path: /run/xtables.lock - type: "" - name: xtables-lock diff --git a/deploy/loadbalancer/templates/deployment.yaml b/deploy/loadbalancer/templates/deployment.yaml deleted file mode 100644 index 1f5a50d38..000000000 --- a/deploy/loadbalancer/templates/deployment.yaml +++ /dev/null @@ -1,87 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "loadbalancer.fullname" . }}-controller - labels: - {{- include "loadbalancer.labels" . | nindent 4 }} -spec: - replicas: {{ .Values.controller.replicaCount }} - selector: - matchLabels: - {{- include "loadbalancer.selectorLabels" . | nindent 6 }} - template: - metadata: - {{- with .Values.podAnnotations }} - annotations: - {{- toYaml . | nindent 8 }} - {{- end }} - labels: - {{- include "loadbalancer.selectorLabels" . | nindent 8 }} - spec: - {{- with .Values.imagePullSecrets }} - imagePullSecrets: - {{- toYaml . | nindent 8 }} - {{- end }} - serviceAccountName: {{ include "loadbalancer.serviceAccountName" . }} - containers: - - name: controller - command: - - "/loadbalancer" - args: - - "--mode=controller" - - "--health-probe-bind-address=:8081" - - "--metrics-bind-address=127.0.0.1:8080" - - "--leader-elect" - image: "{{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:v{{ .Values.image.tag | default .Chart.AppVersion }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - ports: - - name: health - containerPort: 8081 - protocol: TCP - - name: webhook-server - containerPort: 9443 - protocol: TCP - livenessProbe: - httpGet: - path: /healthz - port: health - initialDelaySeconds: 15 - periodSeconds: 20 - env: - - name: RPC_PORT - value: "{{ .Values.agent.rpcPort }}" - - name: TRAFFIC_NODE_LABELS - value: {{ include "loadbalancer.nodeLabels" . | quote}} - - name: ENDPOINTS_LABELS - value: {{ include "loadbalancer.endpointLabels" . | quote }} - - name: SERVICE_LABELS - value: {{ include "loadbalancer.serviceLabels" . | quote }} - - name: CLOUD_PROVIDER - {{- if regexMatch ".*-eks-.*" .Capabilities.KubeVersion.GitVersion }} - value: "aws" - {{- else }} - {{ fail "loadbalancer is unsupported on this kubernetes provider" }} - {{- end }} - readinessProbe: - httpGet: - path: /readyz - port: health - initialDelaySeconds: 5 - periodSeconds: 10 - resources: - {{- toYaml .Values.controller.resources | nindent 12 }} - volumeMounts: - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - terminationGracePeriodSeconds: 10 - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} - volumes: diff --git a/deploy/loadbalancer/templates/serviceaccount.yaml b/deploy/loadbalancer/templates/serviceaccount.yaml deleted file mode 100644 index 8effac3e3..000000000 --- a/deploy/loadbalancer/templates/serviceaccount.yaml +++ /dev/null @@ -1,12 +0,0 @@ -{{- if .Values.serviceAccount.create -}} -apiVersion: v1 -kind: ServiceAccount -metadata: - name: {{ include "loadbalancer.serviceAccountName" . }} - labels: - {{- include "loadbalancer.labels" . | nindent 4 }} - {{- with .Values.serviceAccount.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -{{- end }} diff --git a/deploy/loadbalancer/values-production.yaml b/deploy/loadbalancer/values-production.yaml deleted file mode 100644 index 001ae927f..000000000 --- a/deploy/loadbalancer/values-production.yaml +++ /dev/null @@ -1,18 +0,0 @@ -controller: - replicaCount: 2 - resources: - limits: - cpu: 100m - memory: 50Mi - requests: - cpu: 100m - memory: 50Mi - -agent: - resources: - limits: - cpu: 100m - memory: 30Mi - requests: - cpu: 100m - memory: 30Mi diff --git a/deploy/loadbalancer/values.yaml b/deploy/loadbalancer/values.yaml deleted file mode 100644 index cffabc111..000000000 --- a/deploy/loadbalancer/values.yaml +++ /dev/null @@ -1,111 +0,0 @@ -# Default values for loadbalancer. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -image: - registry: registry.cn-hangzhou.aliyuncs.com - repository: apecloud/loadbalancer - pullPolicy: Always - # Overrides the image tag whose default is the chart appVersion. - tag: "" - -controller: - replicaCount: 1 - resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # TODO(user): Configure the resources accordingly based on the project requirements. - # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - # limits: - # cpu: 500m - # memory: 128Mi - # requests: - # cpu: 10m - # memory: 64Mi - -agent: - rpcPort: 19200 - resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # TODO(user): Configure the resources accordingly based on the project requirements. - # More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - # limits: - # cpu: 500m - # memory: 128Mi - # requests: - # cpu: 10m - # memory: 64Mi - -nameOverride: "" -fullnameOverride: "" -enableDebugLog: true - -serviceAccount: - # Specifies whether a service account should be created - create: true - # Annotations to add to the service account - annotations: {} - # The name of the service account to use. - # If not set and create is true, a name is generated using the fullname template - name: "" - -imagePullSecrets: [] - -podAnnotations: {} - -tolerations: - - key: kb-controller - operator: Equal - value: "true" - effect: NoSchedule - -affinity: - nodeAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - preference: - matchExpressions: - - key: kb-controller - operator: In - values: - - "true" - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchLabels: - app: loadbalancer - topologyKey: kubernetes.io/hostname - -nodeSelector: - kubernetes.io/os: linux - -podSecurityContext: - privileged: true - capabilities: - add: - - ALL - - CHOWN - - KILL - - NET_ADMIN - - NET_RAW - - IPC_LOCK - - SYS_MODULE - - SYS_ADMIN - - SYS_RESOURCE - - DAC_OVERRIDE - - FOWNER - - SETGID - - SETUID - -nodeLabelList: - -serviceLabelList: - - app.kubernetes.io/managed-by:kubeblocks - -endpointLabelList: - - app.kubernetes.io/managed-by:kubeblocks diff --git a/docker/Dockerfile-loadbalancer b/docker/Dockerfile-loadbalancer deleted file mode 100644 index b50c6f898..000000000 --- a/docker/Dockerfile-loadbalancer +++ /dev/null @@ -1,49 +0,0 @@ -# Build the loadbalancer binary -FROM --platform=${BUILDPLATFORM} golang:1.19 as builder - -## docker buildx buid injected build-args: -#BUILDPLATFORM — matches the current machine. (e.g. linux/amd64) -#BUILDOS — os component of BUILDPLATFORM, e.g. linux -#BUILDARCH — e.g. amd64, arm64, riscv64 -#BUILDVARIANT — used to set ARM variant, e.g. v7 -#TARGETPLATFORM — The value set with --platform flag on build -#TARGETOS - OS component from --platform, e.g. linux -#TARGETARCH - Architecture from --platform, e.g. arm64 -#TARGETVARIANT - -ARG TARGETOS -ARG TARGETARCH - - -ARG LD_FLAGS="-s -w" - -ENV GONOPROXY=github.com/apecloud -ENV GONOSUMDB=github.com/apecloud -ENV GOPRIVATE=github.com/apecloud -ENV GOPROXY=https://goproxy.cn - -WORKDIR /workspace -# Copy the Go Modules manifests -COPY go.mod go.mod -COPY go.sum go.sum -# cache deps before building and copying source so that we don't need to re-download as much -# and so that source changes don't invalidate our downloaded layer -RUN go mod download - -# Copy the go source -COPY apis/ apis/ -COPY version/ version/ -COPY controllers/ controllers/ -COPY internal/ internal/ -COPY cmd/loadbalancer/ cmd/loadbalancer/ - -# Build -RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -ldflags="${LD_FLAGS}" -a -o loadbalancer ./cmd/loadbalancer/ - -# https://gallery.ecr.aws/eks-distro-build-tooling/eks-distro-minimal-base-iptables -FROM public.ecr.aws/eks-distro-build-tooling/eks-distro-minimal-base-iptables:2023-03-06-1678131614.2 -WORKDIR / -COPY --from=builder /workspace/loadbalancer . - - -ENTRYPOINT ["/loadbalancer"] diff --git a/docker/docker.mk b/docker/docker.mk index a7cdaf6ef..2a61c35ab 100644 --- a/docker/docker.mk +++ b/docker/docker.mk @@ -85,34 +85,6 @@ else endif endif -.PHONY: build-loadbalancer-image -build-loadbalancer-image: generate ## Push docker image with the loadbalancer. -ifneq ($(BUILDX_ENABLED), true) - docker build . -t ${IMG}:${VERSION} -t ${IMG}:latest -f $(DOCKERFILE_DIR)/Dockerfile-loadbalancer -else -ifeq ($(TAG_LATEST), true) - docker buildx build . $(DOCKER_BUILD_ARGS) --platform $(BUILDX_PLATFORMS) -t ${IMG}:latest -f $(DOCKERFILE_DIR)/Dockerfile-loadbalancer $(BUILDX_ARGS) -else - docker buildx build . $(DOCKER_BUILD_ARGS) --platform $(BUILDX_PLATFORMS) -t ${IMG}:${VERSION} -f $(DOCKERFILE_DIR)/Dockerfile-loadbalancer $(BUILDX_ARGS) -endif -endif - -.PHONY: push-loadbalancer-image -push-loadbalancer-image: generate ## Push docker image with the loadbalancer. -ifneq ($(BUILDX_ENABLED), true) -ifeq ($(TAG_LATEST), true) - docker push ${IMG}:latest -else - docker push ${IMG}:${VERSION} -endif -else -ifeq ($(TAG_LATEST), true) - docker buildx build . $(DOCKER_BUILD_ARGS) --platform $(BUILDX_PLATFORMS) -t ${IMG}:latest -f $(DOCKERFILE_DIR)/Dockerfile-loadbalancer --push $(BUILDX_ARGS) -else - docker buildx build . $(DOCKER_BUILD_ARGS) --platform $(BUILDX_PLATFORMS) -t ${IMG}:${VERSION} -f $(DOCKERFILE_DIR)/Dockerfile-loadbalancer --push $(BUILDX_ARGS) -endif -endif - .PHONY: build-tools-image build-tools-image: generate ## Build tools container image. ifneq ($(BUILDX_ENABLED), true) diff --git a/docs/user_docs/kubeblocks-for-mysql/migration/migration.md b/docs/user_docs/kubeblocks-for-mysql/migration/migration.md index 54d39201d..e6f4c6694 100644 --- a/docs/user_docs/kubeblocks-for-mysql/migration/migration.md +++ b/docs/user_docs/kubeblocks-for-mysql/migration/migration.md @@ -30,7 +30,7 @@ The Kubernetes ClusterIP of ApeCloud MySQL is exposed by default in the EKS envi 2. Install ApeCloud MySQL. Refer to [Create an ApeCloud MySQL cluster on AWS](./../../quick-start/create-a-mysql-cluster-on-aws.md) for details. 3. Fill in the cluster name and run the command below to expose the external IP of the cluster. ```bash - KBCLI_EXPERIMENTAL_EXPOSE="1" kbcli cluster expose ${mysql clustrName} --enable=true --type=vpc + kbcli cluster expose ${mysql clustrName} --enable=true --type=vpc ``` Run the command below to view the external IP:Port address which can be accessed by the same VPC machine but outside the EKS cluster. diff --git a/internal/cli/cluster/helper.go b/internal/cli/cluster/helper.go index 281c85f98..53e5ef68e 100644 --- a/internal/cli/cluster/helper.go +++ b/internal/cli/cluster/helper.go @@ -190,7 +190,7 @@ func GetExternalAddr(svc *corev1.Service) string { return ingress.IP } } - if svc.GetAnnotations()[types.ServiceLBTypeAnnotationKey] != types.ServiceLBTypeAnnotationValue { + if svc.GetAnnotations()[types.ServiceHAVIPTypeAnnotationKey] != types.ServiceHAVIPTypeAnnotationValue { return "" } return svc.GetAnnotations()[types.ServiceFloatingIPAnnotationKey] diff --git a/internal/cli/testing/fake.go b/internal/cli/testing/fake.go index 9335333c5..788b70b21 100644 --- a/internal/cli/testing/fake.go +++ b/internal/cli/testing/fake.go @@ -339,7 +339,7 @@ func FakeServices() *corev1.ServiceList { annotations[types.ServiceFloatingIPAnnotationKey] = item.floatingIP } if item.exposed { - annotations[types.ServiceLBTypeAnnotationKey] = types.ServiceLBTypeAnnotationValue + annotations[types.ServiceHAVIPTypeAnnotationKey] = types.ServiceHAVIPTypeAnnotationValue } svc.ObjectMeta.SetAnnotations(annotations) diff --git a/internal/cli/types/types.go b/internal/cli/types/types.go index 87028146f..418c73a69 100644 --- a/internal/cli/types/types.go +++ b/internal/cli/types/types.go @@ -92,9 +92,9 @@ const ( // Annotations const ( - ServiceLBTypeAnnotationKey = "service.kubernetes.io/kubeblocks-loadbalancer-type" - ServiceLBTypeAnnotationValue = "private-ip" - ServiceFloatingIPAnnotationKey = "service.kubernetes.io/kubeblocks-loadbalancer-floating-ip" + ServiceHAVIPTypeAnnotationKey = "service.kubernetes.io/kubeblocks-havip-type" + ServiceHAVIPTypeAnnotationValue = "private-ip" + ServiceFloatingIPAnnotationKey = "service.kubernetes.io/kubeblocks-havip-floating-ip" ) // DataProtection API group From 64d37ea7094003dcb9bfa509438439e3abebafe3 Mon Sep 17 00:00:00 2001 From: Ziang Guo Date: Fri, 31 Mar 2023 17:47:57 +0800 Subject: [PATCH 20/80] feat: support class definition (#1551) --- PROJECT | 10 +- apis/apps/v1alpha1/classfamily_types.go | 199 ++++++++++++++ apis/apps/v1alpha1/classfamily_types_test.go | 83 ++++++ .../apps.kubeblocks.io_classfamilies.yaml | 141 ++++++++++ config/crd/kustomization.yaml | 3 + .../cainjection_in_apps_classfamilies.yaml | 7 + .../webhook_in_apps_classfamilies.yaml | 16 ++ config/rbac/apps_classfamily_editor_role.yaml | 31 +++ config/rbac/apps_classfamily_viewer_role.yaml | 27 ++ config/rbac/role.yaml | 8 + config/samples/apps_v1alpha1_classfamily.yaml | 12 + controllers/apps/cluster_controller.go | 10 + deploy/apecloud-mysql/templates/class.yaml | 63 +++++ deploy/helm/config/rbac/role.yaml | 8 + .../apps.kubeblocks.io_classfamilies.yaml | 141 ++++++++++ deploy/helm/templates/class/classfamily.yaml | 44 +++ docs/user_docs/cli/cli.md | 9 + docs/user_docs/cli/kbcli.md | 1 + docs/user_docs/cli/kbcli_class.md | 45 +++ docs/user_docs/cli/kbcli_class_create.md | 63 +++++ docs/user_docs/cli/kbcli_class_list.md | 54 ++++ docs/user_docs/cli/kbcli_class_template.md | 47 ++++ docs/user_docs/cli/kbcli_cluster_create.md | 3 + docs/user_docs/cli/kbcli_cluster_vscale.md | 1 + go.mod | 2 +- internal/class/class_utils.go | 257 +++++++++++++++++ internal/class/class_utils_test.go | 99 +++++++ internal/class/types.go | 168 ++++++++++++ internal/class/types_test.go | 91 ++++++ internal/cli/cmd/class/class.go | 41 +++ internal/cli/cmd/class/class_test.go | 44 +++ internal/cli/cmd/class/create.go | 259 ++++++++++++++++++ internal/cli/cmd/class/create_test.go | 146 ++++++++++ internal/cli/cmd/class/list.go | 113 ++++++++ internal/cli/cmd/class/list_test.go | 84 ++++++ internal/cli/cmd/class/suite_test.go | 75 +++++ internal/cli/cmd/class/template.go | 82 ++++++ internal/cli/cmd/class/template_test.go | 45 +++ internal/cli/cmd/cli.go | 2 + internal/cli/cmd/cluster/cluster_test.go | 14 +- internal/cli/cmd/cluster/create.go | 54 +++- internal/cli/cmd/cluster/create_test.go | 13 +- .../cli/cmd/cluster/dataprotection_test.go | 6 +- internal/cli/cmd/cluster/describe_test.go | 1 - internal/cli/cmd/cluster/operations.go | 2 + internal/cli/create/create.go | 6 + internal/cli/testing/fake.go | 37 ++- internal/cli/testing/testdata/class.yaml | 61 +++++ .../testing/testdata/classfamily-general.yaml | 23 ++ .../classfamily-memory-optimized.yaml | 18 ++ .../cli/testing/testdata/custom_class.yaml | 33 +++ internal/cli/types/types.go | 10 + internal/constant/const.go | 5 + .../lifecycle/cluster_plan_builder.go | 2 + .../lifecycle/transformer_fill_class.go | 157 +++++++++++ 55 files changed, 2943 insertions(+), 33 deletions(-) create mode 100644 apis/apps/v1alpha1/classfamily_types.go create mode 100644 apis/apps/v1alpha1/classfamily_types_test.go create mode 100644 config/crd/bases/apps.kubeblocks.io_classfamilies.yaml create mode 100644 config/crd/patches/cainjection_in_apps_classfamilies.yaml create mode 100644 config/crd/patches/webhook_in_apps_classfamilies.yaml create mode 100644 config/rbac/apps_classfamily_editor_role.yaml create mode 100644 config/rbac/apps_classfamily_viewer_role.yaml create mode 100644 config/samples/apps_v1alpha1_classfamily.yaml create mode 100644 deploy/apecloud-mysql/templates/class.yaml create mode 100644 deploy/helm/crds/apps.kubeblocks.io_classfamilies.yaml create mode 100644 deploy/helm/templates/class/classfamily.yaml create mode 100644 docs/user_docs/cli/kbcli_class.md create mode 100644 docs/user_docs/cli/kbcli_class_create.md create mode 100644 docs/user_docs/cli/kbcli_class_list.md create mode 100644 docs/user_docs/cli/kbcli_class_template.md create mode 100644 internal/class/class_utils.go create mode 100644 internal/class/class_utils_test.go create mode 100644 internal/class/types.go create mode 100644 internal/class/types_test.go create mode 100644 internal/cli/cmd/class/class.go create mode 100644 internal/cli/cmd/class/class_test.go create mode 100644 internal/cli/cmd/class/create.go create mode 100644 internal/cli/cmd/class/create_test.go create mode 100644 internal/cli/cmd/class/list.go create mode 100644 internal/cli/cmd/class/list_test.go create mode 100644 internal/cli/cmd/class/suite_test.go create mode 100644 internal/cli/cmd/class/template.go create mode 100644 internal/cli/cmd/class/template_test.go create mode 100644 internal/cli/testing/testdata/class.yaml create mode 100644 internal/cli/testing/testdata/classfamily-general.yaml create mode 100644 internal/cli/testing/testdata/classfamily-memory-optimized.yaml create mode 100644 internal/cli/testing/testdata/custom_class.yaml create mode 100644 internal/controller/lifecycle/transformer_fill_class.go diff --git a/PROJECT b/PROJECT index 7dc13fa29..847d7728e 100644 --- a/PROJECT +++ b/PROJECT @@ -113,7 +113,6 @@ resources: crdVersion: v1 namespaced: true domain: troubleshoot.sh - group: kind: HostPreflight path: github.com/apecloud/kubeblocks/externalapis/preflight/v1beta2 version: v1beta2 @@ -121,8 +120,15 @@ resources: crdVersion: v1 namespaced: true domain: troubleshoot.sh - group: kind: Preflight path: github.com/apecloud/kubeblocks/externalapis/preflight/v1beta2 version: v1beta2 +- api: + crdVersion: v1 + namespaced: true + domain: kubeblocks.io + group: apps + kind: ClassFamily + path: github.com/apecloud/kubeblocks/apis/apps/v1alpha1 + version: v1alpha1 version: "3" diff --git a/apis/apps/v1alpha1/classfamily_types.go b/apis/apps/v1alpha1/classfamily_types.go new file mode 100644 index 000000000..e833e3019 --- /dev/null +++ b/apis/apps/v1alpha1/classfamily_types.go @@ -0,0 +1,199 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "golang.org/x/exp/slices" + "gopkg.in/inf.v0" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// ClassFamilySpec defines the desired state of ClassFamily +type ClassFamilySpec struct { + // Class family models, generally, a model is a specific constraint for CPU, memory and their relation. + Models []ClassFamilyModel `json:"models,omitempty"` +} + +type ClassFamilyModel struct { + // The constraint for vcpu cores. + // +kubebuilder:validation:Required + CPU CPUConstraint `json:"cpu,omitempty"` + + // The constraint for memory size. + // +kubebuilder:validation:Required + Memory MemoryConstraint `json:"memory,omitempty"` +} + +type CPUConstraint struct { + // The maximum count of vcpu cores, [Min, Max] defines a range for valid vcpu cores, and the value in this range + // must be multiple times of Step. It's useful to define a large number of valid values without defining them one by + // one. Please see the documentation for Step for some examples. + // If Slots is specified, Max, Min, and Step are ignored + // +optional + Max *resource.Quantity `json:"max,omitempty"` + + // The minimum count of vcpu cores, [Min, Max] defines a range for valid vcpu cores, and the value in this range + // must be multiple times of Step. It's useful to define a large number of valid values without defining them one by + // one. Please see the documentation for Step for some examples. + // If Slots is specified, Max, Min, and Step are ignored + // +optional + Min *resource.Quantity `json:"min,omitempty"` + + // The minimum granularity of vcpu cores, [Min, Max] defines a range for valid vcpu cores and the value in this range must be + // multiple times of Step. + // For example: + // 1. Min is 2, Max is 8, Step is 2, and the valid vcpu core is {2, 4, 6, 8}. + // 2. Min is 0.5, Max is 2, Step is 0.5, and the valid vcpu core is {0.5, 1, 1.5, 2}. + // +optional + Step *resource.Quantity `json:"step,omitempty"` + + // The valid vcpu cores, it's useful if you want to define valid vcpu cores explicitly. + // If Slots is specified, Max, Min, and Step are ignored + // +optional + Slots []resource.Quantity `json:"slots,omitempty"` +} + +type MemoryConstraint struct { + // The size of memory per vcpu core. + // For example: 1Gi, 200Mi. + // If SizePerCPU is specified, MinPerCPU and MaxPerCPU are ignore. + // +optional + SizePerCPU *resource.Quantity `json:"sizePerCPU,omitempty"` + + // The maximum size of memory per vcpu core, [MinPerCPU, MaxPerCPU] defines a range for valid memory size per vcpu core. + // It is useful on GCP as the ratio between the CPU and memory may be a range. + // If SizePerCPU is specified, MinPerCPU and MaxPerCPU are ignored. + // Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types + // +optional + MaxPerCPU *resource.Quantity `json:"maxPerCPU,omitempty"` + + // The minimum size of memory per vcpu core, [MinPerCPU, MaxPerCPU] defines a range for valid memory size per vcpu core. + // It is useful on GCP as the ratio between the CPU and memory may be a range. + // If SizePerCPU is specified, MinPerCPU and MaxPerCPU are ignored. + // Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types + // +optional + MinPerCPU *resource.Quantity `json:"minPerCPU,omitempty"` +} + +// +kubebuilder:object:root=true +// +kubebuilder:resource:categories={kubeblocks,all},scope=Cluster,shortName=cf + +// ClassFamily is the Schema for the classfamilies API +type ClassFamily struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec ClassFamilySpec `json:"spec,omitempty"` +} + +// +kubebuilder:object:root=true + +// ClassFamilyList contains a list of ClassFamily +type ClassFamilyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []ClassFamily `json:"items"` +} + +func init() { + SchemeBuilder.Register(&ClassFamily{}, &ClassFamilyList{}) +} + +// ValidateCPU validate if the CPU matches the class family model constraint +func (m *ClassFamilyModel) ValidateCPU(cpu resource.Quantity) bool { + if m == nil { + return false + } + if m.CPU.Min != nil && m.CPU.Min.Cmp(cpu) > 0 { + return false + } + if m.CPU.Max != nil && m.CPU.Max.Cmp(cpu) < 0 { + return false + } + if m.CPU.Slots != nil && slices.Index(m.CPU.Slots, cpu) < 0 { + return false + } + return true +} + +// ValidateMemory validate if the memory matches the class family model constraint +func (m *ClassFamilyModel) ValidateMemory(cpu *resource.Quantity, memory *resource.Quantity) bool { + if m == nil { + return false + } + + if memory == nil { + return true + } + + // fast path if cpu is specified + if cpu != nil && m.Memory.SizePerCPU != nil { + return inf.NewDec(1, 0).Mul(cpu.AsDec(), m.Memory.SizePerCPU.AsDec()).Cmp(memory.AsDec()) == 0 + } + + if cpu != nil && m.Memory.MaxPerCPU != nil && m.Memory.MinPerCPU != nil { + maxMemory := inf.NewDec(1, 0).Mul(cpu.AsDec(), m.Memory.MaxPerCPU.AsDec()) + minMemory := inf.NewDec(1, 0).Mul(cpu.AsDec(), m.Memory.MinPerCPU.AsDec()) + return maxMemory.Cmp(memory.AsDec()) >= 0 && minMemory.Cmp(memory.AsDec()) <= 0 + } + + // TODO slow path if cpu is not specified + + return true +} + +// ValidateResourceRequirements validate if the resource matches the class family model constraints +func (m *ClassFamilyModel) ValidateResourceRequirements(r *corev1.ResourceRequirements) bool { + var ( + cpu = r.Requests.Cpu() + memory = r.Requests.Memory() + ) + + if m == nil { + return false + } + + if cpu.IsZero() && memory.IsZero() { + return true + } + + if !m.ValidateCPU(*cpu) { + return false + } + + if !m.ValidateMemory(cpu, memory) { + return false + } + + return true +} + +// FindMatchingModels find all class family models that resource matches +func (c *ClassFamily) FindMatchingModels(r *corev1.ResourceRequirements) []ClassFamilyModel { + if c == nil { + return nil + } + var models []ClassFamilyModel + for _, model := range c.Spec.Models { + if model.ValidateResourceRequirements(r) { + models = append(models, model) + } + } + return models +} diff --git a/apis/apps/v1alpha1/classfamily_types_test.go b/apis/apps/v1alpha1/classfamily_types_test.go new file mode 100644 index 000000000..be491451d --- /dev/null +++ b/apis/apps/v1alpha1/classfamily_types_test.go @@ -0,0 +1,83 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "testing" + + "github.com/stretchr/testify/assert" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + "k8s.io/apimachinery/pkg/util/yaml" +) + +const classFamilyBytes = ` +# API scope: cluster +# ClusterClassFamily +apiVersion: "apps.kubeblocks.io/v1alpha1" +kind: "ClassFamily" +metadata: + name: kb-class-family-general +spec: + models: + - cpu: + min: 0.5 + max: 128 + step: 0.5 + memory: + sizePerCPU: 4Gi + - cpu: + slots: [0.1, 0.2, 0.4, 0.6, 0.8, 1] + memory: + minPerCPU: 200Mi + - cpu: + min: 0.1 + max: 64 + step: 0.1 + memory: + minPerCPU: 4Gi + maxPerCPU: 8Gi +` + +func TestClassFamily_ValidateResourceRequirements(t *testing.T) { + var cf ClassFamily + err := yaml.Unmarshal([]byte(classFamilyBytes), &cf) + if err != nil { + panic("Failed to unmarshal class family: %v" + err.Error()) + } + cases := []struct { + cpu string + memory string + expect bool + }{ + {cpu: "0.5", memory: "2Gi", expect: true}, + {cpu: "0.2", memory: "40Mi", expect: true}, + {cpu: "1", memory: "6Gi", expect: true}, + {cpu: "2", memory: "20Gi", expect: false}, + {cpu: "2", memory: "6Gi", expect: false}, + } + + for _, item := range cases { + requirements := &corev1.ResourceRequirements{ + Requests: map[corev1.ResourceName]resource.Quantity{ + corev1.ResourceCPU: resource.MustParse(item.cpu), + corev1.ResourceMemory: resource.MustParse(item.memory), + }, + } + assert.Equal(t, item.expect, len(cf.FindMatchingModels(requirements)) > 0) + } +} diff --git a/config/crd/bases/apps.kubeblocks.io_classfamilies.yaml b/config/crd/bases/apps.kubeblocks.io_classfamilies.yaml new file mode 100644 index 000000000..1c4f66d32 --- /dev/null +++ b/config/crd/bases/apps.kubeblocks.io_classfamilies.yaml @@ -0,0 +1,141 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.0 + creationTimestamp: null + name: classfamilies.apps.kubeblocks.io +spec: + group: apps.kubeblocks.io + names: + categories: + - kubeblocks + - all + kind: ClassFamily + listKind: ClassFamilyList + plural: classfamilies + shortNames: + - cf + singular: classfamily + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: ClassFamily is the Schema for the classfamilies API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: ClassFamilySpec defines the desired state of ClassFamily + properties: + models: + description: Class family models, generally, a model is a specific + constraint for CPU, memory and their relation. + items: + properties: + cpu: + description: The constraint for vcpu cores. + properties: + max: + anyOf: + - type: integer + - type: string + description: The maximum count of vcpu cores, [Min, Max] + defines a range for valid vcpu cores, and the value in + this range must be multiple times of Step. It's useful + to define a large number of valid values without defining + them one by one. Please see the documentation for Step + for some examples. If Slots is specified, Max, Min, and + Step are ignored + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + min: + anyOf: + - type: integer + - type: string + description: The minimum count of vcpu cores, [Min, Max] + defines a range for valid vcpu cores, and the value in + this range must be multiple times of Step. It's useful + to define a large number of valid values without defining + them one by one. Please see the documentation for Step + for some examples. If Slots is specified, Max, Min, and + Step are ignored + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + slots: + description: The valid vcpu cores, it's useful if you want + to define valid vcpu cores explicitly. If Slots is specified, + Max, Min, and Step are ignored + items: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: array + step: + anyOf: + - type: integer + - type: string + description: 'The minimum granularity of vcpu cores, [Min, + Max] defines a range for valid vcpu cores and the value + in this range must be multiple times of Step. For example: + 1. Min is 2, Max is 8, Step is 2, and the valid vcpu core + is {2, 4, 6, 8}. 2. Min is 0.5, Max is 2, Step is 0.5, + and the valid vcpu core is {0.5, 1, 1.5, 2}.' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + memory: + description: The constraint for memory size. + properties: + maxPerCPU: + anyOf: + - type: integer + - type: string + description: 'The maximum size of memory per vcpu core, + [MinPerCPU, MaxPerCPU] defines a range for valid memory + size per vcpu core. It is useful on GCP as the ratio between + the CPU and memory may be a range. If SizePerCPU is specified, + MinPerCPU and MaxPerCPU are ignored. Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + minPerCPU: + anyOf: + - type: integer + - type: string + description: 'The minimum size of memory per vcpu core, + [MinPerCPU, MaxPerCPU] defines a range for valid memory + size per vcpu core. It is useful on GCP as the ratio between + the CPU and memory may be a range. If SizePerCPU is specified, + MinPerCPU and MaxPerCPU are ignored. Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + sizePerCPU: + anyOf: + - type: integer + - type: string + description: 'The size of memory per vcpu core. For example: + 1Gi, 200Mi. If SizePerCPU is specified, MinPerCPU and + MaxPerCPU are ignore.' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + type: object + type: array + type: object + type: object + served: true + storage: true diff --git a/config/crd/kustomization.yaml b/config/crd/kustomization.yaml index 69e51f6b6..7de82f64b 100644 --- a/config/crd/kustomization.yaml +++ b/config/crd/kustomization.yaml @@ -13,6 +13,7 @@ resources: - bases/dataprotection.kubeblocks.io_restorejobs.yaml - bases/dataprotection.kubeblocks.io_backuppolicytemplates.yaml - bases/extensions.kubeblocks.io_addons.yaml +- bases/apps.kubeblocks.io_classfamilies.yaml #+kubebuilder:scaffold:crdkustomizeresource patchesStrategicMerge: @@ -32,6 +33,7 @@ patchesStrategicMerge: #- patches/webhook_in_hostpreflights.yaml #- patches/webhook_in_preflights.yaml #- patches/webhook_in_addons.yaml +#- patches/webhook_in_classfamilies.yaml #+kubebuilder:scaffold:crdkustomizewebhookpatch # [CERTMANAGER] To enable cert-manager, uncomment all the sections with [CERTMANAGER] prefix. @@ -50,6 +52,7 @@ patchesStrategicMerge: #- patches/cainjection_in_hostpreflights.yaml #- patches/cainjection_in_preflights.yaml #- patches/cainjection_in_addonspecs.yaml +#- patches/cainjection_in_classfamilies.yaml #+kubebuilder:scaffold:crdkustomizecainjectionpatch # the following config is for teaching kustomize how to do kustomization for CRDs. diff --git a/config/crd/patches/cainjection_in_apps_classfamilies.yaml b/config/crd/patches/cainjection_in_apps_classfamilies.yaml new file mode 100644 index 000000000..bc1f3ff5c --- /dev/null +++ b/config/crd/patches/cainjection_in_apps_classfamilies.yaml @@ -0,0 +1,7 @@ +# The following patch adds a directive for certmanager to inject CA into the CRD +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME) + name: classfamilies.apps.kubeblocks.io diff --git a/config/crd/patches/webhook_in_apps_classfamilies.yaml b/config/crd/patches/webhook_in_apps_classfamilies.yaml new file mode 100644 index 000000000..1667132fe --- /dev/null +++ b/config/crd/patches/webhook_in_apps_classfamilies.yaml @@ -0,0 +1,16 @@ +# The following patch enables a conversion webhook for the CRD +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: classfamilies.apps.kubeblocks.io +spec: + conversion: + strategy: Webhook + webhook: + clientConfig: + service: + namespace: system + name: webhook-service + path: /convert + conversionReviewVersions: + - v1 diff --git a/config/rbac/apps_classfamily_editor_role.yaml b/config/rbac/apps_classfamily_editor_role.yaml new file mode 100644 index 000000000..5a06154b4 --- /dev/null +++ b/config/rbac/apps_classfamily_editor_role.yaml @@ -0,0 +1,31 @@ +# permissions for end users to edit classfamilies. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: clusterrole + app.kubernetes.io/instance: classfamily-editor-role + app.kubernetes.io/component: rbac + app.kubernetes.io/created-by: kubeblocks + app.kubernetes.io/part-of: kubeblocks + app.kubernetes.io/managed-by: kustomize + name: classfamily-editor-role +rules: +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies/status + verbs: + - get diff --git a/config/rbac/apps_classfamily_viewer_role.yaml b/config/rbac/apps_classfamily_viewer_role.yaml new file mode 100644 index 000000000..d82810999 --- /dev/null +++ b/config/rbac/apps_classfamily_viewer_role.yaml @@ -0,0 +1,27 @@ +# permissions for end users to view classfamilies. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/name: clusterrole + app.kubernetes.io/instance: classfamily-viewer-role + app.kubernetes.io/component: rbac + app.kubernetes.io/created-by: kubeblocks + app.kubernetes.io/part-of: kubeblocks + app.kubernetes.io/managed-by: kustomize + name: classfamily-viewer-role +rules: +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies + verbs: + - get + - list + - watch +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies/status + verbs: + - get diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml index b3607a9e1..e2747ca6a 100644 --- a/config/rbac/role.yaml +++ b/config/rbac/role.yaml @@ -103,6 +103,14 @@ rules: - statefulsets/status verbs: - get +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies + verbs: + - get + - list + - watch - apiGroups: - apps.kubeblocks.io resources: diff --git a/config/samples/apps_v1alpha1_classfamily.yaml b/config/samples/apps_v1alpha1_classfamily.yaml new file mode 100644 index 000000000..af312721a --- /dev/null +++ b/config/samples/apps_v1alpha1_classfamily.yaml @@ -0,0 +1,12 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClassFamily +metadata: + labels: + app.kubernetes.io/name: classfamily + app.kubernetes.io/instance: classfamily-sample + app.kubernetes.io/part-of: kubeblocks + app.kuberentes.io/managed-by: kustomize + app.kubernetes.io/created-by: kubeblocks + name: classfamily-sample +spec: + # TODO(user): Add fields here diff --git a/controllers/apps/cluster_controller.go b/controllers/apps/cluster_controller.go index 180a58d06..018abcf9e 100644 --- a/controllers/apps/cluster_controller.go +++ b/controllers/apps/cluster_controller.go @@ -30,7 +30,10 @@ import ( ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/builder" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/handler" "sigs.k8s.io/controller-runtime/pkg/log" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/source" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" dataprotectionv1alpha1 "github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1" @@ -91,6 +94,9 @@ import ( // +kubebuilder:rbac:groups=dataprotection.kubeblocks.io,resources=backuppolicies,verbs=get;list;delete;deletecollection // +kubebuilder:rbac:groups=dataprotection.kubeblocks.io,resources=backups,verbs=get;list;delete;deletecollection +// classfamily get list +// +kubebuilder:rbac:groups=apps.kubeblocks.io,resources=classfamilies,verbs=get;list;watch + // ClusterReconciler reconciles a Cluster object type ClusterReconciler struct { client.Client @@ -160,6 +166,10 @@ func (r *ClusterReconciler) SetupWithManager(mgr ctrl.Manager) error { if viper.GetBool("VOLUMESNAPSHOT") { b.Owns(&snapshotv1.VolumeSnapshot{}, builder.OnlyMetadata, builder.Predicates{}) } + b.Watches(&source.Kind{Type: &appsv1alpha1.ClassFamily{}}, + &handler.EnqueueRequestForObject{}, + builder.WithPredicates(predicate.NewPredicateFuncs(func(object client.Object) bool { return true })), + ) return b.Complete(r) } diff --git a/deploy/apecloud-mysql/templates/class.yaml b/deploy/apecloud-mysql/templates/class.yaml new file mode 100644 index 000000000..af0db9a7c --- /dev/null +++ b/deploy/apecloud-mysql/templates/class.yaml @@ -0,0 +1,63 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: kb.classes.default.apecloud-mysql.mysql + labels: + class.kubeblocks.io/level: component + class.kubeblocks.io/provider: kubeblocks + apps.kubeblocks.io/component-def-ref: mysql + clusterdefinition.kubeblocks.io/name: apecloud-mysql +data: + families-20230223162700: | + - family: kb-class-family-general + template: | + cpu: {{ printf "{{ or .cpu 1 }}" }} + memory: {{ printf "{{ or .memory 4 }}Gi" }} + storage: + - name: data + size: {{ printf "{{ or .dataStorageSize 10 }}Gi" }} + - name: log + size: {{ printf "{{ or .logStorageSize 1 }}Gi" }} + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - name: {{ printf "general-{{ .cpu }}c{{ .memory }}g" }} + classes: + - args: [1, 1, 100, 10] + - args: [2, 2, 100, 10] + - args: [2, 4, 100, 10] + - args: [2, 8, 100, 10] + - args: [4, 16, 100, 10] + - args: [8, 32, 100, 10] + - args: [16, 64, 200, 10] + - args: [32, 128, 200, 10] + - args: [64, 256, 200, 10] + - args: [128, 512, 200, 10] + + - family: kb-class-family-memory-optimized + template: | + cpu: {{ printf "{{ or .cpu 1 }}" }} + memory: {{ printf "{{ or .memory 8 }}Gi" }} + storage: + - name: data + size: {{ printf "{{ or .dataStorageSize 10 }}Gi" }} + - name: log + size: {{ printf "{{ or .logStorageSize 1 }}Gi" }} + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - name: {{ printf "mo-{{ .cpu }}c{{ .memory }}g" }} + classes: + # 1:8 + - args: [2, 16, 100, 10] + - args: [4, 32, 100, 10] + - args: [8, 64, 100, 10] + - args: [12, 96, 100, 10] + - args: [24, 192, 200, 10] + - args: [48, 384, 200, 10] + # 1:16 + - args: [2, 32, 100, 10] + - args: [4, 64, 100, 10] + - args: [8, 128, 100, 10] + - args: [16, 256, 100, 10] + - args: [32, 512, 200, 10] + - args: [48, 768, 200, 10] + - args: [64, 1024, 200, 10] \ No newline at end of file diff --git a/deploy/helm/config/rbac/role.yaml b/deploy/helm/config/rbac/role.yaml index b3607a9e1..e2747ca6a 100644 --- a/deploy/helm/config/rbac/role.yaml +++ b/deploy/helm/config/rbac/role.yaml @@ -103,6 +103,14 @@ rules: - statefulsets/status verbs: - get +- apiGroups: + - apps.kubeblocks.io + resources: + - classfamilies + verbs: + - get + - list + - watch - apiGroups: - apps.kubeblocks.io resources: diff --git a/deploy/helm/crds/apps.kubeblocks.io_classfamilies.yaml b/deploy/helm/crds/apps.kubeblocks.io_classfamilies.yaml new file mode 100644 index 000000000..1c4f66d32 --- /dev/null +++ b/deploy/helm/crds/apps.kubeblocks.io_classfamilies.yaml @@ -0,0 +1,141 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.9.0 + creationTimestamp: null + name: classfamilies.apps.kubeblocks.io +spec: + group: apps.kubeblocks.io + names: + categories: + - kubeblocks + - all + kind: ClassFamily + listKind: ClassFamilyList + plural: classfamilies + shortNames: + - cf + singular: classfamily + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: ClassFamily is the Schema for the classfamilies API + properties: + apiVersion: + description: 'APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' + type: string + kind: + description: 'Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + metadata: + type: object + spec: + description: ClassFamilySpec defines the desired state of ClassFamily + properties: + models: + description: Class family models, generally, a model is a specific + constraint for CPU, memory and their relation. + items: + properties: + cpu: + description: The constraint for vcpu cores. + properties: + max: + anyOf: + - type: integer + - type: string + description: The maximum count of vcpu cores, [Min, Max] + defines a range for valid vcpu cores, and the value in + this range must be multiple times of Step. It's useful + to define a large number of valid values without defining + them one by one. Please see the documentation for Step + for some examples. If Slots is specified, Max, Min, and + Step are ignored + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + min: + anyOf: + - type: integer + - type: string + description: The minimum count of vcpu cores, [Min, Max] + defines a range for valid vcpu cores, and the value in + this range must be multiple times of Step. It's useful + to define a large number of valid values without defining + them one by one. Please see the documentation for Step + for some examples. If Slots is specified, Max, Min, and + Step are ignored + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + slots: + description: The valid vcpu cores, it's useful if you want + to define valid vcpu cores explicitly. If Slots is specified, + Max, Min, and Step are ignored + items: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: array + step: + anyOf: + - type: integer + - type: string + description: 'The minimum granularity of vcpu cores, [Min, + Max] defines a range for valid vcpu cores and the value + in this range must be multiple times of Step. For example: + 1. Min is 2, Max is 8, Step is 2, and the valid vcpu core + is {2, 4, 6, 8}. 2. Min is 0.5, Max is 2, Step is 0.5, + and the valid vcpu core is {0.5, 1, 1.5, 2}.' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + memory: + description: The constraint for memory size. + properties: + maxPerCPU: + anyOf: + - type: integer + - type: string + description: 'The maximum size of memory per vcpu core, + [MinPerCPU, MaxPerCPU] defines a range for valid memory + size per vcpu core. It is useful on GCP as the ratio between + the CPU and memory may be a range. If SizePerCPU is specified, + MinPerCPU and MaxPerCPU are ignored. Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + minPerCPU: + anyOf: + - type: integer + - type: string + description: 'The minimum size of memory per vcpu core, + [MinPerCPU, MaxPerCPU] defines a range for valid memory + size per vcpu core. It is useful on GCP as the ratio between + the CPU and memory may be a range. If SizePerCPU is specified, + MinPerCPU and MaxPerCPU are ignored. Reference: https://cloud.google.com/compute/docs/general-purpose-machines#custom_machine_types' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + sizePerCPU: + anyOf: + - type: integer + - type: string + description: 'The size of memory per vcpu core. For example: + 1Gi, 200Mi. If SizePerCPU is specified, MinPerCPU and + MaxPerCPU are ignore.' + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + type: object + type: object + type: array + type: object + type: object + served: true + storage: true diff --git a/deploy/helm/templates/class/classfamily.yaml b/deploy/helm/templates/class/classfamily.yaml new file mode 100644 index 000000000..14ad1038f --- /dev/null +++ b/deploy/helm/templates/class/classfamily.yaml @@ -0,0 +1,44 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClassFamily +metadata: + name: kb-class-family-general + labels: + classfamily.kubeblocks.io/provider: kubeblocks +spec: + models: + - cpu: + min: "0.5" + max: 2 + step: "0.5" + memory: + sizePerCPU: 1Gi + - cpu: + min: 2 + max: 2 + memory: + sizePerCPU: 2Gi + - cpu: + slots: [2, 4, 8, 16, 24, 32, 48, 64, 96, 128] + memory: + sizePerCPU: 4Gi + +--- + +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClassFamily +metadata: + name: kb-class-family-memory-optimized + labels: + classfamily.kubeblocks.io/provider: kubeblocks +spec: + models: + - cpu: + slots: [2, 4, 8, 12, 24, 48] + memory: + sizePerCPU: 8Gi + - cpu: + min: 2 + max: 128 + step: 2 + memory: + sizePerCPU: 16Gi \ No newline at end of file diff --git a/docs/user_docs/cli/cli.md b/docs/user_docs/cli/cli.md index 5498fc635..9ee6411f1 100644 --- a/docs/user_docs/cli/cli.md +++ b/docs/user_docs/cli/cli.md @@ -36,6 +36,15 @@ Run a benchmark. * [kbcli bench tpcc](kbcli_bench_tpcc.md) - Run a TPCC benchmark. +## [class](kbcli_class.md) + +Manage classes + +* [kbcli class create](kbcli_class_create.md) - Create a class +* [kbcli class list](kbcli_class_list.md) - List classes +* [kbcli class template](kbcli_class_template.md) - Generate class definition template + + ## [cluster](kbcli_cluster.md) Cluster command. diff --git a/docs/user_docs/cli/kbcli.md b/docs/user_docs/cli/kbcli.md index 21bfb63a2..8935f3900 100644 --- a/docs/user_docs/cli/kbcli.md +++ b/docs/user_docs/cli/kbcli.md @@ -58,6 +58,7 @@ kbcli [flags] * [kbcli alert](kbcli_alert.md) - Manage alert receiver, include add, list and delete receiver. * [kbcli backup-config](kbcli_backup-config.md) - KubeBlocks backup config. * [kbcli bench](kbcli_bench.md) - Run a benchmark. +* [kbcli class](kbcli_class.md) - Manage classes * [kbcli cluster](kbcli_cluster.md) - Cluster command. * [kbcli clusterdefinition](kbcli_clusterdefinition.md) - ClusterDefinition command. * [kbcli clusterversion](kbcli_clusterversion.md) - ClusterVersion command. diff --git a/docs/user_docs/cli/kbcli_class.md b/docs/user_docs/cli/kbcli_class.md new file mode 100644 index 000000000..fc5536db4 --- /dev/null +++ b/docs/user_docs/cli/kbcli_class.md @@ -0,0 +1,45 @@ +--- +title: kbcli class +--- + +Manage classes + +### Options + +``` + -h, --help help for class +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "$HOME/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use +``` + +### SEE ALSO + + +* [kbcli class create](kbcli_class_create.md) - Create a class +* [kbcli class list](kbcli_class_list.md) - List classes +* [kbcli class template](kbcli_class_template.md) - Generate class definition template + +#### Go Back to [CLI Overview](cli.md) Homepage. + diff --git a/docs/user_docs/cli/kbcli_class_create.md b/docs/user_docs/cli/kbcli_class_create.md new file mode 100644 index 000000000..966cd13d7 --- /dev/null +++ b/docs/user_docs/cli/kbcli_class_create.md @@ -0,0 +1,63 @@ +--- +title: kbcli class create +--- + +Create a class + +``` +kbcli class create [NAME] [flags] +``` + +### Examples + +``` + # Create a class following class family kubeblocks-general-classes for component mysql in cluster definition apecloud-mysql, which have 1 cpu core, 2Gi memory and storage is 10Gi + kbcli class create custom-1c2g --cluster-definition apecloud-mysql --type mysql --class-family kubeblocks-general-classes --cpu 1 --memory 2Gi --storage name=data,size=10Gi + + # Create classes for component mysql in cluster definition apecloud-mysql, where classes is defined in file + kbcli class create --cluster-definition apecloud-mysql --type mysql --file ./classes.yaml +``` + +### Options + +``` + --class-family string Specify class family + --cluster-definition string Specify cluster definition, run "kbcli cluster-definition list" to show all available cluster definition + --cpu string Specify component cpu cores + --file string Specify file path which contains YAML definition of class + -h, --help help for create + --memory string Specify component memory size + --storage stringArray Specify component storage disks + --type string Specify component type +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "$HOME/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use +``` + +### SEE ALSO + +* [kbcli class](kbcli_class.md) - Manage classes + +#### Go Back to [CLI Overview](cli.md) Homepage. + diff --git a/docs/user_docs/cli/kbcli_class_list.md b/docs/user_docs/cli/kbcli_class_list.md new file mode 100644 index 000000000..798d52e85 --- /dev/null +++ b/docs/user_docs/cli/kbcli_class_list.md @@ -0,0 +1,54 @@ +--- +title: kbcli class list +--- + +List classes + +``` +kbcli class list [flags] +``` + +### Examples + +``` + # List all components classes in cluster definition apecloud-mysql + kbcli class list --cluster-definition apecloud-mysql +``` + +### Options + +``` + --cluster-definition string Specify cluster definition, run "kbcli cluster-definition list" to show all available cluster definition + -h, --help help for list +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "$HOME/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use +``` + +### SEE ALSO + +* [kbcli class](kbcli_class.md) - Manage classes + +#### Go Back to [CLI Overview](cli.md) Homepage. + diff --git a/docs/user_docs/cli/kbcli_class_template.md b/docs/user_docs/cli/kbcli_class_template.md new file mode 100644 index 000000000..92af595ad --- /dev/null +++ b/docs/user_docs/cli/kbcli_class_template.md @@ -0,0 +1,47 @@ +--- +title: kbcli class template +--- + +Generate class definition template + +``` +kbcli class template [flags] +``` + +### Options + +``` + -h, --help help for template + -o, --output string Output class definition template to a file +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "$HOME/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use +``` + +### SEE ALSO + +* [kbcli class](kbcli_class.md) - Manage classes + +#### Go Back to [CLI Overview](cli.md) Homepage. + diff --git a/docs/user_docs/cli/kbcli_cluster_create.md b/docs/user_docs/cli/kbcli_cluster_create.md index 0648bbfd5..424296190 100644 --- a/docs/user_docs/cli/kbcli_cluster_create.md +++ b/docs/user_docs/cli/kbcli_cluster_create.md @@ -35,6 +35,9 @@ kbcli cluster create [CLUSTER_NAME] [flags] # Create a cluster and set cpu to 1 core, memory to 1Gi, storage size to 20Gi and replicas to 3 kbcli cluster create mycluster --cluster-definition apecloud-mysql --set cpu=1,memory=1Gi,storage=20Gi,replicas=3 + # Create a cluster and set class to general-1c4g + kbcli cluster create myclsuter --cluster-definition apecloud-mysql --set class=general-1c4g + # Create a cluster and use a URL to set cluster resource kbcli cluster create mycluster --cluster-definition apecloud-mysql --set-file https://kubeblocks.io/yamls/my.yaml diff --git a/docs/user_docs/cli/kbcli_cluster_vscale.md b/docs/user_docs/cli/kbcli_cluster_vscale.md index 10067b092..1817bc4fe 100644 --- a/docs/user_docs/cli/kbcli_cluster_vscale.md +++ b/docs/user_docs/cli/kbcli_cluster_vscale.md @@ -18,6 +18,7 @@ kbcli cluster vscale [flags] ### Options ``` + --class string Component class --components strings Component names to this operations --cpu string Requested and limited size of component cpu -h, --help help for vscale diff --git a/go.mod b/go.mod index 222f80938..a96d6442c 100644 --- a/go.mod +++ b/go.mod @@ -77,6 +77,7 @@ require ( golang.org/x/sys v0.5.0 google.golang.org/grpc v1.52.0 google.golang.org/protobuf v1.28.1 + gopkg.in/inf.v0 v0.9.1 gopkg.in/yaml.v2 v2.4.0 helm.sh/helm/v3 v3.11.1 k8s.io/api v0.26.1 @@ -359,7 +360,6 @@ require ( google.golang.org/api v0.107.0 // indirect google.golang.org/appengine v1.6.7 // indirect google.golang.org/genproto v0.0.0-20230104163317-caabf589fcbf // indirect - gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/ini.v1 v1.67.0 // indirect gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect gopkg.in/warnings.v0 v0.1.2 // indirect diff --git a/internal/class/class_utils.go b/internal/class/class_utils.go new file mode 100644 index 000000000..dfe420dd0 --- /dev/null +++ b/internal/class/class_utils.go @@ -0,0 +1,257 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "context" + "fmt" + "sort" + "strconv" + "strings" + "text/template" + "time" + + "github.com/ghodss/yaml" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" + + "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/constant" +) + +// GetCustomClassConfigMapName Returns the name of the ConfigMap containing the custom classes +func GetCustomClassConfigMapName(cdName string, componentName string) string { + return fmt.Sprintf("kb.classes.custom.%s.%s", cdName, componentName) +} + +// ChooseComponentClasses Choose the classes to be used for a given component with some constraints +func ChooseComponentClasses(classes map[string]*ComponentClass, filters map[string]resource.Quantity) *ComponentClass { + var candidates []*ComponentClass + for _, cls := range classes { + cpu, ok := filters[corev1.ResourceCPU.String()] + if ok && !cpu.Equal(cls.CPU) { + continue + } + memory, ok := filters[corev1.ResourceMemory.String()] + if ok && !memory.Equal(cls.Memory) { + continue + } + candidates = append(candidates, cls) + } + if len(candidates) == 0 { + return nil + } + sort.Sort(ByClassCPUAndMemory(candidates)) + return candidates[0] +} + +func GetClassFamilies(dynamic dynamic.Interface) (map[string]*v1alpha1.ClassFamily, error) { + objs, err := dynamic.Resource(types.ClassFamilyGVR()).Namespace("").List(context.TODO(), metav1.ListOptions{ + //LabelSelector: types.ClassFamilyProviderLabelKey, + }) + if err != nil { + return nil, err + } + var classFamilyList v1alpha1.ClassFamilyList + if err = runtime.DefaultUnstructuredConverter.FromUnstructured(objs.UnstructuredContent(), &classFamilyList); err != nil { + return nil, err + } + + result := make(map[string]*v1alpha1.ClassFamily) + for _, cf := range classFamilyList.Items { + if _, ok := cf.GetLabels()[types.ClassFamilyProviderLabelKey]; !ok { + continue + } + result[cf.GetName()] = &cf + } + return result, nil +} + +// GetClasses Get all classes, including kubeblocks default classes and user custom classes +func GetClasses(client kubernetes.Interface, cdName string) (map[string]map[string]*ComponentClass, error) { + selector := fmt.Sprintf("%s=%s,%s", constant.ClusterDefLabelKey, cdName, types.ClassProviderLabelKey) + cmList, err := client.CoreV1().ConfigMaps(metav1.NamespaceAll).List(context.TODO(), metav1.ListOptions{ + LabelSelector: selector, + }) + if err != nil { + return nil, err + } + return ParseClasses(cmList) +} + +func ParseClasses(cmList *corev1.ConfigMapList) (map[string]map[string]*ComponentClass, error) { + var ( + componentClasses = make(map[string]map[string]*ComponentClass) + ) + for _, cm := range cmList.Items { + if _, ok := cm.GetLabels()[types.ClassProviderLabelKey]; !ok { + continue + } + level := cm.GetLabels()[types.ClassLevelLabelKey] + switch level { + case "component": + componentType := cm.GetLabels()[constant.KBAppComponentDefRefLabelKey] + if componentType == "" { + return nil, fmt.Errorf("failed to find component type") + } + classes, err := ParseComponentClasses(cm.Data) + if err != nil { + return nil, err + } + if _, ok := componentClasses[componentType]; !ok { + componentClasses[componentType] = classes + } else { + for k, v := range classes { + if _, exists := componentClasses[componentType][k]; exists { + return nil, fmt.Errorf("duplicate component class %s", k) + } + componentClasses[componentType][k] = v + } + } + case "cluster": + // TODO + default: + return nil, fmt.Errorf("invalid class level: %s", level) + } + } + + return componentClasses, nil +} + +type classVersion int64 + +// ParseComponentClasses parse configmap.data to component classes +func ParseComponentClasses(data map[string]string) (map[string]*ComponentClass, error) { + versions := make(map[classVersion][]*ComponentClassFamilyDef) + + for k, v := range data { + // ConfigMap data key follows the format: families-[version] + // version is the timestamp in unix microseconds which class is created + parts := strings.SplitAfterN(k, "-", 2) + if len(parts) != 2 { + return nil, fmt.Errorf("invalid key: %s", k) + } + version, err := strconv.ParseInt(parts[1], 10, 64) + if err != nil { + return nil, fmt.Errorf("invalid key: %s", k) + } + var families []*ComponentClassFamilyDef + if err := yaml.Unmarshal([]byte(v), &families); err != nil { + return nil, err + } + versions[classVersion(version)] = families + } + + genClassDef := func(nameTpl string, bodyTpl string, vars []string, args []string) (ComponentClassDef, error) { + var def ComponentClassDef + values := make(map[string]interface{}) + for index, key := range vars { + values[key] = args[index] + } + + classStr, err := renderTemplate(bodyTpl, values) + if err != nil { + return def, err + } + + if err = yaml.Unmarshal([]byte(classStr), &def); err != nil { + return def, err + } + + def.Name, err = renderTemplate(nameTpl, values) + if err != nil { + return def, err + } + return def, nil + } + + parser := func(family *ComponentClassFamilyDef, series ComponentClassSeriesDef, class ComponentClassDef) (*ComponentClass, error) { + var ( + err error + def = class + ) + + if len(class.Args) > 0 { + def, err = genClassDef(series.Name, family.Template, family.Vars, class.Args) + if err != nil { + return nil, err + } + + if class.Name != "" { + def.Name = class.Name + } + } + + result := &ComponentClass{ + Name: def.Name, + Family: family.Family, + CPU: resource.MustParse(def.CPU), + Memory: resource.MustParse(def.Memory), + } + + for _, disk := range def.Storage { + result.Storage = append(result.Storage, &Disk{ + Name: disk.Name, + Class: disk.Class, + Size: resource.MustParse(disk.Size), + }) + } + + return result, nil + } + + result := make(map[string]*ComponentClass) + for _, families := range versions { + for _, family := range families { + for _, series := range family.Series { + for _, class := range series.Classes { + out, err := parser(family, series, class) + if err != nil { + return nil, err + } + if _, exists := result[out.Name]; exists { + return nil, fmt.Errorf("duplicate component class name: %s", out.Name) + } + result[out.Name] = out + } + } + } + } + return result, nil +} + +func renderTemplate(tpl string, values map[string]interface{}) (string, error) { + engine, err := template.New("").Parse(tpl) + if err != nil { + return "", err + } + var buf strings.Builder + if err := engine.Execute(&buf, values); err != nil { + return "", err + } + return buf.String(), nil +} + +// BuildClassDefinitionVersion generate the key in the configmap data field +func BuildClassDefinitionVersion() string { + return fmt.Sprintf("version-%s", time.Now().Format("20060102150405")) +} diff --git a/internal/class/class_utils_test.go b/internal/class/class_utils_test.go new file mode 100644 index 000000000..f68ec1fb0 --- /dev/null +++ b/internal/class/class_utils_test.go @@ -0,0 +1,99 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "k8s.io/apimachinery/pkg/api/resource" +) + +var _ = Describe("utils", func() { + var ( + cpuMin = 1 + cpuMax = 64 + scales = []int{4, 8, 16} + classes map[string]*ComponentClass + ) + + genComponentClasses := func(cpuMin int, cpuMax int, scales []int) map[string]*ComponentClass { + results := make(map[string]*ComponentClass) + for cpu := cpuMin; cpu <= cpuMax; cpu++ { + for _, scale := range scales { + name := fmt.Sprintf("cpu-%d-scale-%d", cpu, scale) + results[name] = &ComponentClass{ + Name: name, + CPU: resource.MustParse(fmt.Sprintf("%d", cpu)), + Memory: resource.MustParse(fmt.Sprintf("%dGi", cpu*scale)), + } + } + } + return results + } + + BeforeEach(func() { + classes = genComponentClasses(cpuMin, cpuMax, scales) + }) + + AfterEach(func() { + // Add any teardown steps that needs to be executed after each test + }) + + buildFilters := func(cpu string, memory string) map[string]resource.Quantity { + result := make(map[string]resource.Quantity) + if cpu != "" { + result["cpu"] = resource.MustParse(cpu) + } + if memory != "" { + result["memory"] = resource.MustParse(memory) + } + return result + } + + Context("sort component classes", func() { + It("should match one class by cpu and memory", func() { + class := ChooseComponentClasses(classes, buildFilters("1", "4Gi")) + Expect(class.CPU.String()).Should(Equal("1")) + Expect(class.Memory.String()).Should(Equal("4Gi")) + }) + + It("match multiple classes by cpu", func() { + class := ChooseComponentClasses(classes, buildFilters("1", "")) + Expect(class.CPU.String()).Should(Equal("1")) + Expect(class.Memory.String()).Should(Equal("4Gi")) + }) + + It("match multiple classes by memory", func() { + class := ChooseComponentClasses(classes, buildFilters("", "16Gi")) + Expect(class.CPU.String()).Should(Equal("1")) + Expect(class.Memory.String()).Should(Equal("16Gi")) + }) + + It("not match any classes by cpu", func() { + class := ChooseComponentClasses(classes, buildFilters(fmt.Sprintf("%d", cpuMax+1), "")) + Expect(class).Should(BeNil()) + }) + + It("not match any classes by memory", func() { + class := ChooseComponentClasses(classes, buildFilters("", "1Pi")) + Expect(class).Should(BeNil()) + }) + }) +}) diff --git a/internal/class/types.go b/internal/class/types.go new file mode 100644 index 000000000..dc43a38e2 --- /dev/null +++ b/internal/class/types.go @@ -0,0 +1,168 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "fmt" + "sort" + "strings" + + "gopkg.in/inf.v0" + "k8s.io/apimachinery/pkg/api/resource" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" +) + +func GetMinCPUAndMemory(model appsv1alpha1.ClassFamilyModel) (*resource.Quantity, *resource.Quantity) { + var ( + minCPU resource.Quantity + minMemory resource.Quantity + ) + + if len(model.CPU.Slots) > 0 { + minCPU = model.CPU.Slots[0] + } + + if model.CPU.Min != nil && minCPU.Cmp(*model.CPU.Min) < 0 { + minCPU = *model.CPU.Min + } + var memory *inf.Dec + if model.Memory.MinPerCPU != nil { + memory = inf.NewDec(1, 0).Mul(minCPU.AsDec(), model.Memory.MinPerCPU.AsDec()) + } else { + memory = inf.NewDec(1, 0).Mul(minCPU.AsDec(), model.Memory.SizePerCPU.AsDec()) + } + minMemory = resource.MustParse(memory.String()) + return &minCPU, &minMemory +} + +type ClassModelWithFamilyName struct { + Family string + Model appsv1alpha1.ClassFamilyModel +} + +type ByModelList []ClassModelWithFamilyName + +func (m ByModelList) Len() int { + return len(m) +} + +func (m ByModelList) Less(i, j int) bool { + cpu1, mem1 := GetMinCPUAndMemory(m[i].Model) + cpu2, mem2 := GetMinCPUAndMemory(m[j].Model) + switch cpu1.Cmp(*cpu2) { + case 1: + return false + case -1: + return true + } + switch mem1.Cmp(*mem2) { + case 1: + return false + case -1: + return true + } + return false +} + +func (m ByModelList) Swap(i, j int) { + m[i], m[j] = m[j], m[i] +} + +type ComponentClass struct { + Name string `json:"name,omitempty"` + CPU resource.Quantity `json:"cpu,omitempty"` + Memory resource.Quantity `json:"memory,omitempty"` + Storage []*Disk `json:"storage,omitempty"` + Family string `json:"-"` +} + +var _ sort.Interface = ByClassCPUAndMemory{} + +type ByClassCPUAndMemory []*ComponentClass + +func (b ByClassCPUAndMemory) Len() int { + return len(b) +} + +func (b ByClassCPUAndMemory) Less(i, j int) bool { + if out := b[i].CPU.Cmp(b[j].CPU); out != 0 { + return out < 0 + } + + if out := b[i].Memory.Cmp(b[j].Memory); out != 0 { + return out < 0 + } + + return false +} + +func (b ByClassCPUAndMemory) Swap(i, j int) { + b[i], b[j] = b[j], b[i] +} + +type Filters map[string]resource.Quantity + +func (f Filters) String() string { + var result []string + for k, v := range f { + result = append(result, fmt.Sprintf("%s=%v", k, v.Value())) + } + return strings.Join(result, ",") +} + +type Disk struct { + Name string `json:"name,omitempty"` + Size resource.Quantity `json:"size,omitempty"` + Class string `json:"class,omitempty"` +} + +func (d Disk) String() string { + return fmt.Sprintf("%s=%s", d.Name, d.Size.String()) +} + +type ProviderComponentClassDef struct { + Provider string `json:"provider,omitempty"` + Args []string `json:"args,omitempty"` +} + +type DiskDef struct { + Name string `json:"name,omitempty"` + Size string `json:"size,omitempty"` + Class string `json:"class,omitempty"` +} + +type ComponentClassDef struct { + Name string `json:"name,omitempty"` + CPU string `json:"cpu,omitempty"` + Memory string `json:"memory,omitempty"` + Storage []DiskDef `json:"storage,omitempty"` + Args []string `json:"args,omitempty"` + Variants []ProviderComponentClassDef `json:"variants,omitempty"` +} + +type ComponentClassSeriesDef struct { + Name string `json:"name,omitempty"` + Classes []ComponentClassDef `json:"classes,omitempty"` +} + +type ComponentClassFamilyDef struct { + Family string `json:"family"` + Template string `json:"template,omitempty"` + Vars []string `json:"vars,omitempty"` + Series []ComponentClassSeriesDef `json:"series,omitempty"` +} diff --git a/internal/class/types_test.go b/internal/class/types_test.go new file mode 100644 index 000000000..ad65a631b --- /dev/null +++ b/internal/class/types_test.go @@ -0,0 +1,91 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "sort" + "testing" + + "github.com/ghodss/yaml" + "github.com/stretchr/testify/assert" + "k8s.io/apimachinery/pkg/api/resource" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" +) + +const classFamilyBytes = ` +# API scope: cluster +# ClusterClassFamily +apiVersion: "apps.kubeblocks.io/v1alpha1" +kind: "ClassFamily" +metadata: + name: kb-class-family-general +spec: + models: + - cpu: + min: 0.5 + max: 128 + step: 0.5 + memory: + sizePerCPU: 4Gi + - cpu: + slots: [0.1, 0.2, 0.4, 0.6, 0.8, 1] + memory: + minPerCPU: 200Mi + - cpu: + min: 0.1 + max: 64 + step: 0.1 + memory: + minPerCPU: 4Gi + maxPerCPU: 8Gi +` + +func TestClassFamily_ByClassCPUAndMemory(t *testing.T) { + buildClass := func(cpu string, memory string) *ComponentClass { + return &ComponentClass{CPU: resource.MustParse(cpu), Memory: resource.MustParse(memory)} + } + classes := []*ComponentClass{ + buildClass("1", "2Gi"), + buildClass("1", "1Gi"), + buildClass("2", "0.5Gi"), + buildClass("1", "1Gi"), + buildClass("0.5", "10Gi"), + } + sort.Sort(ByClassCPUAndMemory(classes)) + candidate := classes[0] + if candidate.CPU != resource.MustParse("0.5") || candidate.Memory != resource.MustParse("10Gi") { + t.Errorf("case failed") + } +} + +func TestClassFamily_ModelList(t *testing.T) { + var cf appsv1alpha1.ClassFamily + err := yaml.Unmarshal([]byte(classFamilyBytes), &cf) + if err != nil { + panic("Failed to unmarshal class family: %v" + err.Error()) + } + var models []ClassModelWithFamilyName + for _, model := range cf.Spec.Models { + models = append(models, ClassModelWithFamilyName{Family: cf.Name, Model: model}) + } + resource.MustParse("200Mi") + sort.Sort(ByModelList(models)) + cpu, memory := GetMinCPUAndMemory(models[0].Model) + assert.Equal(t, cpu.Cmp(resource.MustParse("0.1")) == 0, true) + assert.Equal(t, memory.Cmp(resource.MustParse("20Mi")) == 0, true) +} diff --git a/internal/cli/cmd/class/class.go b/internal/cli/cmd/class/class.go new file mode 100644 index 000000000..4b10d796f --- /dev/null +++ b/internal/cli/cmd/class/class.go @@ -0,0 +1,41 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "github.com/spf13/cobra" + "k8s.io/cli-runtime/pkg/genericclioptions" + cmdutil "k8s.io/kubectl/pkg/cmd/util" +) + +const ( + CustomClassNamespace = "kube-system" + CMDataKeyDefinition = "definition" +) + +func NewClassCommand(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { + cmd := &cobra.Command{ + Use: "class", + Short: "Manage classes", + } + + cmd.AddCommand(NewCreateCommand(f, streams)) + cmd.AddCommand(NewListCommand(f, streams)) + cmd.AddCommand(NewTemplateCmd(streams)) + + return cmd +} diff --git a/internal/cli/cmd/class/class_test.go b/internal/cli/cmd/class/class_test.go new file mode 100644 index 000000000..58544ac58 --- /dev/null +++ b/internal/cli/cmd/class/class_test.go @@ -0,0 +1,44 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "k8s.io/cli-runtime/pkg/genericclioptions" + cmdtesting "k8s.io/kubectl/pkg/cmd/testing" +) + +var _ = Describe("class", func() { + var streams genericclioptions.IOStreams + var tf *cmdtesting.TestFactory + + BeforeEach(func() { + streams, _, _, _ = genericclioptions.NewTestIOStreams() + tf = cmdtesting.NewTestFactory() + }) + + AfterEach(func() { + tf.Cleanup() + }) + + It("command should succeed", func() { + cmd := NewClassCommand(tf, streams) + Expect(cmd).ShouldNot(BeNil()) + }) +}) diff --git a/internal/cli/cmd/class/create.go b/internal/cli/cmd/class/create.go new file mode 100644 index 000000000..1e809d7a0 --- /dev/null +++ b/internal/cli/cmd/class/create.go @@ -0,0 +1,259 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "context" + "fmt" + "os" + "strings" + + "github.com/ghodss/yaml" + "github.com/spf13/cobra" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" + cmdutil "k8s.io/kubectl/pkg/cmd/util" + "k8s.io/kubectl/pkg/util/templates" + + "github.com/apecloud/kubeblocks/internal/class" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/cli/util" + "github.com/apecloud/kubeblocks/internal/constant" +) + +type CreateOptions struct { + genericclioptions.IOStreams + + Factory cmdutil.Factory + client kubernetes.Interface + dynamic dynamic.Interface + ClusterDefRef string + ClassFamily string + ComponentType string + ClassName string + CPU string + Memory string + Storage []string + File string +} + +var classCreateExamples = templates.Examples(` + # Create a class following class family kubeblocks-general-classes for component mysql in cluster definition apecloud-mysql, which have 1 cpu core, 2Gi memory and storage is 10Gi + kbcli class create custom-1c2g --cluster-definition apecloud-mysql --type mysql --class-family kubeblocks-general-classes --cpu 1 --memory 2Gi --storage name=data,size=10Gi + + # Create classes for component mysql in cluster definition apecloud-mysql, where classes is defined in file + kbcli class create --cluster-definition apecloud-mysql --type mysql --file ./classes.yaml +`) + +func NewCreateCommand(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { + o := CreateOptions{IOStreams: streams} + cmd := &cobra.Command{ + Use: "create [NAME]", + Short: "Create a class", + Example: classCreateExamples, + Run: func(cmd *cobra.Command, args []string) { + util.CheckErr(o.complete(f)) + util.CheckErr(o.validate(args)) + util.CheckErr(o.run()) + }, + } + cmd.Flags().StringVar(&o.ClusterDefRef, "cluster-definition", "", "Specify cluster definition, run \"kbcli cluster-definition list\" to show all available cluster definition") + util.CheckErr(cmd.MarkFlagRequired("cluster-definition")) + cmd.Flags().StringVar(&o.ComponentType, "type", "", "Specify component type") + util.CheckErr(cmd.MarkFlagRequired("type")) + + cmd.Flags().StringVar(&o.ClassFamily, "class-family", "", "Specify class family") + cmd.Flags().StringVar(&o.CPU, corev1.ResourceCPU.String(), "", "Specify component cpu cores") + cmd.Flags().StringVar(&o.Memory, corev1.ResourceMemory.String(), "", "Specify component memory size") + cmd.Flags().StringArrayVar(&o.Storage, corev1.ResourceStorage.String(), []string{}, "Specify component storage disks") + + cmd.Flags().StringVar(&o.File, "file", "", "Specify file path which contains YAML definition of class") + + return cmd +} + +func (o *CreateOptions) validate(args []string) error { + // just validate creating by resource arguments + if o.File != "" { + return nil + } + + // validate cpu and memory + if _, err := resource.ParseQuantity(o.CPU); err != nil { + return err + } + if _, err := resource.ParseQuantity(o.Memory); err != nil { + return err + } + + // validate class name + if len(args) == 0 { + return fmt.Errorf("missing class name") + } + o.ClassName = args[0] + + return nil +} + +func (o *CreateOptions) complete(f cmdutil.Factory) error { + var err error + if o.client, err = f.KubernetesClientSet(); err != nil { + return err + } + if o.dynamic, err = f.DynamicClient(); err != nil { + return err + } + return nil +} + +func (o *CreateOptions) run() error { + componentClasses, err := class.GetClasses(o.client, o.ClusterDefRef) + if err != nil { + return err + } + + classes, ok := componentClasses[o.ComponentType] + if !ok { + classes = make(map[string]*class.ComponentClass) + } + + families, err := class.GetClassFamilies(o.dynamic) + if err != nil { + return err + } + + var ( + // new class definition version key + cmK = class.BuildClassDefinitionVersion() + // new class definition version value + cmV string + // newly created class names + classNames []string + ) + + if o.File != "" { + data, err := os.ReadFile(o.File) + if err != nil { + return err + } + newClasses, err := class.ParseComponentClasses(map[string]string{cmK: string(data)}) + if err != nil { + return err + } + for name, cls := range newClasses { + if _, ok = families[cls.Family]; !ok { + return fmt.Errorf("family %s is not found", cls.Family) + } + if _, ok = classes[name]; ok { + return fmt.Errorf("class name conflicted %s", name) + } + classNames = append(classNames, name) + } + cmV = string(data) + } else { + if _, ok = classes[o.ClassName]; ok { + return fmt.Errorf("class name conflicted %s", o.ClassName) + } + if _, ok = families[o.ClassFamily]; !ok { + return fmt.Errorf("family %s is not found", o.ClassFamily) + } + def, err := o.buildClassFamilyDef() + if err != nil { + return err + } + data, err := yaml.Marshal([]*class.ComponentClassFamilyDef{def}) + if err != nil { + return err + } + cmV = string(data) + classNames = append(classNames, o.ClassName) + } + + cmName := class.GetCustomClassConfigMapName(o.ClusterDefRef, o.ComponentType) + cm, err := o.client.CoreV1().ConfigMaps(CustomClassNamespace).Get(context.TODO(), cmName, metav1.GetOptions{}) + if err != nil && !errors.IsNotFound(err) { + return err + } + + if err == nil { + cm.Data[cmK] = cmV + if _, err = o.client.CoreV1().ConfigMaps(cm.GetNamespace()).Update(context.TODO(), cm, metav1.UpdateOptions{}); err != nil { + return err + } + } else { + cm = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: class.GetCustomClassConfigMapName(o.ClusterDefRef, o.ComponentType), + Namespace: CustomClassNamespace, + Labels: map[string]string{ + constant.ClusterDefLabelKey: o.ClusterDefRef, + types.ClassProviderLabelKey: "user", + types.ClassLevelLabelKey: "component", + constant.KBAppComponentDefRefLabelKey: o.ComponentType, + }, + }, + Data: map[string]string{cmK: cmV}, + } + if _, err = o.client.CoreV1().ConfigMaps(CustomClassNamespace).Create(context.TODO(), cm, metav1.CreateOptions{}); err != nil { + return err + } + } + _, _ = fmt.Fprintf(o.Out, "Successfully created class [%s].", strings.Join(classNames, ",")) + return nil +} + +func (o *CreateOptions) buildClassFamilyDef() (*class.ComponentClassFamilyDef, error) { + clsDef := class.ComponentClassDef{Name: o.ClassName, CPU: o.CPU, Memory: o.Memory} + for _, disk := range o.Storage { + kvs := strings.Split(disk, ",") + def := class.DiskDef{} + for _, kv := range kvs { + parts := strings.Split(kv, "=") + if len(parts) != 2 { + return nil, fmt.Errorf("invalid storage disk: %s", disk) + } + switch parts[0] { + case "name": + def.Name = parts[1] + case "size": + def.Size = parts[1] + case "class": + def.Class = parts[1] + default: + return nil, fmt.Errorf("invalid storage disk: %s", disk) + } + } + // validate disk size + if _, err := resource.ParseQuantity(def.Size); err != nil { + return nil, fmt.Errorf("invalid disk size: %s", disk) + } + if def.Name == "" { + return nil, fmt.Errorf("invalid disk name: %s", disk) + } + clsDef.Storage = append(clsDef.Storage, def) + } + def := &class.ComponentClassFamilyDef{ + Family: o.ClassFamily, + Series: []class.ComponentClassSeriesDef{{Classes: []class.ComponentClassDef{clsDef}}}, + } + return def, nil +} diff --git a/internal/cli/cmd/class/create_test.go b/internal/cli/cmd/class/create_test.go new file mode 100644 index 000000000..f529487de --- /dev/null +++ b/internal/cli/cmd/class/create_test.go @@ -0,0 +1,146 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "bytes" + "net/http" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/cli-runtime/pkg/resource" + "k8s.io/client-go/kubernetes/scheme" + clientfake "k8s.io/client-go/rest/fake" + cmdtesting "k8s.io/kubectl/pkg/cmd/testing" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/testing" +) + +var _ = Describe("create", func() { + var ( + o *CreateOptions + cd *appsv1alpha1.ClusterDefinition + out *bytes.Buffer + tf *cmdtesting.TestFactory + streams genericclioptions.IOStreams + ) + + fillResources := func(o *CreateOptions, cpu string, memory string, storage []string) { + o.CPU = cpu + o.Memory = memory + o.Storage = storage + } + + BeforeEach(func() { + cd = testing.FakeClusterDef() + + streams, _, out, _ = genericclioptions.NewTestIOStreams() + tf = testing.NewTestFactory(namespace) + + codec := scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...) + httpResp := func(obj runtime.Object) *http.Response { + return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, obj)} + } + cms := testing.FakeComponentClassDef(cd, classDef) + + resources := map[string]runtime.Object{ + "/api/v1/configmaps": cms, + } + + tf.UnstructuredClient = &clientfake.RESTClient{ + GroupVersion: schema.GroupVersion{Group: "core", Version: "v1"}, + NegotiatedSerializer: resource.UnstructuredPlusDefaultContentConfig().NegotiatedSerializer, + Client: clientfake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { + if req.Method == "POST" { + return httpResp(&corev1.ConfigMap{}), nil + } + resource, ok := resources[req.URL.Path] + if !ok { + return nil, errors.NewNotFound(schema.GroupResource{}, req.URL.Path) + } + return httpResp(resource), nil + }), + } + tf.Client = tf.UnstructuredClient + tf.FakeDynamicClient = testing.FakeDynamicClient(&generalClassFamily, &memoryOptimizedClassFamily, cd) + + o = &CreateOptions{ + Factory: tf, + IOStreams: streams, + ClusterDefRef: cd.Name, + ComponentType: testing.ComponentDefName, + } + Expect(o.complete(tf)).ShouldNot(HaveOccurred()) + }) + + AfterEach(func() { + tf.Cleanup() + }) + + It("should succeed to new command", func() { + cmd := NewCreateCommand(tf, streams) + Expect(cmd).ShouldNot(BeNil()) + }) + + Context("with resource arguments", func() { + + It("should fail if required arguments is missing", func() { + o.ClassFamily = generalClassFamily.Name + fillResources(o, "", "48Gi", nil) + Expect(o.validate([]string{"general-12c48g"})).Should(HaveOccurred()) + fillResources(o, "12", "", nil) + Expect(o.validate([]string{"general-12c48g"})).Should(HaveOccurred()) + fillResources(o, "12", "48g", nil) + Expect(o.validate([]string{})).Should(HaveOccurred()) + }) + + It("should succeed with required arguments", func() { + o.ClassFamily = generalClassFamily.Name + fillResources(o, "12", "48Gi", []string{"name=data,size=10Gi", "name=log,size=1Gi"}) + Expect(o.validate([]string{"general-12c48g"})).ShouldNot(HaveOccurred()) + Expect(o.run()).ShouldNot(HaveOccurred()) + Expect(out.String()).Should(ContainSubstring(o.ClassName)) + }) + + It("should fail if class name is conflicted", func() { + o.ClassName = "general-1c1g" + fillResources(o, "1", "1Gi", []string{"name=data,size=10Gi", "name=log,size=1Gi"}) + Expect(o.run()).Should(HaveOccurred()) + }) + }) + + Context("with class definitions file", func() { + It("should succeed", func() { + o.File = testCustomClassDefsPath + Expect(o.run()).ShouldNot(HaveOccurred()) + Expect(out.String()).Should(ContainSubstring("custom-1c1g")) + Expect(out.String()).Should(ContainSubstring("custom-200c400g")) + // memory optimized classes + Expect(out.String()).Should(ContainSubstring("custom-1c32g")) + Expect(out.String()).Should(ContainSubstring("custom-2c64g")) + }) + + }) + +}) diff --git a/internal/cli/cmd/class/list.go b/internal/cli/cmd/class/list.go new file mode 100644 index 000000000..aae922ea2 --- /dev/null +++ b/internal/cli/cmd/class/list.go @@ -0,0 +1,113 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "fmt" + "sort" + "strings" + + "github.com/spf13/cobra" + "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/kubernetes" + cmdutil "k8s.io/kubectl/pkg/cmd/util" + "k8s.io/kubectl/pkg/util/templates" + + "github.com/apecloud/kubeblocks/internal/class" + "github.com/apecloud/kubeblocks/internal/cli/printer" + "github.com/apecloud/kubeblocks/internal/cli/util" +) + +type ListOptions struct { + ClusterDefRef string + Factory cmdutil.Factory + client *kubernetes.Clientset + genericclioptions.IOStreams +} + +var listClassExamples = templates.Examples(` + # List all components classes in cluster definition apecloud-mysql + kbcli class list --cluster-definition apecloud-mysql +`) + +func NewListCommand(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { + o := &ListOptions{IOStreams: streams} + cmd := &cobra.Command{ + Use: "list", + Short: "List classes", + Example: listClassExamples, + Run: func(cmd *cobra.Command, args []string) { + util.CheckErr(o.complete(f)) + util.CheckErr(o.run()) + }, + } + cmd.Flags().StringVar(&o.ClusterDefRef, "cluster-definition", "", "Specify cluster definition, run \"kbcli cluster-definition list\" to show all available cluster definition") + util.CheckErr(cmd.MarkFlagRequired("cluster-definition")) + return cmd +} + +func (o *ListOptions) complete(f cmdutil.Factory) error { + var err error + o.client, err = f.KubernetesClientSet() + if err != nil { + return err + } + return err +} + +func (o *ListOptions) run() error { + componentClasses, err := class.GetClasses(o.client, o.ClusterDefRef) + if err != nil { + return err + } + familyClassMap := make(map[string]map[string][]*class.ComponentClass) + for compName, items := range componentClasses { + for _, item := range items { + if _, ok := familyClassMap[item.Family]; !ok { + familyClassMap[item.Family] = make(map[string][]*class.ComponentClass) + } + familyClassMap[item.Family][compName] = append(familyClassMap[item.Family][compName], item) + } + } + var familyNames []string + for name := range familyClassMap { + familyNames = append(familyNames, name) + } + sort.Strings(familyNames) + for _, family := range familyNames { + for compName, classes := range familyClassMap[family] { + o.printClassFamily(family, compName, classes) + } + _, _ = fmt.Fprint(o.Out, "\n") + } + return nil +} + +func (o *ListOptions) printClassFamily(family string, compName string, classes []*class.ComponentClass) { + tbl := printer.NewTablePrinter(o.Out) + _, _ = fmt.Fprintf(o.Out, "\nFamily %s:\n", family) + tbl.SetHeader("COMPONENT", "CLASS", "CPU", "MEMORY", "STORAGE") + sort.Sort(class.ByClassCPUAndMemory(classes)) + for _, class := range classes { + var disks []string + for _, disk := range class.Storage { + disks = append(disks, disk.String()) + } + tbl.AddRow(compName, class.Name, class.CPU.String(), class.Memory.String(), strings.Join(disks, ",")) + } + tbl.Print() +} diff --git a/internal/cli/cmd/class/list_test.go b/internal/cli/cmd/class/list_test.go new file mode 100644 index 000000000..c872514bf --- /dev/null +++ b/internal/cli/cmd/class/list_test.go @@ -0,0 +1,84 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "bytes" + "net/http" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/cli-runtime/pkg/resource" + "k8s.io/client-go/kubernetes/scheme" + clientfake "k8s.io/client-go/rest/fake" + cmdtesting "k8s.io/kubectl/pkg/cmd/testing" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/testing" +) + +var _ = Describe("list", func() { + var ( + cd *appsv1alpha1.ClusterDefinition + out *bytes.Buffer + tf *cmdtesting.TestFactory + streams genericclioptions.IOStreams + ) + + BeforeEach(func() { + cd = testing.FakeClusterDef() + + streams, _, out, _ = genericclioptions.NewTestIOStreams() + tf = testing.NewTestFactory(namespace) + + _ = corev1.AddToScheme(scheme.Scheme) + codec := scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...) + httpResp := func(obj runtime.Object) *http.Response { + return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, obj)} + } + + tf.UnstructuredClient = &clientfake.RESTClient{ + GroupVersion: schema.GroupVersion{Group: "core", Version: "v1"}, + NegotiatedSerializer: resource.UnstructuredPlusDefaultContentConfig().NegotiatedSerializer, + Client: clientfake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { + return map[string]*http.Response{ + "/api/v1/configmaps": httpResp(testing.FakeComponentClassDef(cd, classDef)), + }[req.URL.Path], nil + }), + } + tf.Client = tf.UnstructuredClient + }) + + AfterEach(func() { + tf.Cleanup() + }) + + It("should succeed", func() { + cmd := NewListCommand(tf, streams) + Expect(cmd).ShouldNot(BeNil()) + cmd.Run(cmd, []string{"--cluster-definition", cd.GetName()}) + Expect(out.String()).To(ContainSubstring("general-1c1g")) + Expect(out.String()).To(ContainSubstring(testing.ComponentDefName)) + Expect(out.String()).To(ContainSubstring(generalClassFamily.Name)) + Expect(out.String()).To(ContainSubstring(memoryOptimizedClassFamily.Name)) + }) +}) diff --git a/internal/cli/cmd/class/suite_test.go b/internal/cli/cmd/class/suite_test.go new file mode 100644 index 000000000..ef9544a90 --- /dev/null +++ b/internal/cli/cmd/class/suite_test.go @@ -0,0 +1,75 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "os" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/util/yaml" + "k8s.io/client-go/kubernetes/scheme" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" +) + +const ( + namespace = "test" + testDefaultClassDefsPath = "../../testing/testdata/class.yaml" + testCustomClassDefsPath = "../../testing/testdata/custom_class.yaml" + testGeneralClassFamilyPath = "../../testing/testdata/classfamily-general.yaml" + testMemoryOptimizedClassFamilyPath = "../../testing/testdata/classfamily-memory-optimized.yaml" +) + +var ( + classDef []byte + generalFamilyDef []byte + memoryOptimizedFamilyDef []byte + generalClassFamily appsv1alpha1.ClassFamily + memoryOptimizedClassFamily appsv1alpha1.ClassFamily +) + +var _ = BeforeSuite(func() { + var err error + + classDef, err = os.ReadFile(testDefaultClassDefsPath) + Expect(err).ShouldNot(HaveOccurred()) + + generalFamilyDef, err = os.ReadFile(testGeneralClassFamilyPath) + Expect(err).ShouldNot(HaveOccurred()) + err = yaml.Unmarshal(generalFamilyDef, &generalClassFamily) + Expect(err).ShouldNot(HaveOccurred()) + + memoryOptimizedFamilyDef, err = os.ReadFile(testMemoryOptimizedClassFamilyPath) + Expect(err).ShouldNot(HaveOccurred()) + err = yaml.Unmarshal(memoryOptimizedFamilyDef, &memoryOptimizedClassFamily) + Expect(err).ShouldNot(HaveOccurred()) + + err = appsv1alpha1.AddToScheme(scheme.Scheme) + Expect(err).ShouldNot(HaveOccurred()) + + err = corev1.AddToScheme(scheme.Scheme) + Expect(err).ShouldNot(HaveOccurred()) + +}) + +func TestClass(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Class Suite") +} diff --git a/internal/cli/cmd/class/template.go b/internal/cli/cmd/class/template.go new file mode 100644 index 000000000..3b25e59aa --- /dev/null +++ b/internal/cli/cmd/class/template.go @@ -0,0 +1,82 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "fmt" + "os" + + "github.com/spf13/cobra" + "k8s.io/cli-runtime/pkg/genericclioptions" + + "github.com/apecloud/kubeblocks/internal/cli/util" +) + +const ComponentClassTemplate = ` +- # class family name, such as general, memory-optimized, cpu-optimized etc. + family: kb-class-family-general + # class schema template, you can set default resource values here + template: | + cpu: "{{ or .cpu 1 }}" + memory: "{{ or .memory 4 }}Gi" + storage: + - name: data + size: "{{ or .dataStorageSize 10 }}Gi" + - name: log + size: "{{ or .logStorageSize 1 }}Gi" + # class schema template variables + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - # class name generator, you can reference variables in class schema template + # it's also ok to define static class name in following class definitions + name: "custom-{{ .cpu }}c{{ .memory }}g" + + # class definitions, we support two kinds of class definitions: + # 1. define arguments for class schema variables, class schema will be dynamically generated + # 2. statically define complete class schema + classes: + # arguments for dynamically generated class + - args: [1, 1, 100, 10] +` + +type TemplateOptions struct { + genericclioptions.IOStreams + + outputFile string +} + +func NewTemplateCmd(streams genericclioptions.IOStreams) *cobra.Command { + o := &TemplateOptions{IOStreams: streams} + cmd := &cobra.Command{ + Use: "template", + Short: "Generate class definition template", + Run: func(cmd *cobra.Command, args []string) { + util.CheckErr(o.run()) + }, + } + cmd.Flags().StringVarP(&o.outputFile, "output", "o", "", "Output class definition template to a file") + return cmd +} + +func (o *TemplateOptions) run() error { + if o.outputFile != "" { + return os.WriteFile(o.outputFile, []byte(ComponentClassTemplate), 0644) + } + + _, err := fmt.Fprint(o.Out, ComponentClassTemplate) + return err +} diff --git a/internal/cli/cmd/class/template_test.go b/internal/cli/cmd/class/template_test.go new file mode 100644 index 000000000..e2a239d3d --- /dev/null +++ b/internal/cli/cmd/class/template_test.go @@ -0,0 +1,45 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package class + +import ( + "bytes" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "k8s.io/cli-runtime/pkg/genericclioptions" +) + +var _ = Describe("template", func() { + var ( + out *bytes.Buffer + streams genericclioptions.IOStreams + ) + + BeforeEach(func() { + streams, _, out, _ = genericclioptions.NewTestIOStreams() + }) + + It("command should succeed", func() { + cmd := NewTemplateCmd(streams) + Expect(cmd).ShouldNot(BeNil()) + + cmd.Run(cmd, []string{}) + Expect(out.String()).ShouldNot(BeEmpty()) + }) +}) diff --git a/internal/cli/cmd/cli.go b/internal/cli/cmd/cli.go index bdd643722..914337911 100644 --- a/internal/cli/cmd/cli.go +++ b/internal/cli/cmd/cli.go @@ -32,6 +32,7 @@ import ( "github.com/apecloud/kubeblocks/internal/cli/cmd/alert" "github.com/apecloud/kubeblocks/internal/cli/cmd/backupconfig" "github.com/apecloud/kubeblocks/internal/cli/cmd/bench" + "github.com/apecloud/kubeblocks/internal/cli/cmd/class" "github.com/apecloud/kubeblocks/internal/cli/cmd/cluster" "github.com/apecloud/kubeblocks/internal/cli/cmd/clusterdefinition" "github.com/apecloud/kubeblocks/internal/cli/cmd/clusterversion" @@ -98,6 +99,7 @@ A Command Line Interface for KubeBlocks`, dashboard.NewDashboardCmd(f, ioStreams), clusterversion.NewClusterVersionCmd(f, ioStreams), clusterdefinition.NewClusterDefinitionCmd(f, ioStreams), + class.NewClassCommand(f, ioStreams), alert.NewAlertCmd(f, ioStreams), addon.NewAddonCmd(f, ioStreams), ) diff --git a/internal/cli/cmd/cluster/cluster_test.go b/internal/cli/cmd/cluster/cluster_test.go index f79acac2b..c43f6fa4c 100644 --- a/internal/cli/cmd/cluster/cluster_test.go +++ b/internal/cli/cmd/cluster/cluster_test.go @@ -17,6 +17,8 @@ limitations under the License. package cluster import ( + "os" + . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" "k8s.io/cli-runtime/pkg/genericclioptions" @@ -31,6 +33,7 @@ import ( var _ = Describe("Cluster", func() { const testComponentPath = "../../testing/testdata/component.yaml" + const testClassDefsPath = "../../testing/testdata/class.yaml" var streams genericclioptions.IOStreams var tf *cmdtesting.TestFactory @@ -38,7 +41,8 @@ var _ = Describe("Cluster", func() { BeforeEach(func() { streams, _, _, _ = genericclioptions.NewTestIOStreams() tf = cmdtesting.NewTestFactory().WithNamespace("default") - tf.FakeDynamicClient = testing.FakeDynamicClient(testing.FakeClusterDef(), testing.FakeClusterVersion()) + cd := testing.FakeClusterDef() + tf.FakeDynamicClient = testing.FakeDynamicClient(cd, testing.FakeClusterVersion()) tf.Client = &clientfake.RESTClient{} }) @@ -77,9 +81,13 @@ var _ = Describe("Cluster", func() { }) It("run", func() { - tf.FakeDynamicClient = testing.FakeDynamicClient(testing.FakeClusterDef()) + clusterDef := testing.FakeClusterDef() + tf.FakeDynamicClient = testing.FakeDynamicClient(clusterDef) + data, err := os.ReadFile(testClassDefsPath) + Expect(err).NotTo(HaveOccurred()) + clientSet := testing.FakeClientSet(testing.FakeComponentClassDef(clusterDef, data)) o := &CreateOptions{ - BaseOptions: create.BaseOptions{IOStreams: streams, Name: "test", Dynamic: tf.FakeDynamicClient}, + BaseOptions: create.BaseOptions{IOStreams: streams, Name: "test", Dynamic: tf.FakeDynamicClient, ClientSet: clientSet}, SetFile: "", ClusterDefRef: testing.ClusterDefName, ClusterVersionRef: "cluster-version", diff --git a/internal/cli/cmd/cluster/create.go b/internal/cli/cmd/cluster/create.go index 9dbe343eb..eca3ddf03 100644 --- a/internal/cli/cmd/cluster/create.go +++ b/internal/cli/cmd/cluster/create.go @@ -76,6 +76,9 @@ var clusterCreateExample = templates.Examples(` # Create a cluster and set cpu to 1 core, memory to 1Gi, storage size to 20Gi and replicas to 3 kbcli cluster create mycluster --cluster-definition apecloud-mysql --set cpu=1,memory=1Gi,storage=20Gi,replicas=3 + # Create a cluster and set class to general-1c4g + kbcli cluster create myclsuter --cluster-definition apecloud-mysql --set class=general-1c4g + # Create a cluster and use a URL to set cluster resource kbcli cluster create mycluster --cluster-definition apecloud-mysql --set-file https://kubeblocks.io/yamls/my.yaml @@ -106,6 +109,7 @@ type setKey string const ( keyType setKey = "type" keyCPU setKey = "cpu" + keyClass setKey = "class" keyMemory setKey = "memory" keyReplicas setKey = "replicas" keyStorage setKey = "storage" @@ -292,13 +296,49 @@ func (o *CreateOptions) buildComponents() ([]map[string]interface{}, error) { return nil, err } - if components, err = buildClusterComp(cd, compSets); err != nil { + componentObjs, err := buildClusterComp(cd, compSets) + if err != nil { + return nil, err + } + for _, compObj := range componentObjs { + comp, err := runtime.DefaultUnstructuredConverter.ToUnstructured(compObj) + if err != nil { + return nil, err + } + components = append(components, comp) + } + + if err = o.buildClassMappings(componentObjs, compSets); err != nil { return nil, err } } return components, nil } +func (o *CreateOptions) buildClassMappings(components []*appsv1alpha1.ClusterComponentSpec, setsMap map[string]map[setKey]string) error { + classMappings := make(map[string]string) + for _, comp := range components { + sets, ok := setsMap[comp.ComponentDefRef] + if !ok { + continue + } + class, ok := sets[keyClass] + if !ok { + continue + } + classMappings[comp.Name] = class + } + bytes, err := json.Marshal(classMappings) + if err != nil { + return err + } + if o.Annotations == nil { + o.Annotations = make(map[string]string) + } + o.Annotations[types.ComponentClassAnnotationKey] = string(bytes) + return nil +} + // MultipleSourceComponents get component data from multiple source, such as stdin, URI and local file func MultipleSourceComponents(fileName string, in io.Reader) ([]byte, error) { var data io.Reader @@ -411,7 +451,7 @@ func setEnableAllLogs(c *appsv1alpha1.Cluster, cd *appsv1alpha1.ClusterDefinitio } } -func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map[setKey]string) ([]map[string]interface{}, error) { +func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map[setKey]string) ([]*appsv1alpha1.ClusterComponentSpec, error) { getVal := func(key setKey, sets map[setKey]string) string { // get value from set values if sets != nil { @@ -429,7 +469,7 @@ func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map return val } - var comps []map[string]interface{} + var comps []*appsv1alpha1.ClusterComponentSpec for _, c := range cd.Spec.ComponentDefs { sets := map[setKey]string{} @@ -470,11 +510,7 @@ func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map }, }}, } - comp, err := runtime.DefaultUnstructuredConverter.ToUnstructured(compObj) - if err != nil { - return nil, err - } - comps = append(comps, comp) + comps = append(comps, compObj) } return comps, nil } @@ -483,7 +519,7 @@ func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map // specified in the set, use the cluster definition default component name. func buildCompSetsMap(values []string, cd *appsv1alpha1.ClusterDefinition) (map[string]map[setKey]string, error) { allSets := map[string]map[setKey]string{} - keys := []string{string(keyCPU), string(keyType), string(keyStorage), string(keyMemory), string(keyReplicas)} + keys := []string{string(keyCPU), string(keyType), string(keyStorage), string(keyMemory), string(keyReplicas), string(keyClass)} parseKey := func(key string) setKey { for _, k := range keys { if strings.EqualFold(k, key) { diff --git a/internal/cli/cmd/cluster/create_test.go b/internal/cli/cmd/cluster/create_test.go index 004b85200..d2916ec2b 100644 --- a/internal/cli/cmd/cluster/create_test.go +++ b/internal/cli/cmd/cluster/create_test.go @@ -28,7 +28,6 @@ import ( "github.com/spf13/viper" corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/json" "k8s.io/cli-runtime/pkg/genericclioptions" @@ -134,12 +133,11 @@ var _ = Describe("create", func() { }) }) - checkComponent := func(comps []map[string]interface{}, storage string, replicas int32, cpu string, memory string) { + checkComponent := func(comps []*appsv1alpha1.ClusterComponentSpec, storage string, replicas int32, cpu string, memory string) { Expect(comps).ShouldNot(BeNil()) Expect(len(comps)).Should(Equal(2)) - comp := &appsv1alpha1.ClusterComponentSpec{} - _ = runtime.DefaultUnstructuredConverter.FromUnstructured(comps[0], comp) + comp := comps[0] Expect(getResource(comp.VolumeClaimTemplates[0].Spec.Resources, corev1.ResourceStorage)).Should(Equal(storage)) Expect(comp.Replicas).Should(BeEquivalentTo(replicas)) @@ -222,13 +220,14 @@ var _ = Describe("create", func() { true, }, { - []string{"cpu=1,memory=2Gi,storage=10Gi"}, + []string{"cpu=1,memory=2Gi,storage=10Gi,class=general-1c2g"}, []string{"my-comp"}, map[string]map[setKey]string{ "my-comp": { keyCPU: "1", keyMemory: "2Gi", keyStorage: "10Gi", + keyClass: "general-1c2g", }, }, true, @@ -287,18 +286,20 @@ var _ = Describe("create", func() { true, }, { - []string{"type=comp1,cpu=1,memory=2Gi", "type=comp2,storage=10Gi,cpu=2"}, + []string{"type=comp1,cpu=1,memory=2Gi,class=general-2c4g", "type=comp2,storage=10Gi,cpu=2,class=mo-1c8g"}, []string{"my-comp"}, map[string]map[setKey]string{ "comp1": { keyType: "comp1", keyCPU: "1", keyMemory: "2Gi", + keyClass: "general-2c4g", }, "comp2": { keyType: "comp2", keyCPU: "2", keyStorage: "10Gi", + keyClass: "mo-1c8g", }, }, true, diff --git a/internal/cli/cmd/cluster/dataprotection_test.go b/internal/cli/cmd/cluster/dataprotection_test.go index ed43a80ab..34f2e0430 100644 --- a/internal/cli/cmd/cluster/dataprotection_test.go +++ b/internal/cli/cmd/cluster/dataprotection_test.go @@ -88,7 +88,8 @@ var _ = Describe("DataProtection", func() { secrets := testing.FakeSecrets(testing.Namespace, testing.ClusterName) pods := testing.FakePods(1, testing.Namespace, testing.ClusterName) tf.FakeDynamicClient = fake.NewSimpleDynamicClient( - scheme.Scheme, &secrets.Items[0], cluster, template, &pods.Items[0]) + scheme.Scheme, &secrets.Items[0], cluster, clusterDef, template, &pods.Items[0]) + tf.Client = &clientfake.RESTClient{} cmd := NewCreateBackupCmd(tf, streams) Expect(cmd).ShouldNot(BeNil()) // must succeed otherwise exit 1 and make test fails @@ -192,6 +193,9 @@ var _ = Describe("DataProtection", func() { pods := testing.FakePods(1, testing.Namespace, clusterName) tf.FakeDynamicClient = fake.NewSimpleDynamicClient( scheme.Scheme, &secrets.Items[0], &pods.Items[0], cluster, template) + tf.FakeDynamicClient = fake.NewSimpleDynamicClient( + scheme.Scheme, &secrets.Items[0], &pods.Items[0], clusterDef, cluster, template) + tf.Client = &clientfake.RESTClient{} // create backup cmd := NewCreateBackupCmd(tf, streams) Expect(cmd).ShouldNot(BeNil()) diff --git a/internal/cli/cmd/cluster/describe_test.go b/internal/cli/cmd/cluster/describe_test.go index 61cdabb34..95cc3dacd 100644 --- a/internal/cli/cmd/cluster/describe_test.go +++ b/internal/cli/cmd/cluster/describe_test.go @@ -48,7 +48,6 @@ var _ = Describe("Expose", func() { cluster = testing.FakeCluster(clusterName, namespace) pods = testing.FakePods(3, namespace, clusterName) ) - BeforeEach(func() { streams, _, _, _ = genericclioptions.NewTestIOStreams() tf = testing.NewTestFactory(namespace) diff --git a/internal/cli/cmd/cluster/operations.go b/internal/cli/cmd/cluster/operations.go index e1d15d025..4f493add6 100644 --- a/internal/cli/cmd/cluster/operations.go +++ b/internal/cli/cmd/cluster/operations.go @@ -63,6 +63,7 @@ type OperationsOptions struct { // VerticalScaling options CPU string `json:"cpu"` Memory string `json:"memory"` + Class string `json:"class"` // HorizontalScaling options Replicas int `json:"replicas"` @@ -640,6 +641,7 @@ func NewVerticalScalingCmd(f cmdutil.Factory, streams genericclioptions.IOStream o.buildCommonFlags(cmd) cmd.Flags().StringVar(&o.CPU, "cpu", "", "Requested and limited size of component cpu") cmd.Flags().StringVar(&o.Memory, "memory", "", "Requested and limited size of component memory") + cmd.Flags().StringVar(&o.Class, "class", "", "Component class") } return create.BuildCommand(inputs) } diff --git a/internal/cli/create/create.go b/internal/cli/create/create.go index d7323c5bb..480492770 100644 --- a/internal/cli/create/create.go +++ b/internal/cli/create/create.go @@ -111,6 +111,8 @@ type BaseOptions struct { // Quiet minimize unnecessary output Quiet bool + ClientSet kubernetes.Interface + genericclioptions.IOStreams } @@ -151,6 +153,10 @@ func (o *BaseOptions) Complete(inputs Inputs, args []string) error { return err } + if o.ClientSet, err = inputs.Factory.KubernetesClientSet(); err != nil { + return err + } + // do custom options complete if inputs.Complete != nil { if err = inputs.Complete(); err != nil { diff --git a/internal/cli/testing/fake.go b/internal/cli/testing/fake.go index 788b70b21..099342c50 100644 --- a/internal/cli/testing/fake.go +++ b/internal/cli/testing/fake.go @@ -35,16 +35,18 @@ import ( ) const ( - ClusterName = "fake-cluster-name" - Namespace = "fake-namespace" - ClusterVersionName = "fake-cluster-version" - ClusterDefName = "fake-cluster-definition" - ComponentName = "fake-component-name" - ComponentDefName = "fake-component-type" - NodeName = "fake-node-name" - SecretName = "fake-secret-conn-credential" - StorageClassName = "fake-storage-class" - PVCName = "fake-pvc" + ClusterName = "fake-cluster-name" + Namespace = "fake-namespace" + ClusterVersionName = "fake-cluster-version" + ClusterDefName = "fake-cluster-definition" + ComponentName = "fake-component-name" + ComponentDefName = "fake-component-type" + NodeName = "fake-node-name" + SecretName = "fake-secret-conn-credential" + StorageClassName = "fake-storage-class" + PVCName = "fake-pvc" + GeneralClassFamily = "kb-class-family-general" + MemoryOptimizedClassFamily = "kb-class-family-memory-optimized" KubeBlocksRepoName = "fake-kubeblocks-repo" KubeBlocksChartName = "fake-kubeblocks" @@ -252,6 +254,21 @@ func FakeClusterDef() *appsv1alpha1.ClusterDefinition { return clusterDef } +func FakeComponentClassDef(clusterDef *appsv1alpha1.ClusterDefinition, def []byte) *corev1.ConfigMapList { + result := &corev1.ConfigMapList{} + cm := &corev1.ConfigMap{} + cm.Name = fmt.Sprintf("fake-kubeblocks-classes-%s", ComponentName) + cm.SetLabels(map[string]string{ + types.ClassLevelLabelKey: "component", + constant.KBAppComponentDefRefLabelKey: ComponentDefName, + types.ClassProviderLabelKey: "kubeblocks", + constant.ClusterDefLabelKey: clusterDef.Name, + }) + cm.Data = map[string]string{"families-20230223162700": string(def)} + result.Items = append(result.Items, *cm) + return result +} + func FakeClusterVersion() *appsv1alpha1.ClusterVersion { cv := &appsv1alpha1.ClusterVersion{} cv.Name = ClusterVersionName diff --git a/internal/cli/testing/testdata/class.yaml b/internal/cli/testing/testdata/class.yaml new file mode 100644 index 000000000..9e1c965fd --- /dev/null +++ b/internal/cli/testing/testdata/class.yaml @@ -0,0 +1,61 @@ + +- # class family name, such as general, memory-optimized, cpu-optimized etc. + family: kb-class-family-general + # class schema template, you can set default resource values here + template: | + cpu: "{{ or .cpu 1 }}" + memory: "{{ or .memory 4 }}Gi" + storage: + - name: data + size: "{{ or .dataStorageSize 10 }}Gi" + - name: log + size: "{{ or .logStorageSize 1 }}Gi" + # class schema template variables + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - # class name generator, you can reference variables in class schema template + # it's also ok to define static class name in following class definitions + name: "general-{{ .cpu }}c{{ .memory }}g" + + # class definitions, we support two kinds of class definitions: + # 1. define arguments for class schema variables, class schema will be dynamically generated + # 2. statically define complete class schema + classes: + - args: [1, 1, 100, 10] + - args: [2, 2, 100, 10] + - args: [2, 4, 100, 10] + - args: [2, 8, 100, 10] + - args: [4, 16, 100, 10] + - args: [8, 32, 100, 10] + - args: [16, 64, 200, 10] + - args: [32, 128, 200, 10] + - args: [64, 256, 200, 10] + - args: [128, 512, 200, 10] + +- family: kb-class-family-memory-optimized + template: | + cpu: "{{ or .cpu 1 }}" + memory: "{{ or .memory 8 }}Gi" + storage: + - name: data + size: "{{ or .dataStorageSize 10 }}Gi" + - name: log + size: "{{ or .logStorageSize 1 }}Gi" + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - name: "mo-{{ .cpu }}c{{ .memory }}g" + classes: + - args: [2, 16, 100, 10] + - args: [4, 32, 100, 10] + - args: [8, 64, 100, 10] + - args: [12, 96, 100, 10] + - args: [24, 192, 200, 10] + - args: [48, 384, 200, 10] + - args: [2, 32, 100, 10] + - args: [4, 64, 100, 10] + - args: [8, 128, 100, 10] + - args: [16, 256, 100, 10] + - args: [32, 512, 200, 10] + - args: [48, 768, 200, 10] + - args: [64, 1024, 200, 10] + - args: [128, 2048, 200, 10] diff --git a/internal/cli/testing/testdata/classfamily-general.yaml b/internal/cli/testing/testdata/classfamily-general.yaml new file mode 100644 index 000000000..0ba574d26 --- /dev/null +++ b/internal/cli/testing/testdata/classfamily-general.yaml @@ -0,0 +1,23 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClassFamily +metadata: + name: kb-class-family-general + labels: + classfamily.kubeblocks.io/provider: kubeblocks +spec: + models: + - cpu: + min: 0.5 + max: 2 + step: 0.5 + memory: + sizePerCPU: 1Gi + - cpu: + min: 2 + max: 2 + memory: + sizePerCPU: 2Gi + - cpu: + slots: [2, 4, 8, 16, 24, 32, 48, 64, 96, 128] + memory: + sizePerCPU: 4Gi \ No newline at end of file diff --git a/internal/cli/testing/testdata/classfamily-memory-optimized.yaml b/internal/cli/testing/testdata/classfamily-memory-optimized.yaml new file mode 100644 index 000000000..b02f488b9 --- /dev/null +++ b/internal/cli/testing/testdata/classfamily-memory-optimized.yaml @@ -0,0 +1,18 @@ +apiVersion: apps.kubeblocks.io/v1alpha1 +kind: ClassFamily +metadata: + name: kb-class-family-memory-optimized + labels: + classfamily.kubeblocks.io/provider: kubeblocks +spec: + models: + - cpu: + slots: [2, 4, 8, 12, 24, 48] + memory: + sizePerCPU: 8Gi + - cpu: + min: 2 + max: 128 + step: 2 + memory: + sizePerCPU: 16Gi \ No newline at end of file diff --git a/internal/cli/testing/testdata/custom_class.yaml b/internal/cli/testing/testdata/custom_class.yaml new file mode 100644 index 000000000..b51a653ea --- /dev/null +++ b/internal/cli/testing/testdata/custom_class.yaml @@ -0,0 +1,33 @@ +- family: kb-class-family-general + template: | + cpu: "{{ or .cpu 1 }}" + memory: "{{ or .memory 4 }}Gi" + storage: + - name: data + size: "{{ or .dataStorageSize 10 }}Gi" + - name: log + size: "{{ or .logStorageSize 1 }}Gi" + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - name: "custom-{{ .cpu }}c{{ .memory }}g" + classes: + - args: [1, 1, 100, 10] + - name: custom-200c400g + cpu: 200 + memory: 400Gi + +- family: kb-class-family-memory-optimized + template: | + cpu: "{{ or .cpu 1 }}" + memory: "{{ or .memory 4 }}Gi" + storage: + - name: data + size: "{{ or .dataStorageSize 10 }}Gi" + - name: log + size: "{{ or .logStorageSize 1 }}Gi" + vars: [cpu, memory, dataStorageSize, logStorageSize] + series: + - name: "custom-{{ .cpu }}c{{ .memory }}g" + classes: + - args: [1, 32, 100, 10] + - args: [2, 64, 100, 10] diff --git a/internal/cli/types/types.go b/internal/cli/types/types.go index 418c73a69..324700fef 100644 --- a/internal/cli/types/types.go +++ b/internal/cli/types/types.go @@ -72,6 +72,7 @@ const ( ResourceClusterVersions = "clusterversions" ResourceOpsRequests = "opsrequests" ResourceConfigConstraintVersions = "configconstraints" + ResourceClassFamily = "classfamilies" KindCluster = "Cluster" KindClusterDef = "ClusterDefinition" KindClusterVersion = "ClusterVersion" @@ -95,6 +96,11 @@ const ( ServiceHAVIPTypeAnnotationKey = "service.kubernetes.io/kubeblocks-havip-type" ServiceHAVIPTypeAnnotationValue = "private-ip" ServiceFloatingIPAnnotationKey = "service.kubernetes.io/kubeblocks-havip-floating-ip" + + ClassLevelLabelKey = "class.kubeblocks.io/level" + ClassProviderLabelKey = "class.kubeblocks.io/provider" + ClassFamilyProviderLabelKey = "classfamily.kubeblocks.io/provider" + ComponentClassAnnotationKey = "cluster.kubeblocks.io/component-class" ) // DataProtection API group @@ -196,6 +202,10 @@ func AddonGVR() schema.GroupVersionResource { return schema.GroupVersionResource{Group: ExtensionsAPIGroup, Version: ExtensionsAPIVersion, Resource: ResourceAddons} } +func ClassFamilyGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: AppsAPIGroup, Version: AppsAPIVersion, Resource: ResourceClassFamily} +} + func CRDGVR() schema.GroupVersionResource { return schema.GroupVersionResource{ Group: "apiextensions.k8s.io", diff --git a/internal/constant/const.go b/internal/constant/const.go index 417bec672..c10ae30d7 100644 --- a/internal/constant/const.go +++ b/internal/constant/const.go @@ -61,6 +61,7 @@ const ( // kubeblocks.io labels ClusterDefLabelKey = "clusterdefinition.kubeblocks.io/name" KBAppComponentLabelKey = "apps.kubeblocks.io/component-name" + KBAppComponentDefRefLabelKey = "apps.kubeblocks.io/component-def-ref" ConsensusSetAccessModeLabelKey = "cs.apps.kubeblocks.io/access-mode" AppConfigTypeLabelKey = "apps.kubeblocks.io/config-type" WorkloadTypeLabelKey = "apps.kubeblocks.io/workload-type" @@ -71,6 +72,7 @@ const ( ClusterAccountLabelKey = "account.kubeblocks.io/name" VolumeTypeLabelKey = "kubeblocks.io/volume-type" KBManagedByKey = "apps.kubeblocks.io/managed-by" // KBManagedByKey marks resources that auto created during operation + ClassProviderLabelKey = "class.kubeblocks.io/provider" // kubeblocks.io annotations OpsRequestAnnotationKey = "kubeblocks.io/ops-request" // OpsRequestAnnotationKey OpsRequest annotation key in Cluster @@ -110,6 +112,9 @@ const ( // configuration finalizer ConfigurationTemplateFinalizerName = "config.kubeblocks.io/finalizer" + + // ClassAnnotationKey is used to specify the class of components + ClassAnnotationKey = "cluster.kubeblocks.io/component-class" ) const ( diff --git a/internal/controller/lifecycle/cluster_plan_builder.go b/internal/controller/lifecycle/cluster_plan_builder.go index 5c37beaa6..14e40c12d 100644 --- a/internal/controller/lifecycle/cluster_plan_builder.go +++ b/internal/controller/lifecycle/cluster_plan_builder.go @@ -193,6 +193,8 @@ func (c *clusterPlanBuilder) Build() (graph.Plan, error) { chain := &graph.TransformerChain{ // init dag, that is put cluster vertex into dag &initTransformer{cluster: c.cluster, originCluster: &c.originCluster}, + // fill class related info + &fillClass{cc: *cr, cli: c.cli, ctx: c.ctx}, // fix cd&cv labels of cluster &fixClusterLabelsTransformer{}, // cluster to K8s objects and put them into dag diff --git a/internal/controller/lifecycle/transformer_fill_class.go b/internal/controller/lifecycle/transformer_fill_class.go new file mode 100644 index 000000000..73651e9d4 --- /dev/null +++ b/internal/controller/lifecycle/transformer_fill_class.go @@ -0,0 +1,157 @@ +package lifecycle + +import ( + "encoding/json" + "fmt" + "sort" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/class" + "github.com/apecloud/kubeblocks/internal/constant" + "github.com/apecloud/kubeblocks/internal/controller/graph" + intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" +) + +// fixClusterLabelsTransformer fill the class related info to cluster +type fillClass struct { + cc clusterRefResources + cli client.Client + ctx intctrlutil.RequestCtx +} + +func (r *fillClass) Transform(dag *graph.DAG) error { + rootVertex, err := findRootVertex(dag) + if err != nil { + return err + } + cluster, _ := rootVertex.obj.(*appsv1alpha1.Cluster) + return r.fillClass(r.ctx, cluster, r.cc.cd) +} + +func (r *fillClass) fillClass(reqCtx intctrlutil.RequestCtx, cluster *appsv1alpha1.Cluster, clusterDefinition appsv1alpha1.ClusterDefinition) error { + var ( + value = cluster.GetAnnotations()[constant.ClassAnnotationKey] + componentClassMapping = make(map[string]string) + cmList corev1.ConfigMapList + ) + if value != "" { + if err := json.Unmarshal([]byte(value), &componentClassMapping); err != nil { + return err + } + } + + cmLabels := []client.ListOption{ + client.MatchingLabels{constant.ClusterDefLabelKey: clusterDefinition.Name}, + client.HasLabels{constant.ClassProviderLabelKey}, + } + if err := r.cli.List(reqCtx.Ctx, &cmList, cmLabels...); err != nil { + return err + } + compClasses, err := class.ParseClasses(&cmList) + if err != nil { + return err + } + + var classFamilyList appsv1alpha1.ClassFamilyList + if err = r.cli.List(reqCtx.Ctx, &classFamilyList); err != nil { + return err + } + + // TODO use this function to get matched class families if class is not specified and component has no classes + _ = func(comp appsv1alpha1.ClusterComponentSpec) *class.ComponentClass { + var candidates []class.ClassModelWithFamilyName + for _, family := range classFamilyList.Items { + models := family.FindMatchingModels(&comp.Resources) + for _, model := range models { + candidates = append(candidates, class.ClassModelWithFamilyName{Family: family.Name, Model: model}) + } + } + if len(candidates) == 0 { + return nil + } + sort.Sort(class.ByModelList(candidates)) + candidate := candidates[0] + cpu, memory := class.GetMinCPUAndMemory(candidate.Model) + cls := &class.ComponentClass{ + Name: fmt.Sprintf("%s-%vc%vg", candidate.Family, cpu.AsDec().String(), memory.AsDec().String()), + CPU: *cpu, + Memory: *memory, + } + return cls + } + + matchComponentClass := func(comp appsv1alpha1.ClusterComponentSpec, classes map[string]*class.ComponentClass) *class.ComponentClass { + filters := class.Filters(make(map[string]resource.Quantity)) + if comp.Resources.Requests.Cpu() != nil { + filters[corev1.ResourceCPU.String()] = *comp.Resources.Requests.Cpu() + } + if comp.Resources.Requests.Memory() != nil { + filters[corev1.ResourceMemory.String()] = *comp.Resources.Requests.Memory() + } + return class.ChooseComponentClasses(classes, filters) + } + + for idx, comp := range cluster.Spec.ComponentSpecs { + classes := compClasses[comp.ComponentDefRef] + + var cls *class.ComponentClass + className, ok := componentClassMapping[comp.Name] + // TODO another case if len(classFamilyList.Items) > 0, use matchClassFamilies to find matching class family: + switch { + case ok: + cls = classes[className] + if cls == nil { + return fmt.Errorf("unknown component class %s", className) + } + case classes != nil: + cls = matchComponentClass(comp, classes) + if cls == nil { + return fmt.Errorf("can not find matching class for component %s", comp.Name) + } + } + if cls == nil { + // TODO reconsider handling policy for this case + continue + } + componentClassMapping[comp.Name] = cls.Name + requests := corev1.ResourceList{ + corev1.ResourceCPU: cls.CPU, + corev1.ResourceMemory: cls.Memory, + } + requests.DeepCopyInto(&comp.Resources.Requests) + requests.DeepCopyInto(&comp.Resources.Limits) + var volumes []appsv1alpha1.ClusterComponentVolumeClaimTemplate + if len(comp.VolumeClaimTemplates) > 0 { + volumes = comp.VolumeClaimTemplates + } else { + volumes = buildVolumeClaimByClass(cls) + } + comp.VolumeClaimTemplates = volumes + cluster.Spec.ComponentSpecs[idx] = comp + } + return nil +} + +func buildVolumeClaimByClass(cls *class.ComponentClass) []appsv1alpha1.ClusterComponentVolumeClaimTemplate { + var volumes []appsv1alpha1.ClusterComponentVolumeClaimTemplate + for _, disk := range cls.Storage { + volume := appsv1alpha1.ClusterComponentVolumeClaimTemplate{ + Name: disk.Name, + Spec: &corev1.PersistentVolumeClaimSpec{ + // TODO define access mode in class + AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceStorage: disk.Size, + }, + }, + }, + } + volumes = append(volumes, volume) + } + return volumes +} From 2ef3bfc885360a60da8e03b7bef21382705e1181 Mon Sep 17 00:00:00 2001 From: shaojiang Date: Fri, 31 Mar 2023 18:00:49 +0800 Subject: [PATCH 21/80] fix: remove useless yaml field and examples (#2363) --- Makefile | 2 - examples/dataprotection/backup_job.yaml | 28 ------- examples/dataprotection/backup_policy.yaml | 52 ------------ .../dataprotection/backup_remote_pvc.yaml | 11 --- examples/dataprotection/backup_tool.yaml | 84 ------------------- examples/dataprotection/restore_job.yaml | 38 --------- .../smoketest/postgresql/07_backuppolicy.yaml | 1 - .../smoketest/wesql/11_backuppolicy_full.yaml | 1 - 8 files changed, 217 deletions(-) delete mode 100644 examples/dataprotection/backup_job.yaml delete mode 100644 examples/dataprotection/backup_policy.yaml delete mode 100644 examples/dataprotection/backup_remote_pvc.yaml delete mode 100644 examples/dataprotection/backup_tool.yaml delete mode 100644 examples/dataprotection/restore_job.yaml diff --git a/Makefile b/Makefile index 7911f4fcc..809237e65 100644 --- a/Makefile +++ b/Makefile @@ -312,12 +312,10 @@ endif .PHONY: install install: manifests kustomize ## Install CRDs into the K8s cluster specified in ~/.kube/config. ($(KUSTOMIZE) build config/crd | kubectl replace -f -) || ($(KUSTOMIZE) build config/crd | kubectl create -f -) - $(KUSTOMIZE) build $(shell $(GO) env GOPATH)/pkg/mod/github.com/kubernetes-csi/external-snapshotter/client/v6@v6.0.1/config/crd | kubectl apply -f - .PHONY: uninstall uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified in ~/.kube/config. Call with ignore-not-found=true to ignore resource not found errors during deletion. $(KUSTOMIZE) build config/crd | kubectl delete --ignore-not-found=$(ignore-not-found) -f - - $(KUSTOMIZE) build $(shell $(GO) env GOPATH)/pkg/mod/github.com/kubernetes-csi/external-snapshotter/client/v6@v6.0.1/config/crd | kubectl delete --ignore-not-found=$(ignore-not-found) -f - .PHONY: deploy deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config. diff --git a/examples/dataprotection/backup_job.yaml b/examples/dataprotection/backup_job.yaml deleted file mode 100644 index 25a6467e3..000000000 --- a/examples/dataprotection/backup_job.yaml +++ /dev/null @@ -1,28 +0,0 @@ -# Standard Kubernetes Kind declaration. Required. -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -# Standard Kubernetes Kind declaration. Required. -kind: Backup -# Standard Kubernetes metadata. Required. -metadata: - # Backup name. Maybe any valid Kubernetes object name. Required. - name: backup-demo - - labels: - dataprotection.kubeblocks.io/backup-type: snapshot - # if type is incremental, parentName is required. - #dataprotection.kubeblocks.io/parent-backup-name: null - # reference db cluster name - clusters.apps.kubeblocks.io/name: wesql-cluster - backuppolicies.dataprotection.kubeblocks.io/name: backup-policy-demo - # backup index begin to 0 (full) for incremental. - dataprotection.kubeblocks.io/backup-index: "0" - -# Parameters about the backup. Required. -spec: - backupPolicyName: backup-policy-demo - - # Backup Type. full or incremental or snapshot. if unset, default is full. - backupType: snapshot - - # ttl value from backup policy optional, not clean if unset. - ttl: 168h0m0s diff --git a/examples/dataprotection/backup_policy.yaml b/examples/dataprotection/backup_policy.yaml deleted file mode 100644 index 6c1654116..000000000 --- a/examples/dataprotection/backup_policy.yaml +++ /dev/null @@ -1,52 +0,0 @@ -# Standard Kubernetes Kind declaration. Required. -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -# Standard Kubernetes Kind declaration. Required. -kind: BackupPolicy -# Standard Kubernetes metadata. Required. -metadata: - # Backup name. Maybe any valid Kubernetes object name. Required. - name: backup-policy-demo -# Parameters about the backup. Required. -spec: - # schedule will be ignored when create a backup session in manual. - # [optional] - schedule: "0 3 * * *" - # The amount of time before this backup is eligible for garbage collection. If not specified, - # a default value of 30 days will be used. The default can be configured on the server - # if set to infinity, - # valid value: 30d : 30 days: - # infinity : backups will not be cleaned. - ttl: 168h0m0s - - backupToolName: xtrabackup-mysql - - # inherit from backup config - backupPolicyTemplateName: backup-config-mysql - # database cluster service selector - target: - databaseEngine: mysql - # ref definition is more general, but need to integrate with apps, define labelsSelector instead. - labelsSelector: - matchLabels: - app.kubernetes.io/instance: wesql-cluster - - # the target database secret that backup tool can connect. - secret: - # the secret name - name: wesql-cluster - - hooks: - preCommands: - - touch /data/mysql/.restore - postCommands: - - rm -f /data/mysql/.restore - - # backup remote volume from CSI driver definition. - remoteVolume: - name: backup-remote-volume - persistentVolumeClaim: - claimName: backup-s3-pvc - - - # Number of backup retries on fail. - onFailAttempted: 3 diff --git a/examples/dataprotection/backup_remote_pvc.yaml b/examples/dataprotection/backup_remote_pvc.yaml deleted file mode 100644 index b53f90cc9..000000000 --- a/examples/dataprotection/backup_remote_pvc.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: backup-s3-pvc -spec: - storageClassName: csi-s3 - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi \ No newline at end of file diff --git a/examples/dataprotection/backup_tool.yaml b/examples/dataprotection/backup_tool.yaml deleted file mode 100644 index a4149076f..000000000 --- a/examples/dataprotection/backup_tool.yaml +++ /dev/null @@ -1,84 +0,0 @@ -# Standard Kubernetes Kind declaration. Required. -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -# Standard Kubernetes Kind declaration. Required. -kind: BackupTool -# Standard Kubernetes metadata. Required. -metadata: - # Backup name. Maybe any valid Kubernetes object name. Required. - name: xtrabackup-mysql - -spec: - # backup tool image - image: registry.cn-hangzhou.aliyuncs.com/apecloud/percona-xtrabackup - - # database engine to support in the backup. - # Required. - databaseEngine: mysql - - - # Array of database major version to support in the backup. Required. - #databaseEngineVersions: - #- "8.0" - - - # backup tool runtime kind. validate value: job - deployKind: job - - # limit the backup tool container resource - # ref: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - resources: - limits: - # Limits and requests for CPU resources are measured in cpu units. - # In Kubernetes, 1 CPU unit is equivalent to 1 physical CPU core, or 1 virtual core, - # depending on whether the node is a physical host or a virtual machine running inside a physical machine. - cpu: "1" - # Limits and requests for memory are measured in bytes. - # You can express memory as a plain integer or as a fixed-point number - # using one of these quantity suffixes: E, P, T, G, M, k. - # You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. - # For example, the following represent roughly the same value: 128974848, 129e6, 129M, 128974848000m, 123Mi - memory: 2Gi - requests: - cpu: "1" - memory: 64Mi - - env: - - name: DATA_DIR - value: /var/lib/mysql - - # backup tool can support physical restore, in this case, restore must be RESTART database. - physical: - restoreCommands: - - | - echo "BACKUP_DIR=${BACKUP_DIR} BACKUP_NAME=${BACKUP_NAME} DATA_DIR=${DATA_DIR}" && \ - mkdir -p /tmp/data/ && cd /tmp/data \ - && xbstream -x < /${BACKUP_DIR}/${BACKUP_NAME}.xbstream \ - && xtrabackup --decompress --target-dir=/tmp/data/ \ - && find . -name "*.qp"|xargs rm -f \ - && rm -rf ${DATA_DIR}/* \ - && rsync -avrP /tmp/data/ ${DATA_DIR}/ \ - && rm -rf /tmp/data/ \ - && chmod -R 0777 ${DATA_DIR} - - # Optional - incrementalRestoreCommands: [] - - # backup tool can support logical restore, in this case, restore NOT RESTART database. - logical: - restoreCommands: [] - - # Optional - incrementalRestoreCommands: [ ] - - # Array of command that apps can do database backup. - # from invoke args - # the order of commands follows the order of array. - backupCommands: - - #echo "DB_HOST=${DB_HOST} DB_USER=${DB_USER} DB_PASSWORD=${DB_PASSWORD} DATA_DIR=${DATA_DIR} BACKUP_DIR=${BACKUP_DIR} BACKUP_NAME=${BACKUP_NAME}"; - xtrabackup --compress --backup --safe-slave-backup --slave-info --stream=xbstream --host=${DB_HOST} --user=${DB_USER} --datadir=${DATA_DIR} > /${BACKUP_DIR}/${BACKUP_NAME}.xbstream - - # Array of command that apps can do database incremental backup. - # like xtrabackup, that can performs an incremental backup file. - # Required if spec.backupType is defined "incremental". - # Optional - incrementalBackupCommands: [] \ No newline at end of file diff --git a/examples/dataprotection/restore_job.yaml b/examples/dataprotection/restore_job.yaml deleted file mode 100644 index 0f8221acf..000000000 --- a/examples/dataprotection/restore_job.yaml +++ /dev/null @@ -1,38 +0,0 @@ -# Standard Kubernetes Kind declaration. Required. -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -# Standard Kubernetes Kind declaration. Required. -kind: RestoreJob -# Standard Kubernetes metadata. Required. -metadata: - # Backup name. Maybe any valid Kubernetes object name. Required. - name: restore-demo -# Parameters about the backup. Required. -spec: - backupJobName: backup-success-demo - # backup tool can get from backupJob label, no need define - - # array of remote volumes from CSI driver definition. - #volumes: - # - name: restore-remote-volume - # persistentVolumeClaim: - # claimName: csi-s3-pvc - # readOnly: true - - target: - databaseEngine: mysql - labelsSelector: - matchLabels: - mysql.oracle.com/cluster: mycluster - - targetVolumes: - - name: mysql-restore-storage - persistentVolumeClaim: - claimName: datadir-mycluster-0 - - targetVolumeMounts: - - name: mysql-restore-storage - mountPath: /var/lib/mysql - - - # Number of restore retries on fail. - onFailAttempted: 3 \ No newline at end of file diff --git a/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml b/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml index b4332647d..73300ba59 100644 --- a/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml +++ b/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml @@ -15,7 +15,6 @@ spec: claimName: backup-host-path-pvc schedule: 0 3 * * * target: - databaseEngine: postgresql labelsSelector: matchLabels: app.kubernetes.io/instance: mycluster diff --git a/test/e2e/testdata/smoketest/wesql/11_backuppolicy_full.yaml b/test/e2e/testdata/smoketest/wesql/11_backuppolicy_full.yaml index d39aa6b03..587b00f2c 100644 --- a/test/e2e/testdata/smoketest/wesql/11_backuppolicy_full.yaml +++ b/test/e2e/testdata/smoketest/wesql/11_backuppolicy_full.yaml @@ -16,7 +16,6 @@ spec: claimName: backup-host-path-pvc schedule: 0 3 * * * target: - databaseEngine: mysql labelsSelector: matchLabels: app.kubernetes.io/instance: mycluster From ccae95d65f00e1ebe0cd4c3a63def44118870301 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Fri, 31 Mar 2023 19:18:25 +0800 Subject: [PATCH 22/80] feat: refactor reconfigure partI: support sync online update parameters (#2160) (#2256) --- cmd/manager/main.go | 2 + cmd/reloader/app/cmd.go | 89 +++++++++++--- cmd/reloader/app/flags.go | 30 +++-- cmd/reloader/app/proxy.go | 112 ++++++++++++++++++ cmd/reloader/main.go | 2 +- .../configuration/config_manager/builder.go | 5 + .../configuration/config_manager/handler.go | 14 +-- .../config_manager/handler_test.go | 2 +- .../config_manager/reload_util.go | 24 ++++ .../configuration/proto/reconfigure.pb.go | 93 ++++++++++----- .../configuration/proto/reconfigure.proto | 2 + internal/constant/const.go | 1 + 12 files changed, 306 insertions(+), 70 deletions(-) create mode 100644 cmd/reloader/app/proxy.go diff --git a/cmd/manager/main.go b/cmd/manager/main.go index 7500c92b1..e13d6ff2a 100644 --- a/cmd/manager/main.go +++ b/cmd/manager/main.go @@ -93,6 +93,8 @@ func init() { viper.SetDefault("PROBE_SERVICE_GRPC_PORT", 50001) viper.SetDefault("PROBE_SERVICE_LOG_LEVEL", "info") viper.SetDefault("KUBEBLOCKS_SERVICEACCOUNT_NAME", "kubeblocks") + viper.SetDefault("CONFIG_MANAGER_GRPC_PORT", 9901) + viper.SetDefault("CONFIG_MANAGER_LOG_LEVEL", "info") viper.SetDefault(constant.CfgKeyCtrlrMgrNS, "default") } diff --git a/cmd/reloader/app/cmd.go b/cmd/reloader/app/cmd.go index ca017bdc6..c7ba59fa8 100644 --- a/cmd/reloader/app/cmd.go +++ b/cmd/reloader/app/cmd.go @@ -19,6 +19,7 @@ package app import ( "context" "fmt" + "net" "os" "path/filepath" "time" @@ -27,23 +28,25 @@ import ( zaplogfmt "github.com/sykesm/zap-logfmt" "go.uber.org/zap" "go.uber.org/zap/zapcore" + "google.golang.org/grpc" "k8s.io/apimachinery/pkg/util/yaml" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgutil "github.com/apecloud/kubeblocks/internal/configuration" cfgcore "github.com/apecloud/kubeblocks/internal/configuration/config_manager" + cfgproto "github.com/apecloud/kubeblocks/internal/configuration/proto" ) var logger *zap.SugaredLogger -// NewConfigReloadCommand This command is used to reload configuration -func NewConfigReloadCommand(ctx context.Context, name string) *cobra.Command { +// NewConfigManagerCommand This command is used to reload configuration +func NewConfigManagerCommand(ctx context.Context, name string) *cobra.Command { opt := NewVolumeWatcherOpts() cmd := &cobra.Command{ Use: name, Short: name + " provides a mechanism to implement reload config files in a sidecar for kubeblocks.", RunE: func(cmd *cobra.Command, args []string) error { - return runVolumeWatchCommand(ctx, opt) + return runConfigManagerCommand(ctx, opt) }, } @@ -52,7 +55,7 @@ func NewConfigReloadCommand(ctx context.Context, name string) *cobra.Command { return cmd } -func runVolumeWatchCommand(ctx context.Context, opt *VolumeWatcherOpts) error { +func runConfigManagerCommand(ctx context.Context, opt *VolumeWatcherOpts) error { zapLog := initLog(opt.LogLevel) defer func() { _ = zapLog.Sync() @@ -64,9 +67,6 @@ func runVolumeWatchCommand(ctx context.Context, opt *VolumeWatcherOpts) error { return err } - // new volume watcher - watcher := cfgcore.NewVolumeWatcher(opt.VolumeDirs, ctx, logger) - if opt.NotifyHandType == TPLScript && opt.BackupPath == "" { tmpDir, err := os.MkdirTemp(os.TempDir(), "reload-backup-") if err != nil { @@ -75,28 +75,87 @@ func runVolumeWatchCommand(ctx context.Context, opt *VolumeWatcherOpts) error { opt.BackupPath = tmpDir defer os.RemoveAll(tmpDir) } - defer watcher.Close() + + return run(ctx, opt) +} + +func run(ctx context.Context, opt *VolumeWatcherOpts) error { + volumeWatcher, err := startVolumeWatcher(ctx, opt) + if err != nil { + return err + } + defer volumeWatcher.Close() + + serviceOpt := opt.ServiceOpt + if serviceOpt.ContainerRuntimeEnable || serviceOpt.RemoteOnlineUpdateEnable { + if err := startGRPCService(opt, ctx); err != nil { + logger.Error(err, "failed to start grpc service.") + return err + } + } + + logger.Info("config manager started.") + <-ctx.Done() + logger.Info("config manager shutdown.") + return nil + +} + +func startVolumeWatcher(ctx context.Context, opt *VolumeWatcherOpts) (*cfgcore.ConfigMapVolumeWatcher, error) { + volumeWatcher := cfgcore.NewVolumeWatcher(opt.VolumeDirs, ctx, logger) logger.Info("config backup path: ", opt.BackupPath) - eventHandle, err := createHandlerWithWatchType(opt) + eventHandle, err := createHandlerWithVolumeWatch(opt) if err != nil { logger.Error(err, "failed to create event handle.") + return nil, err } - err = watcher.AddHandler(eventHandle).Run() + err = volumeWatcher.AddHandler(eventHandle).Run() if err != nil { logger.Error(err, "failed to handle VolumeWatcher.") + return nil, err + } + return volumeWatcher, nil +} + +func startGRPCService(opt *VolumeWatcherOpts, ctx context.Context) error { + var ( + server *grpc.Server + proxy = &reconfigureProxy{opt: opt.ServiceOpt, ctx: ctx, logger: logger.Named("grpcProxy")} + ) + + if err := proxy.Init(opt); err != nil { return err } - logger.Info("reload started.") - <-ctx.Done() - logger.Info("reload shutdown.") + tcpSpec := fmt.Sprintf("%s:%d", proxy.opt.PodIP, proxy.opt.GrpcPort) + logger.Infof("starting reconfigure service: %s", tcpSpec) + listener, err := net.Listen("tcp", tcpSpec) + if err != nil { + return cfgutil.WrapError(err, "failed to create listener: [%s]", tcpSpec) + } + + server = grpc.NewServer(grpc.UnaryInterceptor(logUnaryServerInterceptor)) + cfgproto.RegisterReconfigureServer(server, proxy) + + go func() { + if err := server.Serve(listener); err != nil { + logger.Error(err, "failed to serve connections from cri") + os.Exit(1) + } + }() + logger.Info("reconfigure service started.") return nil } +func logUnaryServerInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) { + logger.Debugf("info: [%+v]", info) + return handler(ctx, req) +} + func checkOptions(opt *VolumeWatcherOpts) error { - if len(opt.VolumeDirs) == 0 { + if len(opt.VolumeDirs) == 0 && opt.NotifyHandType != TPLScript { return cfgutil.MakeError("require volume directory is null.") } @@ -172,7 +231,7 @@ func initLog(level string) *zap.Logger { return zapLog } -func createHandlerWithWatchType(opt *VolumeWatcherOpts) (cfgcore.WatchEventHandler, error) { +func createHandlerWithVolumeWatch(opt *VolumeWatcherOpts) (cfgcore.WatchEventHandler, error) { logger.Infof("access info: [%d] [%s]", opt.NotifyHandType, opt.ProcessName) switch opt.NotifyHandType { case UnixSignal: diff --git a/cmd/reloader/app/flags.go b/cmd/reloader/app/flags.go index 99e404b65..b86c45d30 100644 --- a/cmd/reloader/app/flags.go +++ b/cmd/reloader/app/flags.go @@ -83,9 +83,13 @@ type ReconfigureServiceOptions struct { GrpcPort int PodIP string + // EnableRemoteOnlineUpdate enable remote online update + RemoteOnlineUpdateEnable bool + // EnableContainerRuntime enable container runtime + ContainerRuntimeEnable bool + DebugMode bool ContainerRuntime cfgutil.CRIType - Disable bool RuntimeEndpoint string } @@ -120,11 +124,12 @@ func NewVolumeWatcherOpts() *VolumeWatcherOpts { return &VolumeWatcherOpts{ // for reconfigure options ServiceOpt: ReconfigureServiceOptions{ - GrpcPort: configManagerDefaultPort, - PodIP: viper.GetString(configPodIPEnvName), - ContainerRuntime: cfgutil.AutoType, - DebugMode: false, - Disable: false, + GrpcPort: configManagerDefaultPort, + PodIP: viper.GetString(configPodIPEnvName), + ContainerRuntime: cfgutil.AutoType, + DebugMode: false, + ContainerRuntimeEnable: false, + RemoteOnlineUpdateEnable: false, }, // for configmap watch NotifyHandType: UnixSignal, @@ -199,8 +204,13 @@ func InstallFlags(flags *pflag.FlagSet, opt *VolumeWatcherOpts) { opt.ServiceOpt.RuntimeEndpoint, "the config set cri runtime endpoint.") - flags.BoolVar(&opt.ServiceOpt.Disable, - "disable-runtime", - opt.ServiceOpt.Disable, - "the config set disable runtime.") + flags.BoolVar(&opt.ServiceOpt.ContainerRuntimeEnable, + "cri-enable", + opt.ServiceOpt.ContainerRuntimeEnable, + "the config set enable cri.") + + flags.BoolVar(&opt.ServiceOpt.RemoteOnlineUpdateEnable, + "operator-update-enable", + opt.ServiceOpt.ContainerRuntimeEnable, + "the config set enable operator update parameter.") } diff --git a/cmd/reloader/app/proxy.go b/cmd/reloader/app/proxy.go new file mode 100644 index 000000000..e1d677bb3 --- /dev/null +++ b/cmd/reloader/app/proxy.go @@ -0,0 +1,112 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package app + +import ( + "context" + + "github.com/spf13/viper" + "go.uber.org/zap" + + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" + cfgcm "github.com/apecloud/kubeblocks/internal/configuration/config_manager" + cfgutil "github.com/apecloud/kubeblocks/internal/configuration/container" + cfgproto "github.com/apecloud/kubeblocks/internal/configuration/proto" +) + +type reconfigureProxy struct { + cfgproto.ReconfigureServer + updater cfgcm.DynamicUpdater + + ctx context.Context + opt ReconfigureServiceOptions + killer cfgutil.ContainerKiller + + logger *zap.SugaredLogger +} + +var stopContainerSignal = viper.GetString(cfgutil.KillContainerSignalEnvName) + +func (r *reconfigureProxy) Init(opt *VolumeWatcherOpts) error { + if err := r.initOnlineUpdater(opt); err != nil { + r.logger.Errorf("init online updater failed: %+v", err) + return err + } + if err := r.initContainerKiller(); err != nil { + r.logger.Errorf("init container killer failed: %+v", err) + return err + } + return nil +} + +func (r *reconfigureProxy) initContainerKiller() error { + if !r.opt.ContainerRuntimeEnable { + r.logger.Info("container killer is disabled.") + return nil + } + + killer, err := cfgutil.NewContainerKiller(r.opt.ContainerRuntime, r.opt.RuntimeEndpoint, r.logger) + if err != nil { + return cfgcore.WrapError(err, "failed to create container killer") + } + if err := killer.Init(r.ctx); err != nil { + return cfgcore.WrapError(err, "failed to init killer") + } + r.killer = killer + return nil +} + +func (r *reconfigureProxy) StopContainer(ctx context.Context, request *cfgproto.StopContainerRequest) (*cfgproto.StopContainerResponse, error) { + if r.killer == nil { + return nil, cfgcore.MakeError("container killer is not initialized.") + } + ds := request.GetContainerIDs() + if len(ds) == 0 { + return &cfgproto.StopContainerResponse{ErrMessage: "not any containerId."}, nil + } + if err := r.killer.Kill(ctx, ds, stopContainerSignal, nil); err != nil { + return nil, err + } + return &cfgproto.StopContainerResponse{}, nil +} + +func (r *reconfigureProxy) OnlineUpgradeParams(_ context.Context, request *cfgproto.OnlineUpgradeParamsRequest) (*cfgproto.OnlineUpgradeParamsResponse, error) { + if r.updater == nil { + return nil, cfgcore.MakeError("online updater is not initialized.") + } + params := request.GetParams() + if len(params) == 0 { + return nil, cfgcore.MakeError("update params not empty.") + } + if err := r.updater(params); err != nil { + return nil, err + } + return &cfgproto.OnlineUpgradeParamsResponse{}, nil +} + +func (r *reconfigureProxy) initOnlineUpdater(opt *VolumeWatcherOpts) error { + if opt.NotifyHandType != TPLScript || !r.opt.RemoteOnlineUpdateEnable { + return nil + } + + updater, err := cfgcm.OnlineUpdateParamsHandle(opt.TPLScriptPath) + if err != nil { + return cfgcore.WrapError(err, "failed to create online updater") + } + r.updater = updater + return nil +} diff --git a/cmd/reloader/main.go b/cmd/reloader/main.go index d5e39a645..fe3b23eb1 100644 --- a/cmd/reloader/main.go +++ b/cmd/reloader/main.go @@ -44,7 +44,7 @@ func main() { }() viper.AutomaticEnv() - cmd := app.NewConfigReloadCommand(ctx, filepath.Base(os.Args[0])) + cmd := app.NewConfigManagerCommand(ctx, filepath.Base(os.Args[0])) if err := cmd.Execute(); err != nil && errors.Cause(err) != context.Canceled { fmt.Println(err) os.Exit(-1) diff --git a/internal/configuration/config_manager/builder.go b/internal/configuration/config_manager/builder.go index 5183c0cbc..f7b418035 100644 --- a/internal/configuration/config_manager/builder.go +++ b/internal/configuration/config_manager/builder.go @@ -22,6 +22,7 @@ import ( "path/filepath" "strings" + "github.com/spf13/viper" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "sigs.k8s.io/controller-runtime/pkg/client" @@ -29,6 +30,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgutil "github.com/apecloud/kubeblocks/internal/configuration" + "github.com/apecloud/kubeblocks/internal/constant" ) const ( @@ -61,6 +63,8 @@ func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []cor } args := buildConfigManagerCommonArgs(volumeDirs) + args = append(args, "--operator-update-enable") + args = append(args, "--tcp", viper.GetString(constant.ConfigManagerGPRCPortEnv)) args = append(args, "--notify-type", string(appsv1alpha1.TPLScriptType)) args = append(args, "--tpl-config", filepath.Join(scriptVolumePath, configTemplateName)) manager.Args = args @@ -142,6 +146,7 @@ func buildConfigManagerCommonArgs(volumeDirs []corev1.VolumeMount) []string { args := make([]string, 0) // set grpc port // args = append(args, "--tcp", viper.GetString(cfgutil.ConfigManagerGPRCPortEnv)) + args = append(args, "--log-level", viper.GetString(constant.ConfigManagerLogLevel)) for _, volume := range volumeDirs { args = append(args, "--volume-dir", volume.MountPath) } diff --git a/internal/configuration/config_manager/handler.go b/internal/configuration/config_manager/handler.go index 39cd35232..d50156a7e 100644 --- a/internal/configuration/config_manager/handler.go +++ b/internal/configuration/config_manager/handler.go @@ -22,7 +22,6 @@ import ( "os/exec" "path/filepath" "strings" - "text/template/parse" "github.com/fsnotify/fsnotify" "github.com/go-logr/logr" @@ -44,8 +43,8 @@ func SetLogger(zapLogger *zap.Logger) { logger = logger.WithName("configmap_volume_watcher") } -// findParentPidFromProcessName get parent pid -func findParentPidFromProcessName(processName string) (PID, error) { +// findPidFromProcessName get parent pid +func findPidFromProcessName(processName string) (PID, error) { allProcess, err := process.Processes() if err != nil { return InvalidPID, err @@ -88,7 +87,7 @@ func CreateSignalHandler(sig appsv1alpha1.SignalType, processName string) (Watch return nil, err } return func(event fsnotify.Event) error { - pid, err := findParentPidFromProcessName(processName) + pid, err := findPidFromProcessName(processName) if err != nil { return err } @@ -160,10 +159,3 @@ func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, return backupLastConfigFiles(currFiles, backupPath) }, nil } - -func checkTPLScript(tplName string, tplContent string) error { - tr := parse.New(tplName) - tr.Mode = parse.SkipFuncCheck - _, err := tr.Parse(tplContent, "", "", make(map[string]*parse.Tree)) - return err -} diff --git a/internal/configuration/config_manager/handler_test.go b/internal/configuration/config_manager/handler_test.go index 570719b55..9ff8a18b5 100644 --- a/internal/configuration/config_manager/handler_test.go +++ b/internal/configuration/config_manager/handler_test.go @@ -39,7 +39,7 @@ func init() { func TestFindParentPidFromProcessName(t *testing.T) { processName := getProcName() fmt.Printf("current test program name: %s\n", processName) - pid, err := findParentPidFromProcessName(processName) + pid, err := findPidFromProcessName(processName) require.Nil(t, err) require.Equal(t, PID(os.Getpid()), pid) } diff --git a/internal/configuration/config_manager/reload_util.go b/internal/configuration/config_manager/reload_util.go index 45ea1e83c..a0ccefcc3 100644 --- a/internal/configuration/config_manager/reload_util.go +++ b/internal/configuration/config_manager/reload_util.go @@ -24,6 +24,7 @@ import ( "os/exec" "path/filepath" "regexp" + "text/template/parse" "github.com/spf13/viper" @@ -38,8 +39,31 @@ type regexFilter = func(fileName string) bool const ( builtInExecFunctionName = "exec" builtInUpdateVariableFunctionName = "exec_sql" + builtInParamsPatchFunctionName = "patch_params" ) +type DynamicUpdater = func(updatedParams map[string]string) error + +func OnlineUpdateParamsHandle(tplScript string) (DynamicUpdater, error) { + tplContent, err := os.ReadFile(tplScript) + if err != nil { + return nil, err + } + if err := checkTPLScript(tplScript, string(tplContent)); err != nil { + return nil, err + } + return func(updatedParams map[string]string) error { + return wrapGoTemplateRun(tplScript, string(tplContent), updatedParams) + }, nil +} + +func checkTPLScript(tplName string, tplContent string) error { + tr := parse.New(tplName) + tr.Mode = parse.SkipFuncCheck + _, err := tr.Parse(tplContent, "", "", make(map[string]*parse.Tree)) + return err +} + func wrapGoTemplateRun(tplName string, tplContent string, updatedParams map[string]string) error { var ( err error diff --git a/internal/configuration/proto/reconfigure.pb.go b/internal/configuration/proto/reconfigure.pb.go index 066ac4d11..1bac00e76 100644 --- a/internal/configuration/proto/reconfigure.pb.go +++ b/internal/configuration/proto/reconfigure.pb.go @@ -119,6 +119,9 @@ type OnlineUpgradeParamsRequest struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields + + ConfigSpec string `protobuf:"bytes,1,opt,name=configSpec,proto3" json:"configSpec,omitempty"` + Params map[string]string `protobuf:"bytes,2,rep,name=params,proto3" json:"params,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` } func (x *OnlineUpgradeParamsRequest) Reset() { @@ -153,6 +156,20 @@ func (*OnlineUpgradeParamsRequest) Descriptor() ([]byte, []int) { return file_reconfigure_proto_rawDescGZIP(), []int{2} } +func (x *OnlineUpgradeParamsRequest) GetConfigSpec() string { + if x != nil { + return x.ConfigSpec + } + return "" +} + +func (x *OnlineUpgradeParamsRequest) GetParams() map[string]string { + if x != nil { + return x.Params + } + return nil +} + type OnlineUpgradeParamsResponse struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -212,29 +229,39 @@ var file_reconfigure_proto_rawDesc = []byte{ 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1e, 0x0a, 0x0a, 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x22, - 0x1c, 0x0a, 0x1a, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, - 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x3d, 0x0a, - 0x1b, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, - 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1e, 0x0a, 0x0a, - 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x0a, 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x32, 0xbb, 0x01, 0x0a, - 0x0b, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x75, 0x72, 0x65, 0x12, 0x4c, 0x0a, 0x0d, - 0x53, 0x74, 0x6f, 0x70, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x12, 0x1b, 0x2e, + 0xbe, 0x01, 0x0a, 0x1a, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, + 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1e, + 0x0a, 0x0a, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x53, 0x70, 0x65, 0x63, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x0a, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x53, 0x70, 0x65, 0x63, 0x12, 0x45, + 0x0a, 0x06, 0x70, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x2d, + 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, + 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x2e, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x70, + 0x61, 0x72, 0x61, 0x6d, 0x73, 0x1a, 0x39, 0x0a, 0x0b, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x45, + 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, + 0x22, 0x3d, 0x0a, 0x1b, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, + 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x1e, 0x0a, 0x0a, 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x0a, 0x65, 0x72, 0x72, 0x4d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x32, + 0xbb, 0x01, 0x0a, 0x0b, 0x52, 0x65, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x75, 0x72, 0x65, 0x12, + 0x4c, 0x0a, 0x0d, 0x53, 0x74, 0x6f, 0x70, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, + 0x12, 0x1b, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x43, 0x6f, 0x6e, + 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1c, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x69, - 0x6e, 0x65, 0x72, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1c, 0x2e, 0x70, 0x72, 0x6f, - 0x74, 0x6f, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x69, 0x6e, 0x65, 0x72, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x5e, 0x0a, 0x13, 0x4f, 0x6e, - 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, - 0x73, 0x12, 0x21, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, - 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, 0x52, 0x65, 0x71, - 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4f, 0x6e, 0x6c, + 0x6e, 0x65, 0x72, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x12, 0x5e, 0x0a, + 0x13, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, + 0x72, 0x61, 0x6d, 0x73, 0x12, 0x21, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, 0x72, 0x61, 0x6d, 0x73, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x42, 0x3d, 0x5a, 0x3b, 0x67, 0x69, - 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x61, 0x70, 0x65, 0x63, 0x6c, 0x6f, 0x75, - 0x64, 0x2f, 0x6b, 0x75, 0x62, 0x65, 0x62, 0x6c, 0x6f, 0x63, 0x6b, 0x73, 0x2f, 0x69, 0x6e, 0x74, - 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x75, 0x72, 0x61, 0x74, - 0x69, 0x6f, 0x6e, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, - 0x33, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, + 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x55, 0x70, 0x67, 0x72, 0x61, 0x64, 0x65, 0x50, 0x61, 0x72, + 0x61, 0x6d, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x00, 0x42, 0x3d, 0x5a, + 0x3b, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x61, 0x70, 0x65, 0x63, + 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x6b, 0x75, 0x62, 0x65, 0x62, 0x6c, 0x6f, 0x63, 0x6b, 0x73, 0x2f, + 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x75, + 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x06, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x33, } var ( @@ -249,23 +276,25 @@ func file_reconfigure_proto_rawDescGZIP() []byte { return file_reconfigure_proto_rawDescData } -var file_reconfigure_proto_msgTypes = make([]protoimpl.MessageInfo, 4) +var file_reconfigure_proto_msgTypes = make([]protoimpl.MessageInfo, 5) var file_reconfigure_proto_goTypes = []interface{}{ (*StopContainerRequest)(nil), // 0: proto.StopContainerRequest (*StopContainerResponse)(nil), // 1: proto.StopContainerResponse (*OnlineUpgradeParamsRequest)(nil), // 2: proto.OnlineUpgradeParamsRequest (*OnlineUpgradeParamsResponse)(nil), // 3: proto.OnlineUpgradeParamsResponse + nil, // 4: proto.OnlineUpgradeParamsRequest.ParamsEntry } var file_reconfigure_proto_depIdxs = []int32{ - 0, // 0: proto.Reconfigure.StopContainer:input_type -> proto.StopContainerRequest - 2, // 1: proto.Reconfigure.OnlineUpgradeParams:input_type -> proto.OnlineUpgradeParamsRequest - 1, // 2: proto.Reconfigure.StopContainer:output_type -> proto.StopContainerResponse - 3, // 3: proto.Reconfigure.OnlineUpgradeParams:output_type -> proto.OnlineUpgradeParamsResponse - 2, // [2:4] is the sub-list for method output_type - 0, // [0:2] is the sub-list for method input_type - 0, // [0:0] is the sub-list for extension type_name - 0, // [0:0] is the sub-list for extension extendee - 0, // [0:0] is the sub-list for field type_name + 4, // 0: proto.OnlineUpgradeParamsRequest.params:type_name -> proto.OnlineUpgradeParamsRequest.ParamsEntry + 0, // 1: proto.Reconfigure.StopContainer:input_type -> proto.StopContainerRequest + 2, // 2: proto.Reconfigure.OnlineUpgradeParams:input_type -> proto.OnlineUpgradeParamsRequest + 1, // 3: proto.Reconfigure.StopContainer:output_type -> proto.StopContainerResponse + 3, // 4: proto.Reconfigure.OnlineUpgradeParams:output_type -> proto.OnlineUpgradeParamsResponse + 3, // [3:5] is the sub-list for method output_type + 1, // [1:3] is the sub-list for method input_type + 1, // [1:1] is the sub-list for extension type_name + 1, // [1:1] is the sub-list for extension extendee + 0, // [0:1] is the sub-list for field type_name } func init() { file_reconfigure_proto_init() } @@ -329,7 +358,7 @@ func file_reconfigure_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: file_reconfigure_proto_rawDesc, NumEnums: 0, - NumMessages: 4, + NumMessages: 5, NumExtensions: 0, NumServices: 1, }, diff --git a/internal/configuration/proto/reconfigure.proto b/internal/configuration/proto/reconfigure.proto index 23212d389..df20a9129 100644 --- a/internal/configuration/proto/reconfigure.proto +++ b/internal/configuration/proto/reconfigure.proto @@ -19,6 +19,8 @@ message StopContainerResponse { } message OnlineUpgradeParamsRequest { + string configSpec = 1; + map params = 2; } message OnlineUpgradeParamsResponse { diff --git a/internal/constant/const.go b/internal/constant/const.go index c10ae30d7..4daf4990d 100644 --- a/internal/constant/const.go +++ b/internal/constant/const.go @@ -184,6 +184,7 @@ const ( const ( ConfigSidecarName = "config-manager" ConfigManagerGPRCPortEnv = "CONFIG_MANAGER_GRPC_PORT" + ConfigManagerLogLevel = "CONFIG_MANAGER_LOG_LEVEL" PodMinReadySecondsEnv = "POD_MIN_READY_SECONDS" ConfigTemplateType = "tpl" From 1cd65633fa701111904af2468411aac1e971158d Mon Sep 17 00:00:00 2001 From: lijingcheng Date: Fri, 31 Mar 2023 19:20:49 +0800 Subject: [PATCH 23/80] support: install option for kubeblocks-csi-driver (#1884) --- .../templates/addons/csi-driver-addon.yaml | 50 +++++++++++++++++++ deploy/helm/values.yaml | 3 ++ docs/release_notes/v0.1.0/template.md | 1 + 3 files changed, 54 insertions(+) create mode 100644 deploy/helm/templates/addons/csi-driver-addon.yaml diff --git a/deploy/helm/templates/addons/csi-driver-addon.yaml b/deploy/helm/templates/addons/csi-driver-addon.yaml new file mode 100644 index 000000000..01705ca90 --- /dev/null +++ b/deploy/helm/templates/addons/csi-driver-addon.yaml @@ -0,0 +1,50 @@ +apiVersion: extensions.kubeblocks.io/v1alpha1 +kind: Addon +metadata: + name: kubeblocks-csi-driver + labels: + {{- include "kubeblocks.labels" . | nindent 4 }} +spec: + description: 'Kubeblocks CSI driver provides a container storage interface used by Container Orchestrators + to manage the lifecycle of block storage for cloud vendors.' + type: Helm + + helm: + # chartLocationURL: https://github.com/apecloud/helm-charts/releases/download/kubeblocks-csi-driver-0.1.0/kubeblocks-csi-driver-0.1.0.tgz + chartLocationURL: https://jihulab.com/api/v4/projects/85949/packages/helm/stable/charts/kubeblocks-csi-driver-0.1.0.tgz + valuesMapping: + valueMap: + replicaCount: controller.replicaCount + jsonMap: + tolerations: controller.tolerations + + resources: + cpu: + requests: controller.resources.requests.cpu + limits: controller.resources.limits.cpu + memory: + requests: controller.resources.requests.memory + limits: controller.resources.limits.memory + extras: + - name: node + jsonMap: + tolerations: node.tolerations + + resources: + cpu: + requests: node.resources.requests.cpu + limits: node.resources.limits.cpu + memory: + requests: node.resources.requests.memory + limits: node.resources.limits.memory + + defaultInstallValues: + - enabled: false + + installable: + autoInstall: {{ get ( get ( .Values | toYaml | fromYaml ) "kubeblocks-csi-driver" ) "enabled" }} + selectors: + - key: KubeGitVersion + operator: Contains + values: + - eks diff --git a/deploy/helm/values.yaml b/deploy/helm/values.yaml index 75de7ea2c..954fce3e9 100644 --- a/deploy/helm/values.yaml +++ b/deploy/helm/values.yaml @@ -1558,6 +1558,9 @@ snapshot-controller: values: - "true" +kubeblocks-csi-driver: + enabled: false + ## csi-s3 settings ## ref: https://artifacthub.io/packages/helm/cloudve/csi-s3#configuration ## diff --git a/docs/release_notes/v0.1.0/template.md b/docs/release_notes/v0.1.0/template.md index 941185398..14324067d 100644 --- a/docs/release_notes/v0.1.0/template.md +++ b/docs/release_notes/v0.1.0/template.md @@ -39,6 +39,7 @@ We would like to extend our appreciation to all contributors who helped make thi * Prometheus and Alertmanager * AlertManager Webhook Adaptor * Grafana + * Kubeblocks CSI driver * S3 CSI driver * Snapshot Controller * KubeBlocks private network Load Balancer From 7947b3769c6847ac5ca0380c08a54da2d38d42f6 Mon Sep 17 00:00:00 2001 From: shaojiang Date: Fri, 31 Mar 2023 19:51:24 +0800 Subject: [PATCH 24/80] fix: add validate on kbcli addon enable cmd. (#2336) Co-authored-by: dengshaojiang --- docs/user_docs/cli/kbcli_addon_enable.md | 4 +++ internal/cli/cmd/addon/addon.go | 21 ++++++++++++ internal/cli/cmd/addon/addon_test.go | 43 ++++++++++++++++++++++++ internal/cli/testing/fake.go | 23 +++++++++++++ 4 files changed, 91 insertions(+) diff --git a/docs/user_docs/cli/kbcli_addon_enable.md b/docs/user_docs/cli/kbcli_addon_enable.md index 7bb174f8f..63ab10df3 100644 --- a/docs/user_docs/cli/kbcli_addon_enable.md +++ b/docs/user_docs/cli/kbcli_addon_enable.md @@ -27,6 +27,9 @@ kbcli addon enable ADDON_NAME [flags] # Enabled "prometheus" addon with helm like custom settings kbcli addon enable prometheus --set prometheus.alertmanager.image.tag=v0.24.0 + + # Force enabled "csi-s3" addon + kbcli addon enable csi-s3 --force ``` ### Options @@ -35,6 +38,7 @@ kbcli addon enable ADDON_NAME [flags] --allow-missing-template-keys If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats. (default true) --cpu stringArray Sets addon CPU resource values (--cpu [extraName:]/) (can specify multiple if has extra items)) --dry-run string[="unchanged"] Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource. (default "none") + --force ignoring the installable restrictions and forcefully enabling. -h, --help help for enable --memory stringArray Sets addon memory resource values (--memory [extraName:]/) (can specify multiple if has extra items)) -o, --output string Output format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file). diff --git a/internal/cli/cmd/addon/addon.go b/internal/cli/cmd/addon/addon.go index ac52422cb..73a656ad4 100644 --- a/internal/cli/cmd/addon/addon.go +++ b/internal/cli/cmd/addon/addon.go @@ -55,6 +55,7 @@ type addonEnableFlags struct { StorageClassSets []string TolerationsSet []string SetValues []string + Force bool } func (r *addonEnableFlags) useDefault() bool { @@ -176,10 +177,14 @@ func newEnableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra # Enabled "prometheus" addon with helm like custom settings kbcli addon enable prometheus --set prometheus.alertmanager.image.tag=v0.24.0 + + # Force enabled "csi-s3" addon + kbcli addon enable csi-s3 --force `), Run: func(cmd *cobra.Command, args []string) { util.CheckErr(o.init(args)) util.CheckErr(o.fetchAddonObj()) + util.CheckErr(o.validate()) util.CheckErr(o.complete(o, cmd, args)) util.CheckErr(o.Run(cmd)) }, @@ -198,6 +203,7 @@ func newEnableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra "Sets addon pod tolerations (--tolerations [extraName:]) (can specify multiple if has extra items))") cmd.Flags().StringArrayVar(&o.addonEnableFlags.SetValues, "set", []string{}, "set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2), it's only being processed if addon's type is helm.") + cmd.Flags().BoolVar(&o.addonEnableFlags.Force, "force", false, "ignoring the installable restrictions and forcefully enabling.") o.Options.AddFlags(cmd) return cmd @@ -277,6 +283,21 @@ func (o *addonCmdOpts) fetchAddonObj() error { return nil } +func (o *addonCmdOpts) validate() error { + if o.addonEnableFlags.Force { + return nil + } + if o.addon.Spec.Installable == nil { + return nil + } + for _, s := range o.addon.Spec.Installable.Selectors { + if !s.MatchesFromConfig() { + return fmt.Errorf("addon %s INSTALLABLE-SELECTOR has no matching requirement", o.Names) + } + } + return nil +} + func addonDescribeHandler(o *addonCmdOpts, cmd *cobra.Command, args []string) error { printRow := func(tbl *printer.TablePrinter, name string, item *extensionsv1alpha1.AddonInstallSpecItem) { pvEnabled := "" diff --git a/internal/cli/cmd/addon/addon_test.go b/internal/cli/cmd/addon/addon_test.go index b6250d11e..229a0830a 100644 --- a/internal/cli/cmd/addon/addon_test.go +++ b/internal/cli/cmd/addon/addon_test.go @@ -17,11 +17,23 @@ limitations under the License. package addon import ( + "net/http" + . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/cli-runtime/pkg/resource" + "k8s.io/client-go/dynamic/fake" + "k8s.io/client-go/kubernetes/scheme" + restfake "k8s.io/client-go/rest/fake" cmdtesting "k8s.io/kubectl/pkg/cmd/testing" + + "github.com/apecloud/kubeblocks/internal/cli/patch" + "github.com/apecloud/kubeblocks/internal/cli/testing" + "github.com/apecloud/kubeblocks/internal/cli/types" ) const ( @@ -65,6 +77,37 @@ var _ = Describe("Manage applications related to KubeBlocks", func() { }) }) + When("Validate at enable an addon", func() { + It("should return error", func() { + o := &addonCmdOpts{ + Options: patch.NewOptions(tf, streams, types.AddonGVR()), + Factory: tf, + IOStreams: streams, + addonEnableFlags: &addonEnableFlags{}, + complete: addonEnableDisableHandler, + } + addonObj := testing.FakeAddon("addon-test") + o.addon = *addonObj + codec := scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...) + httpResp := func(obj runtime.Object) *http.Response { + return &http.Response{StatusCode: http.StatusOK, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, obj)} + } + tf.Client = &restfake.RESTClient{ + GroupVersion: schema.GroupVersion{Group: "version", Version: ""}, + NegotiatedSerializer: resource.UnstructuredPlusDefaultContentConfig().NegotiatedSerializer, + Client: restfake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { + urlPrefix := "/version" + return map[string]*http.Response{ + urlPrefix: httpResp(testing.FakeServices()), + }[req.URL.Path], nil + }), + } + tf.FakeDynamicClient = fake.NewSimpleDynamicClient( + scheme.Scheme, addonObj) + Expect(o.validate()).Should(HaveOccurred()) + }) + }) + // When("Enable an addon", func() { // It("should set addon.spec.install.enabled=true", func() { // By("Checking install helm chart by fake helm action config") diff --git a/internal/cli/testing/fake.go b/internal/cli/testing/fake.go index 099342c50..bf375a56a 100644 --- a/internal/cli/testing/fake.go +++ b/internal/cli/testing/fake.go @@ -30,6 +30,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" dpv1alpha1 "github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1" + extensionsv1alpha1 "github.com/apecloud/kubeblocks/apis/extensions/v1alpha1" "github.com/apecloud/kubeblocks/internal/cli/types" "github.com/apecloud/kubeblocks/internal/constant" ) @@ -444,3 +445,25 @@ func FakeKBDeploy(version string) *appsv1.Deployment { } return deploy } + +func FakeAddon(name string) *extensionsv1alpha1.Addon { + addon := &extensionsv1alpha1.Addon{ + TypeMeta: metav1.TypeMeta{ + APIVersion: fmt.Sprintf("%s/%s", types.ExtensionsAPIGroup, types.ExtensionsAPIVersion), + Kind: "Addon", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: Namespace, + }, + Spec: extensionsv1alpha1.AddonSpec{ + Installable: &extensionsv1alpha1.InstallableSpec{ + Selectors: []extensionsv1alpha1.SelectorRequirement{ + {Key: extensionsv1alpha1.KubeGitVersion, Operator: extensionsv1alpha1.Contains, Values: []string{"k3s"}}, + }, + }, + }, + } + addon.SetCreationTimestamp(metav1.Now()) + return addon +} From fec9bf63b766e1b49ba6ffb5b383b0fe54b92a32 Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Fri, 31 Mar 2023 20:05:33 +0800 Subject: [PATCH 25/80] fix: cli version panic for invalid kubeconfig (#2365) --- internal/cli/util/version.go | 3 ++- internal/cli/util/version_test.go | 21 +++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/internal/cli/util/version.go b/internal/cli/util/version.go index fc18170c7..bcbcd116f 100644 --- a/internal/cli/util/version.go +++ b/internal/cli/util/version.go @@ -19,6 +19,7 @@ package util import ( "context" "fmt" + "reflect" appsv1 "k8s.io/api/apps/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -44,7 +45,7 @@ func GetVersionInfo(client kubernetes.Interface) (map[AppName]string, error) { KBCLIApp: version.GetVersion(), } - if client == nil { + if client == nil || reflect.ValueOf(client).IsNil() { return versionInfo, nil } diff --git a/internal/cli/util/version_test.go b/internal/cli/util/version_test.go index 82e8e74f8..838dbcbe8 100644 --- a/internal/cli/util/version_test.go +++ b/internal/cli/util/version_test.go @@ -20,12 +20,33 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "k8s.io/client-go/kubernetes" + "github.com/apecloud/kubeblocks/internal/cli/testing" ) const kbVersion = "0.3.0" var _ = Describe("version util", func() { + It("get version info when client is nil", func() { + info, err := GetVersionInfo(nil) + Expect(err).Should(Succeed()) + Expect(info).ShouldNot(BeEmpty()) + Expect(info[KubeBlocksApp]).Should(BeEmpty()) + Expect(info[KubernetesApp]).Should(BeEmpty()) + Expect(info[KBCLIApp]).ShouldNot(BeEmpty()) + }) + + It("get version info when client variable is a nil pointer", func() { + var client *kubernetes.Clientset + info, err := GetVersionInfo(client) + Expect(err).Should(Succeed()) + Expect(info).ShouldNot(BeEmpty()) + Expect(info[KubeBlocksApp]).Should(BeEmpty()) + Expect(info[KubernetesApp]).Should(BeEmpty()) + Expect(info[KBCLIApp]).ShouldNot(BeEmpty()) + }) + It("get version info when KubeBlocks is deployed", func() { client := testing.FakeClientSet(testing.FakeKBDeploy(kbVersion)) info, err := GetVersionInfo(client) From a9dc492862eff0801861d1dea5226d0b4732fc80 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Fri, 31 Mar 2023 20:51:18 +0800 Subject: [PATCH 26/80] chore: update addon list cmd list header from INSTALLABLE-SELECTOR to AUTO-INSTALLABLE-SELECTOR (#2341) --- docs/user_docs/cli/kbcli_addon_enable.md | 6 +++++- docs/user_docs/cli/kbcli_addon_list.md | 3 +-- go.mod | 2 +- internal/cli/cmd/addon/addon.go | 27 +++++++++++++----------- internal/cli/list/list.go | 6 ++++-- 5 files changed, 26 insertions(+), 18 deletions(-) diff --git a/docs/user_docs/cli/kbcli_addon_enable.md b/docs/user_docs/cli/kbcli_addon_enable.md index 63ab10df3..c9811c053 100644 --- a/docs/user_docs/cli/kbcli_addon_enable.md +++ b/docs/user_docs/cli/kbcli_addon_enable.md @@ -45,7 +45,11 @@ kbcli addon enable ADDON_NAME [flags] --replicas stringArray Sets addon component replica count (--replicas [extraName:]) (can specify multiple if has extra items)) --set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2), it's only being processed if addon's type is helm. --show-managed-fields If true, keep the managedFields when printing objects in JSON or YAML format. - --storage stringArray Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items)) + --storage stringArray Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items)). + Additional notes: for type=Helm addon and if the value mapped directly to a StatefulSet's volume claim template + the helm upgrade action will failed, to resolved this you will need to disable and re-enable the addon, also noted + that storage size can only be expanded by PVC resizing. + --storage-class stringArray Sets addon storage class name (--storage-class [extraName:]) (can specify multiple if has extra items)) --template string Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]. --tolerations stringArray Sets addon pod tolerations (--tolerations [extraName:]) (can specify multiple if has extra items)) diff --git a/docs/user_docs/cli/kbcli_addon_list.md b/docs/user_docs/cli/kbcli_addon_list.md index 18912f0cd..4e7f276ea 100644 --- a/docs/user_docs/cli/kbcli_addon_list.md +++ b/docs/user_docs/cli/kbcli_addon_list.md @@ -5,13 +5,12 @@ title: kbcli addon list List addons. ``` -kbcli addon list [flags] +kbcli addon list [flags] ``` ### Options ``` - -A, --all-namespace If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace. -h, --help help for list -o, --output format prints the output in the specified format. Allowed values: table, json, yaml, wide (default table) -l, --selector string Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints. diff --git a/go.mod b/go.mod index a96d6442c..54dec3e09 100644 --- a/go.mod +++ b/go.mod @@ -22,6 +22,7 @@ require ( github.com/dapr/dapr v1.9.5 github.com/dapr/go-sdk v1.7.0 github.com/dapr/kit v0.0.3 + github.com/docker/cli v20.10.21+incompatible github.com/docker/docker v20.10.23+incompatible github.com/docker/go-connections v0.4.0 github.com/evanphx/json-patch v5.6.0+incompatible @@ -147,7 +148,6 @@ require ( github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect github.com/dimchansky/utfbom v1.1.1 // indirect github.com/distribution/distribution/v3 v3.0.0-20221208165359-362910506bc2 // indirect - github.com/docker/cli v20.10.21+incompatible // indirect github.com/docker/distribution v2.8.1+incompatible // indirect github.com/docker/docker-credential-helpers v0.7.0 // indirect github.com/docker/go v1.5.1-1.0.20160303222718-d30aec9fd63c // indirect diff --git a/internal/cli/cmd/addon/addon.go b/internal/cli/cmd/addon/addon.go index 73a656ad4..9a72cd7bb 100644 --- a/internal/cli/cmd/addon/addon.go +++ b/internal/cli/cmd/addon/addon.go @@ -23,6 +23,7 @@ import ( "strconv" "strings" + "github.com/docker/cli/cli" "github.com/spf13/cobra" "github.com/spf13/pflag" "github.com/spf13/viper" @@ -84,7 +85,7 @@ type addonCmdOpts struct { // NewAddonCmd for addon functions func NewAddonCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { cmd := &cobra.Command{ - Use: "addon", + Use: "addon COMMAND", Short: "Addon command.", } cmd.AddCommand( @@ -99,15 +100,16 @@ func NewAddonCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra. func newListCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { o := list.NewListOptions(f, streams, types.AddonGVR()) cmd := &cobra.Command{ - Use: "list ", + Use: "list", Short: "List addons.", Aliases: []string{"ls"}, + Args: cli.NoArgs, ValidArgsFunction: util.ResourceNameCompletionFunc(f, o.GVR), Run: func(cmd *cobra.Command, args []string) { util.CheckErr(addonListRun(o)) }, } - o.AddFlags(cmd) + o.AddFlags(cmd, true) return cmd } @@ -121,6 +123,7 @@ func newDescribeCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cob cmd := &cobra.Command{ Use: "describe ADDON_NAME", Short: "Describe an addon specification.", + Args: cli.ExactArgs(1), ValidArgsFunction: util.ResourceNameCompletionFunc(f, types.AddonGVR()), Run: func(cmd *cobra.Command, args []string) { util.CheckErr(o.init(args)) @@ -159,6 +162,7 @@ func newEnableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra cmd := &cobra.Command{ Use: "enable ADDON_NAME", Short: "Enable an addon.", + Args: cli.ExactArgs(1), ValidArgsFunction: util.ResourceNameCompletionFunc(f, types.AddonGVR()), Example: templates.Examples(` # Enabled "prometheus" addon @@ -194,7 +198,11 @@ func newEnableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra cmd.Flags().StringArrayVar(&o.addonEnableFlags.CPUSets, "cpu", []string{}, "Sets addon CPU resource values (--cpu [extraName:]/) (can specify multiple if has extra items))") cmd.Flags().StringArrayVar(&o.addonEnableFlags.StorageSets, "storage", []string{}, - "Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items))") + `Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items)). +Additional notes: for type=Helm addon and if the value mapped directly to a StatefulSet's volume claim template +the helm upgrade action will failed, to resolved this you will need to disable and re-enable the addon, also noted +that storage size can only be expanded by PVC resizing. +`) cmd.Flags().StringArrayVar(&o.addonEnableFlags.ReplicaCountSets, "replicas", []string{}, "Sets addon component replica count (--replicas [extraName:]) (can specify multiple if has extra items))") cmd.Flags().StringArrayVar(&o.addonEnableFlags.StorageClassSets, "storage-class", []string{}, @@ -227,6 +235,7 @@ func newDisableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobr cmd := &cobra.Command{ Use: "disable ADDON_NAME", Short: "Disable an addon.", + Args: cli.ExactArgs(1), ValidArgsFunction: util.ResourceNameCompletionFunc(f, types.AddonGVR()), Run: func(cmd *cobra.Command, args []string) { util.CheckErr(o.init(args)) @@ -240,12 +249,6 @@ func newDisableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobr } func (o *addonCmdOpts) init(args []string) error { - if len(args) == 0 { - return fmt.Errorf("missing addon name") - } - if len(args) > 1 { - return fmt.Errorf("only accept enable/disable single addon item") - } o.Names = args if o.dynamic == nil { var err error @@ -366,7 +369,7 @@ func addonDescribeHandler(o *addonCmdOpts, cmd *cobra.Command, args []string) er autoInstall = o.addon.Spec.Installable.AutoInstall } printer.PrintPairStringToLine("Auto-install", strconv.FormatBool(autoInstall), 0) - printer.PrintPairStringToLine("Installable", strings.Join(o.addon.Spec.Installable.GetSelectorsStrings(), ","), 0) + printer.PrintPairStringToLine("Auto-install selector", strings.Join(o.addon.Spec.Installable.GetSelectorsStrings(), ","), 0) switch o.addon.Status.Phase { case extensionsv1alpha1.AddonEnabled: @@ -745,7 +748,7 @@ func addonListRun(o *list.ListOptions) error { } if err = printer.PrintTable(o.Out, nil, printRows, - "NAME", "TYPE", "STATUS", "EXTRAS", "AUTO-INSTALL", "INSTALLABLE-SELECTOR"); err != nil { + "NAME", "TYPE", "STATUS", "EXTRAS", "AUTO-INSTALL", "AUTO-INSTALLABLE-SELECTOR"); err != nil { return err } return nil diff --git a/internal/cli/list/list.go b/internal/cli/list/list.go index bae0c4087..b054c67e0 100644 --- a/internal/cli/list/list.go +++ b/internal/cli/list/list.go @@ -75,8 +75,10 @@ func NewListOptions(f cmdutil.Factory, streams genericclioptions.IOStreams, } } -func (o *ListOptions) AddFlags(cmd *cobra.Command) { - cmd.Flags().BoolVarP(&o.AllNamespaces, "all-namespace", "A", o.AllNamespaces, "If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.") +func (o *ListOptions) AddFlags(cmd *cobra.Command, isClusterScope ...bool) { + if len(isClusterScope) == 0 || !isClusterScope[0] { + cmd.Flags().BoolVarP(&o.AllNamespaces, "all-namespace", "A", o.AllNamespaces, "If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.") + } cmd.Flags().StringVarP(&o.LabelSelector, "selector", "l", o.LabelSelector, "Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.") cmd.Flags().BoolVar(&o.ShowLabels, "show-labels", false, "When printing, show all labels as the last column (default hide labels column)") printer.AddOutputFlag(cmd, &o.Format) From 7a7eb23744b8a4871154a002097ccdeffc915af0 Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Fri, 31 Mar 2023 22:35:19 +0800 Subject: [PATCH 27/80] feat: cli playground create 3 replicas cluster (#2358) --- internal/cli/cmd/cluster/create.go | 1 - internal/cli/cmd/playground/init.go | 22 ++++++++++++++++------ 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/internal/cli/cmd/cluster/create.go b/internal/cli/cmd/cluster/create.go index eca3ddf03..25b404ec6 100644 --- a/internal/cli/cmd/cluster/create.go +++ b/internal/cli/cmd/cluster/create.go @@ -471,7 +471,6 @@ func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map var comps []*appsv1alpha1.ClusterComponentSpec for _, c := range cd.Spec.ComponentDefs { - sets := map[setKey]string{} if setsMap != nil { sets = setsMap[c.Name] diff --git a/internal/cli/cmd/playground/init.go b/internal/cli/cmd/playground/init.go index 8a4a63458..919fca5f4 100644 --- a/internal/cli/cmd/playground/init.go +++ b/internal/cli/cmd/playground/init.go @@ -187,8 +187,11 @@ func (o *initOptions) installKBAndCluster(k8sClusterName string) error { } // Install database cluster - spinner := printer.Spinner(o.Out, "Create cluster %s (ClusterDefinition: %s, ClusterVersion: %s)", - kbClusterName, o.clusterDef, o.clusterVersion) + clusterInfo := "ClusterDefinition: " + o.clusterDef + if o.clusterVersion != "" { + clusterInfo += ", ClusterVersion: " + o.clusterVersion + } + spinner := printer.Spinner(o.Out, "Create cluster %s (%s)", kbClusterName, clusterInfo) defer spinner(false) if err = o.createCluster(); err != nil { return errors.Wrapf(err, "failed to create cluster %s", kbClusterName) @@ -393,7 +396,8 @@ func (o *initOptions) installKubeBlocks() error { // createCluster construct a cluster create options and run func (o *initOptions) createCluster() error { - options, err := newCreateOptions(o.clusterDef, o.clusterVersion) + // construct a cluster create options and run + options, err := o.newCreateOptions() if err != nil { return err } @@ -437,7 +441,7 @@ func (o *initOptions) checkExistedCluster() error { return nil } -func newCreateOptions(cd string, version string) (*cmdcluster.CreateOptions, error) { +func (o *initOptions) newCreateOptions() (*cmdcluster.CreateOptions, error) { dynamicClient, err := util.NewFactory().DynamicClient() if err != nil { return nil, err @@ -455,9 +459,15 @@ func newCreateOptions(cd string, version string) (*cmdcluster.CreateOptions, err PodAntiAffinity: "Preferred", Tenancy: "SharedNode", }, - ClusterDefRef: cd, - ClusterVersionRef: version, + ClusterDefRef: o.clusterDef, + ClusterVersionRef: o.clusterVersion, + } + + // if we are running on cloud, create cluster with three replicas + if o.cloudProvider != cp.Local { + options.Values = append(options.Values, "replicas=3") } + if err = options.Validate(); err != nil { return nil, err } From 500ecde582b3808e42da2753f58b71a83b0825bc Mon Sep 17 00:00:00 2001 From: Ziang Guo Date: Mon, 3 Apr 2023 10:59:10 +0800 Subject: [PATCH 28/80] chore: export qdrant grpc port 6334 (#2381) --- deploy/chatgpt-retrieval-plugin/templates/deployment.yaml | 4 ++-- deploy/qdrant/templates/clusterdefinition.yaml | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml index b92eecb48..e605215ce 100644 --- a/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml +++ b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml @@ -39,11 +39,11 @@ spec: protocol: TCP livenessProbe: httpGet: - path: / + path: /docs port: http readinessProbe: httpGet: - path: / + path: /docs port: http resources: {{- toYaml .Values.resources | nindent 12 }} diff --git a/deploy/qdrant/templates/clusterdefinition.yaml b/deploy/qdrant/templates/clusterdefinition.yaml index 07dff600e..e55b569d9 100644 --- a/deploy/qdrant/templates/clusterdefinition.yaml +++ b/deploy/qdrant/templates/clusterdefinition.yaml @@ -89,6 +89,8 @@ spec: ports: - name: tcp-qdrant containerPort: 6333 + - name: grpc-qdrant + containerPort: 6334 - name: tcp-metrics containerPort: 9091 command: From 7706472381b921909bc2b23e5443e795a95ff06a Mon Sep 17 00:00:00 2001 From: shanshanying Date: Mon, 3 Apr 2023 11:10:08 +0800 Subject: [PATCH 29/80] fix: systemaccount jobs added build-in tolerations (#2382) --- controllers/apps/components/pod_controller.go | 3 +++ controllers/apps/systemaccount_util.go | 11 +++++++++-- controllers/apps/systemaccount_util_test.go | 11 ++++++++--- internal/controller/component/affinity_utils.go | 4 ++-- internal/controller/component/component.go | 2 +- 5 files changed, 23 insertions(+), 8 deletions(-) diff --git a/controllers/apps/components/pod_controller.go b/controllers/apps/components/pod_controller.go index f20167f57..ea74f949f 100644 --- a/controllers/apps/components/pod_controller.go +++ b/controllers/apps/components/pod_controller.go @@ -95,6 +95,9 @@ func (r *PodReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.R // sync leader status from cluster.status patch := client.MergeFrom(pod.DeepCopy()) + if pod.Annotations == nil { + pod.Annotations = make(map[string]string) + } pod.Annotations[constant.LeaderAnnotationKey] = componentStatus.ConsensusSetStatus.Leader.Pod if err = r.Client.Patch(reqCtx.Ctx, pod, patch); err != nil { return intctrlutil.CheckedRequeueWithError(err, reqCtx.Log, "") diff --git a/controllers/apps/systemaccount_util.go b/controllers/apps/systemaccount_util.go index 732b654b7..9a6910b99 100644 --- a/controllers/apps/systemaccount_util.go +++ b/controllers/apps/systemaccount_util.go @@ -369,9 +369,16 @@ func calibrateJobMetaAndSpec(job *batchv1.Job, cluster *appsv1alpha1.Cluster, co defaultTTLZero := (int32)(0) job.Spec.TTLSecondsAfterFinished = &defaultTTLZero } + // add toleration tolerations := cluster.Spec.Tolerations - if len(tolerations) > 0 { - job.Spec.Template.Spec.Tolerations = cluster.Spec.Tolerations + clusterComp := cluster.GetComponentByName(compKey.componentName) + if clusterComp != nil { + if len(clusterComp.Tolerations) != 0 { + tolerations = clusterComp.Tolerations + } } + // add built-in toleration + tolerations = componetutil.PatchBuiltInToleration(tolerations) + job.Spec.Template.Spec.Tolerations = tolerations } diff --git a/controllers/apps/systemaccount_util_test.go b/controllers/apps/systemaccount_util_test.go index 4648e313c..8d77fa352 100644 --- a/controllers/apps/systemaccount_util_test.go +++ b/controllers/apps/systemaccount_util_test.go @@ -18,7 +18,6 @@ package apps import ( "math/rand" - "reflect" "strings" "testing" @@ -230,8 +229,14 @@ func TestRenderJob(t *testing.T) { assert.NotNil(t, job) calibrateJobMetaAndSpec(job, cluster, compKey, acc.Name) jobToleration := job.Spec.Template.Spec.Tolerations - assert.Equal(t, 1, len(jobToleration)) - assert.True(t, reflect.DeepEqual(toleration[0], jobToleration[0])) + assert.Equal(t, 2, len(jobToleration)) + // make sure the toleration is added to job and contains our built-in toleration + tolerationKeys := make([]string, 0) + for _, t := range jobToleration { + tolerationKeys = append(tolerationKeys, t.Key) + } + assert.Contains(t, tolerationKeys, constant.KubeBlocksDataNodeTolerationKey) + assert.Contains(t, tolerationKeys, toleration[0].Key) case appsv1alpha1.ReferToExisting: assert.False(t, strings.Contains(acc.ProvisionPolicy.SecretRef.Name, constant.ConnCredentialPlaceHolder)) } diff --git a/internal/controller/component/affinity_utils.go b/internal/controller/component/affinity_utils.go index c1a9d7082..8b232f4ee 100644 --- a/internal/controller/component/affinity_utils.go +++ b/internal/controller/component/affinity_utils.go @@ -167,8 +167,8 @@ func patchBuiltInAffinity(affinity *corev1.Affinity) *corev1.Affinity { return affinity } -// patchBuiltInToleration patches built-in tolerations configuration -func patchBuiltInToleration(tolerations []corev1.Toleration) []corev1.Toleration { +// PatchBuiltInToleration patches built-in tolerations configuration +func PatchBuiltInToleration(tolerations []corev1.Toleration) []corev1.Toleration { tolerations = append(tolerations, corev1.Toleration{ Key: intctrlutil.KubeBlocksDataNodeTolerationKey, Operator: corev1.TolerationOpEqual, diff --git a/internal/controller/component/component.go b/internal/controller/component/component.go index cd32cbd44..8916fe2ea 100644 --- a/internal/controller/component/component.go +++ b/internal/controller/component/component.go @@ -95,7 +95,7 @@ func BuildComponent( if len(clusterCompSpec.Tolerations) != 0 { tolerations = clusterCompSpec.Tolerations } - component.PodSpec.Tolerations = patchBuiltInToleration(tolerations) + component.PodSpec.Tolerations = PatchBuiltInToleration(tolerations) if clusterCompSpec.VolumeClaimTemplates != nil { component.VolumeClaimTemplates = clusterCompSpec.ToVolumeClaimTemplates() From 924ab03ea28f5c537e922028e50a7252e329a2ae Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Mon, 3 Apr 2023 11:11:53 +0800 Subject: [PATCH 30/80] fix: cli playground pull latest cloud provider repo (#2373) --- internal/cli/util/git.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/internal/cli/util/git.go b/internal/cli/util/git.go index 438f72e94..520202abc 100644 --- a/internal/cli/util/git.go +++ b/internal/cli/util/git.go @@ -31,8 +31,13 @@ func CloneGitRepo(url, branch, path string) error { if err != nil { return err } - // Pull the latest changes from the origin remote and merge into the current branch - err = w.Pull(&git.PullOptions{RemoteName: "origin"}) + // Pull the latest changes from the origin remote + err = w.Pull(&git.PullOptions{ + RemoteName: "origin", + Progress: os.Stdout, + ReferenceName: plumbing.NewBranchReferenceName(branch), + SingleBranch: true, + }) if err != git.NoErrAlreadyUpToDate && err != git.ErrUnstagedChanges { return err } From 5eed14f8c396e84a94be7799a7ed77c0b339f2f9 Mon Sep 17 00:00:00 2001 From: wangyelei Date: Mon, 3 Apr 2023 13:44:11 +0800 Subject: [PATCH 31/80] chore: add the data directory check before restoring the new cluster (#2360) --- .../apecloud-mysql/templates/backuptool.yaml | 28 +++++++++++-------- deploy/postgresql/templates/backuptool.yaml | 8 ++++-- 2 files changed, 22 insertions(+), 14 deletions(-) diff --git a/deploy/apecloud-mysql/templates/backuptool.yaml b/deploy/apecloud-mysql/templates/backuptool.yaml index d98b0f5cf..5b0cd01c4 100644 --- a/deploy/apecloud-mysql/templates/backuptool.yaml +++ b/deploy/apecloud-mysql/templates/backuptool.yaml @@ -20,19 +20,23 @@ spec: value: /data/mysql/data physical: restoreCommands: - - > + - | set -e; - mkdir -p /tmp/data/ && cd /tmp/data; - xbstream -x < /${BACKUP_DIR}/${BACKUP_NAME}.xbstream; - xtrabackup --decompress --target-dir=/tmp/data/; - xtrabackup --prepare --target-dir=/tmp/data/; - find . -name "*.qp"|xargs rm -f; - rm -rf ${DATA_DIR}/*; - rm -rf ${DATA_DIR}/.xtrabackup_restore_new_cluster; - xtrabackup --move-back --target-dir=/tmp/data/ --datadir=${DATA_DIR}/; - touch ${DATA_DIR}/.xtrabackup_restore_new_cluster; - rm -rf /tmp/data/; - chmod -R 0777 ${DATA_DIR}; + mkdir -p ${DATA_DIR} + res=`ls -A ${DATA_DIR}` + if [ ! -z ${res} ]; then + echo "${DATA_DIR} is not empty! Please make sure that the directory is empty before restoring the backup." + exit 1 + fi + mkdir -p /tmp/data/ && cd /tmp/data + xbstream -x < /${BACKUP_DIR}/${BACKUP_NAME}.xbstream + xtrabackup --decompress --target-dir=/tmp/data/ + xtrabackup --prepare --target-dir=/tmp/data/ + find . -name "*.qp"|xargs rm -f + xtrabackup --move-back --target-dir=/tmp/data/ --datadir=${DATA_DIR}/ + touch ${DATA_DIR}/.xtrabackup_restore_new_cluster + rm -rf /tmp/data/ + chmod -R 0777 ${DATA_DIR} incrementalRestoreCommands: [] logical: restoreCommands: [] diff --git a/deploy/postgresql/templates/backuptool.yaml b/deploy/postgresql/templates/backuptool.yaml index c29c4b18c..d9b7164c1 100644 --- a/deploy/postgresql/templates/backuptool.yaml +++ b/deploy/postgresql/templates/backuptool.yaml @@ -21,10 +21,14 @@ spec: physical: restoreCommands: - | - #!/bin/sh set -e + mkdir -p ${DATA_DIR} + res=`ls -A ${DATA_DIR}` + if [ ! -z ${res} ]; then + echo "${DATA_DIR} is not empty! Please make sure that the directory is empty before restoring the backup." + exit 1 + fi mkdir -p ${DATA_DIR} && mkdir -p ${DATA_DIR}/../arch - rm -rf ${DATA_DIR}/../arch/* && rm -rf ${DATA_DIR}/* cd ${BACKUP_DIR}/${BACKUP_NAME} tar -xvf base.tar.gz -C ${DATA_DIR}/ tar -xvf pg_wal.tar.gz -C ${DATA_DIR}/../arch From 18dfbc6bc7c307953ee567200aa006879e7b761b Mon Sep 17 00:00:00 2001 From: linghan-hub <56351212+linghan-hub@users.noreply.github.com> Date: Mon, 3 Apr 2023 14:07:59 +0800 Subject: [PATCH 32/80] chore: e2e test optimizes the k8s environment init and adjust test case (#2369) --- Makefile | 2 +- deploy/apecloud-mysql-cluster/Chart.yaml | 2 +- .../apecloud-mysql-scale-cluster/Chart.yaml | 2 +- deploy/apecloud-mysql-scale/Chart.yaml | 2 +- deploy/apecloud-mysql/Chart.yaml | 2 +- deploy/chatgpt-retrieval-plugin/Chart.yaml | 2 +- deploy/clickhouse-cluster/Chart.yaml | 2 +- deploy/clickhouse/Chart.yaml | 2 +- deploy/helm/Chart.yaml | 4 +- deploy/kafka-cluster/Chart.yaml | 2 +- deploy/kafka/Chart.yaml | 2 +- deploy/milvus/Chart.yaml | 2 +- deploy/mongodb-cluster/Chart.yaml | 2 +- deploy/mongodb/Chart.yaml | 2 +- deploy/nyancat/Chart.yaml | 4 +- deploy/postgresql-cluster/Chart.yaml | 2 +- .../postgresql-patroni-ha-cluster/Chart.yaml | 2 +- deploy/postgresql-patroni-ha/Chart.yaml | 2 +- deploy/postgresql/Chart.yaml | 2 +- deploy/qdrant-cluster/Chart.yaml | 2 +- deploy/qdrant/Chart.yaml | 2 +- deploy/redis-cluster/Chart.yaml | 2 +- deploy/redis/Chart.yaml | 2 +- test/e2e/e2e_suite_test.go | 6 +- test/e2e/testdata/smoketest/playgroundtest.go | 89 ++++++++++++-- .../postgresql/00_postgresqlcluster.yaml | 2 +- .../smoketest/postgresql/01_vscale.yaml | 3 - ...backuppolicy.yaml => 06_backuppolicy.yaml} | 2 +- .../06_reconfigure_staticparameter.yaml | 18 --- ..._snapshot.yaml => 07_backup_snapshot.yaml} | 2 +- ...e.yaml => 08_backup_snapshot_restore.yaml} | 10 +- .../smoketest/redis/00_rediscluster.yaml | 12 +- test/e2e/testdata/smoketest/smoketestrun.go | 114 ++++++++---------- .../smoketest/wesql/00_wesqlcluster.yaml | 2 +- test/e2e/testdata/smoketest/wesql/04_cv.yaml | 14 +-- .../smoketest/wesql/10_reconfigure.yaml | 2 +- test/e2e/util/client.go | 16 +++ test/e2e/util/smoke_util.go | 31 +---- 38 files changed, 198 insertions(+), 175 deletions(-) rename test/e2e/testdata/smoketest/postgresql/{07_backuppolicy.yaml => 06_backuppolicy.yaml} (91%) delete mode 100644 test/e2e/testdata/smoketest/postgresql/06_reconfigure_staticparameter.yaml rename test/e2e/testdata/smoketest/postgresql/{08_backup_snapshot.yaml => 07_backup_snapshot.yaml} (83%) rename test/e2e/testdata/smoketest/postgresql/{09_backup_snapshot_restore.yaml => 08_backup_snapshot_restore.yaml} (82%) diff --git a/Makefile b/Makefile index 809237e65..515662168 100644 --- a/Makefile +++ b/Makefile @@ -748,7 +748,7 @@ endif .PHONY: render-smoke-testdata-manifests render-smoke-testdata-manifests: ## Update E2E test dataset $(HELM) template mycluster deploy/apecloud-mysql-cluster > test/e2e/testdata/smoketest/wesql/00_wesqlcluster.yaml - $(HELM) template mycluster deploy/postgresqlcluster > test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml + $(HELM) template mycluster deploy/postgresql-cluster > test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml $(HELM) template mycluster deploy/redis > test/e2e/testdata/smoketest/redis/00_rediscluster.yaml $(HELM) template mycluster deploy/redis-cluster >> test/e2e/testdata/smoketest/redis/00_rediscluster.yaml diff --git a/deploy/apecloud-mysql-cluster/Chart.yaml b/deploy/apecloud-mysql-cluster/Chart.yaml index 561977003..262d694e3 100644 --- a/deploy/apecloud-mysql-cluster/Chart.yaml +++ b/deploy/apecloud-mysql-cluster/Chart.yaml @@ -4,6 +4,6 @@ description: An ApeCloud MySQL Cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "8.0.30" diff --git a/deploy/apecloud-mysql-scale-cluster/Chart.yaml b/deploy/apecloud-mysql-scale-cluster/Chart.yaml index 9e6fda3d4..5634d4b3e 100644 --- a/deploy/apecloud-mysql-scale-cluster/Chart.yaml +++ b/deploy/apecloud-mysql-scale-cluster/Chart.yaml @@ -4,7 +4,7 @@ description: An ApeCloud MySQL-Scale Cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 # This is the version number of the ApeCloud MySQL being deployed, # rather than the version number of ApeCloud MySQL-Scale itself. diff --git a/deploy/apecloud-mysql-scale/Chart.yaml b/deploy/apecloud-mysql-scale/Chart.yaml index 12cee23dd..165ca3995 100644 --- a/deploy/apecloud-mysql-scale/Chart.yaml +++ b/deploy/apecloud-mysql-scale/Chart.yaml @@ -5,7 +5,7 @@ description: ApeCloud MySQL-Scale is ApeCloud MySQL proxy. ApeCloud MySQL-Scale type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 # This is the version number of the ApeCloud MySQL being deployed, # rather than the version number of ApeCloud MySQL-Scale itself. diff --git a/deploy/apecloud-mysql/Chart.yaml b/deploy/apecloud-mysql/Chart.yaml index 473f64748..736d7bd5b 100644 --- a/deploy/apecloud-mysql/Chart.yaml +++ b/deploy/apecloud-mysql/Chart.yaml @@ -9,7 +9,7 @@ description: ApeCloud MySQL is fully compatible with MySQL syntax and supports s type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "8.0.30" diff --git a/deploy/chatgpt-retrieval-plugin/Chart.yaml b/deploy/chatgpt-retrieval-plugin/Chart.yaml index f035bc99e..347c5625d 100644 --- a/deploy/chatgpt-retrieval-plugin/Chart.yaml +++ b/deploy/chatgpt-retrieval-plugin/Chart.yaml @@ -5,7 +5,7 @@ description: A demo application for ChatGPT plugin. type: application -version: 0.1.0 +version: 0.5.0-alpha.3 appVersion: 0.1.0 diff --git a/deploy/clickhouse-cluster/Chart.yaml b/deploy/clickhouse-cluster/Chart.yaml index a8abf1f6f..5e3e0cdba 100644 --- a/deploy/clickhouse-cluster/Chart.yaml +++ b/deploy/clickhouse-cluster/Chart.yaml @@ -4,7 +4,7 @@ description: A ClickHouse cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: 22.9.4 diff --git a/deploy/clickhouse/Chart.yaml b/deploy/clickhouse/Chart.yaml index f360e8b59..d7c59ce9d 100644 --- a/deploy/clickhouse/Chart.yaml +++ b/deploy/clickhouse/Chart.yaml @@ -9,7 +9,7 @@ annotations: type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: 22.9.4 diff --git a/deploy/helm/Chart.yaml b/deploy/helm/Chart.yaml index 74937dbd1..b3bb0da0f 100644 --- a/deploy/helm/Chart.yaml +++ b/deploy/helm/Chart.yaml @@ -15,13 +15,13 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to # follow Semantic Versioning. They should reflect the version the application is using. # It is recommended to use it with quotes. -appVersion: 0.5.0-alpha.0 +appVersion: 0.5.0-alpha.3 kubeVersion: '>=1.22.0-0' diff --git a/deploy/kafka-cluster/Chart.yaml b/deploy/kafka-cluster/Chart.yaml index b6cb8f75a..c854cb9b5 100644 --- a/deploy/kafka-cluster/Chart.yaml +++ b/deploy/kafka-cluster/Chart.yaml @@ -4,7 +4,7 @@ description: A Kafka server cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: 3.4.0 diff --git a/deploy/kafka/Chart.yaml b/deploy/kafka/Chart.yaml index 636f75a4c..d8b5b22dc 100644 --- a/deploy/kafka/Chart.yaml +++ b/deploy/kafka/Chart.yaml @@ -11,7 +11,7 @@ annotations: type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: 3.4.0 diff --git a/deploy/milvus/Chart.yaml b/deploy/milvus/Chart.yaml index b7750366c..40702d561 100644 --- a/deploy/milvus/Chart.yaml +++ b/deploy/milvus/Chart.yaml @@ -5,7 +5,7 @@ description: . type: application # This is the chart version -version: 0.1.0 +version: 0.5.0-alpha.3 # This is the version number of milvus appVersion: "2.2.4" diff --git a/deploy/mongodb-cluster/Chart.yaml b/deploy/mongodb-cluster/Chart.yaml index e8f9140ae..2a6b451d4 100644 --- a/deploy/mongodb-cluster/Chart.yaml +++ b/deploy/mongodb-cluster/Chart.yaml @@ -4,7 +4,7 @@ description: A MongoDB cluster Helm chart for KubeBlocks type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "6.0.3" diff --git a/deploy/mongodb/Chart.yaml b/deploy/mongodb/Chart.yaml index f54899ef2..818006870 100644 --- a/deploy/mongodb/Chart.yaml +++ b/deploy/mongodb/Chart.yaml @@ -4,7 +4,7 @@ description: MongoDB is a document database designed for ease of application dev type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "6.0.3" diff --git a/deploy/nyancat/Chart.yaml b/deploy/nyancat/Chart.yaml index 040676076..68a4c6259 100644 --- a/deploy/nyancat/Chart.yaml +++ b/deploy/nyancat/Chart.yaml @@ -4,8 +4,8 @@ description: A demo application for showing database cluster availability. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 -appVersion: 0.5.0-alpha.0 +appVersion: 0.5.0-alpha.3 kubeVersion: '>=1.22.0-0' diff --git a/deploy/postgresql-cluster/Chart.yaml b/deploy/postgresql-cluster/Chart.yaml index 59221a94e..84123966c 100644 --- a/deploy/postgresql-cluster/Chart.yaml +++ b/deploy/postgresql-cluster/Chart.yaml @@ -4,6 +4,6 @@ description: A PostgreSQL cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "14.7.0" diff --git a/deploy/postgresql-patroni-ha-cluster/Chart.yaml b/deploy/postgresql-patroni-ha-cluster/Chart.yaml index 2bfe51851..5985d60c8 100644 --- a/deploy/postgresql-patroni-ha-cluster/Chart.yaml +++ b/deploy/postgresql-patroni-ha-cluster/Chart.yaml @@ -4,6 +4,6 @@ description: A PostgreSQL (with Patroni HA) cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "14.5.0" diff --git a/deploy/postgresql-patroni-ha/Chart.yaml b/deploy/postgresql-patroni-ha/Chart.yaml index b82b5a880..40bb628be 100644 --- a/deploy/postgresql-patroni-ha/Chart.yaml +++ b/deploy/postgresql-patroni-ha/Chart.yaml @@ -4,7 +4,7 @@ description: A PostgreSQL (with Patroni HA) cluster definition Helm chart for Ku type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "14.5.0" diff --git a/deploy/postgresql/Chart.yaml b/deploy/postgresql/Chart.yaml index 202b4b80b..27d08942a 100644 --- a/deploy/postgresql/Chart.yaml +++ b/deploy/postgresql/Chart.yaml @@ -4,7 +4,7 @@ description: A PostgreSQL cluster definition Helm chart for Kubernetes type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "14.7.0" diff --git a/deploy/qdrant-cluster/Chart.yaml b/deploy/qdrant-cluster/Chart.yaml index 601fdbf63..84fbf4dbd 100644 --- a/deploy/qdrant-cluster/Chart.yaml +++ b/deploy/qdrant-cluster/Chart.yaml @@ -4,6 +4,6 @@ description: A Qdrant cluster Helm chart for KubeBlocks. type: application -version: 0.1.0 +version: 0.5.0-alpha.3 appVersion: "1.1.0" diff --git a/deploy/qdrant/Chart.yaml b/deploy/qdrant/Chart.yaml index 4a278bd3f..8898897de 100644 --- a/deploy/qdrant/Chart.yaml +++ b/deploy/qdrant/Chart.yaml @@ -5,7 +5,7 @@ description: . type: application # This is the chart version. -version: 0.1.0 +version: 0.5.0-alpha.3 # This is the version number of qdrant. appVersion: "1.1.0" diff --git a/deploy/redis-cluster/Chart.yaml b/deploy/redis-cluster/Chart.yaml index a8f597daf..ea3c6ed3f 100644 --- a/deploy/redis-cluster/Chart.yaml +++ b/deploy/redis-cluster/Chart.yaml @@ -4,7 +4,7 @@ description: An Redis Replication Cluster Helm chart for KubeBlocks. type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "7.0.5" diff --git a/deploy/redis/Chart.yaml b/deploy/redis/Chart.yaml index 20c6562c7..07629aa7c 100644 --- a/deploy/redis/Chart.yaml +++ b/deploy/redis/Chart.yaml @@ -4,7 +4,7 @@ description: A Redis cluster definition Helm chart for Kubernetes type: application -version: 0.5.0-alpha.0 +version: 0.5.0-alpha.3 appVersion: "7.0.5" diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go index 85551378f..ad5a9c4c3 100644 --- a/test/e2e/e2e_suite_test.go +++ b/test/e2e/e2e_suite_test.go @@ -138,6 +138,10 @@ var _ = AfterSuite(func() { }) var _ = Describe("e2e test", func() { + var _ = Describe("KubeBlocks playground init", PlaygroundInit) + + var _ = Describe("KubeBlocks uninstall", UninstallKubeblocks) + var _ = Describe("Check healthy Kubernetes cluster status", EnvCheckTest) var _ = Describe("KubeBlocks operator installation", InstallationTest) @@ -148,5 +152,5 @@ var _ = Describe("e2e test", func() { var _ = Describe("Check environment has been cleaned", EnvGotCleanedTest) - var _ = Describe("KubeBlocks playground test", PlaygroundTest) + var _ = Describe("KubeBlocks playground destroy", PlaygroundDestroy) }) diff --git a/test/e2e/testdata/smoketest/playgroundtest.go b/test/e2e/testdata/smoketest/playgroundtest.go index 2ede152c4..5c042dacf 100644 --- a/test/e2e/testdata/smoketest/playgroundtest.go +++ b/test/e2e/testdata/smoketest/playgroundtest.go @@ -18,6 +18,7 @@ package smoketest import ( "log" + "time" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" @@ -25,20 +26,78 @@ import ( e2eutil "github.com/apecloud/kubeblocks/test/e2e/util" ) -func PlaygroundTest() { +func PlaygroundInit() { BeforeEach(func() { }) AfterEach(func() { }) - Context("KubeBlocks playground test", func() { + Context("KubeBlocks playground init", func() { + It("install kbcli", func() { + err := e2eutil.CheckKbcliExists() + if err != nil { + log.Println(err) + installCmd := "curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash " + log.Println(installCmd) + install := e2eutil.ExecuteCommand(installCmd) + log.Println(install) + } + }) It("kbcli playground init", func() { cmd := "kbcli playground init" - execResult := e2eutil.ExecCommand(cmd) + log.Println(cmd) + init := e2eutil.ExecuteCommand(cmd) + log.Println(init) + }) + It("check kbcli playground cluster and pod status", func() { + checkPlaygroundCluster() + }) + }) +} + +func UninstallKubeblocks() { + BeforeEach(func() { + }) + + AfterEach(func() { + }) + Context("KubeBlocks uninstall", func() { + It("delete mycluster", func() { + commond := "kbcli cluster delete mycluster --auto-approve" + log.Println(commond) + result := e2eutil.ExecuteCommand(commond) + Expect(result).Should(BeTrue()) + }) + It("check mycluster and pod", func() { + commond := "kbcli cluster list -A" + Eventually(func(g Gomega) { + cluster := e2eutil.ExecCommand(commond) + g.Expect(e2eutil.StringStrip(cluster)).Should(Equal("Noclusterfound")) + }, time.Second*10, time.Second*1).Should(Succeed()) + cmd := "kbcli cluster list-instances" + Eventually(func(g Gomega) { + instances := e2eutil.ExecCommand(cmd) + g.Expect(e2eutil.StringStrip(instances)).Should(Equal("Noclusterfound")) + }, time.Second*10, time.Second*1).Should(Succeed()) + }) + It("kbcli kubeblocks uninstall", func() { + cmd := "kbcli kubeblocks uninstall --auto-approve --namespace=kb-system" + log.Println(cmd) + execResult := e2eutil.ExecuteCommand(cmd) log.Println(execResult) - checkPlaygroundInit() }) + }) +} + +func PlaygroundDestroy() { + BeforeEach(func() { + }) + + AfterEach(func() { + }) + + Context("KubeBlocks playground test", func() { It("kbcli playground destroy", func() { cmd := "kbcli playground destroy" execResult := e2eutil.ExecCommand(cmd) @@ -47,16 +106,20 @@ func PlaygroundTest() { }) } -func checkPlaygroundInit() { +func checkPlaygroundCluster() { + commond := "kubectl get pod -n default -l 'app.kubernetes.io/instance in (mycluster)'| grep mycluster |" + + " awk '{print $3}'" + log.Println(commond) + Eventually(func(g Gomega) { + podStatus := e2eutil.ExecCommand(commond) + log.Println(e2eutil.StringStrip(podStatus)) + g.Expect(e2eutil.StringStrip(podStatus)).Should(Equal("Running")) + }, time.Second*180, time.Second*1).Should(Succeed()) cmd := "kbcli cluster list | grep mycluster | awk '{print $6}'" - clusterStatus := e2eutil.ExecCommand(cmd) + log.Println(cmd) Eventually(func(g Gomega) { + clusterStatus := e2eutil.ExecCommand(cmd) + log.Println(e2eutil.StringStrip(clusterStatus)) g.Expect(e2eutil.StringStrip(clusterStatus)).Should(Equal("Running")) - }, timeout, interval).Should(Succeed()) - - commond := "kubectl get pod | grep mycluster | awk '{print $3}'" - podStatus := e2eutil.ExecCommand(commond) - Eventually(func(g Gomega) { - g.Expect(e2eutil.StringStrip(podStatus)).Should(Equal("Running")) - }, timeout, interval).Should(Succeed()) + }, time.Second*360, time.Second*1).Should(Succeed()) } diff --git a/test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml b/test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml index f8c19dbd7..f43e24127 100644 --- a/test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml +++ b/test/e2e/testdata/smoketest/postgresql/00_postgresqlcluster.yaml @@ -5,7 +5,7 @@ kind: Cluster metadata: name: mycluster labels: - helm.sh/chart: pgcluster-0.2.0 + helm.sh/chart: pgcluster-0.5.0-alpha.3 app.kubernetes.io/name: pgcluster app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "14.7.0" diff --git a/test/e2e/testdata/smoketest/postgresql/01_vscale.yaml b/test/e2e/testdata/smoketest/postgresql/01_vscale.yaml index 1eca20eb4..15085de54 100644 --- a/test/e2e/testdata/smoketest/postgresql/01_vscale.yaml +++ b/test/e2e/testdata/smoketest/postgresql/01_vscale.yaml @@ -7,9 +7,6 @@ spec: type: VerticalScaling verticalScaling: - componentName: postgresql - limits: - cpu: "500m" - memory: 500Mi requests: cpu: "500m" memory: 500Mi \ No newline at end of file diff --git a/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml b/test/e2e/testdata/smoketest/postgresql/06_backuppolicy.yaml similarity index 91% rename from test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml rename to test/e2e/testdata/smoketest/postgresql/06_backuppolicy.yaml index 73300ba59..a9a2a82be 100644 --- a/test/e2e/testdata/smoketest/postgresql/07_backuppolicy.yaml +++ b/test/e2e/testdata/smoketest/postgresql/06_backuppolicy.yaml @@ -1,4 +1,4 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 +apiVersion: dataprotection.kubeblocks.io/v1alpha1 kind: BackupPolicy metadata: name: backup-policy-mycluster diff --git a/test/e2e/testdata/smoketest/postgresql/06_reconfigure_staticparameter.yaml b/test/e2e/testdata/smoketest/postgresql/06_reconfigure_staticparameter.yaml deleted file mode 100644 index 12a34daf3..000000000 --- a/test/e2e/testdata/smoketest/postgresql/06_reconfigure_staticparameter.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 -kind: OpsRequest -metadata: - name: ops-reconfigure-staticparameter -spec: - clusterRef: mycluster - reconfigure: - componentName: postgresql - configurations: - - keys: - - key: postgresql.conf - parameters: - - key: logging_collector - value: "False" - - key: max_connections - value: "200" - name: postgresql-configuration - type: Reconfiguring \ No newline at end of file diff --git a/test/e2e/testdata/smoketest/postgresql/08_backup_snapshot.yaml b/test/e2e/testdata/smoketest/postgresql/07_backup_snapshot.yaml similarity index 83% rename from test/e2e/testdata/smoketest/postgresql/08_backup_snapshot.yaml rename to test/e2e/testdata/smoketest/postgresql/07_backup_snapshot.yaml index e97467fb4..65b22b227 100644 --- a/test/e2e/testdata/smoketest/postgresql/08_backup_snapshot.yaml +++ b/test/e2e/testdata/smoketest/postgresql/07_backup_snapshot.yaml @@ -1,4 +1,4 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 +apiVersion: dataprotection.kubeblocks.io/v1alpha1 kind: Backup metadata: labels: diff --git a/test/e2e/testdata/smoketest/postgresql/09_backup_snapshot_restore.yaml b/test/e2e/testdata/smoketest/postgresql/08_backup_snapshot_restore.yaml similarity index 82% rename from test/e2e/testdata/smoketest/postgresql/09_backup_snapshot_restore.yaml rename to test/e2e/testdata/smoketest/postgresql/08_backup_snapshot_restore.yaml index 23440732c..83df02f3d 100644 --- a/test/e2e/testdata/smoketest/postgresql/09_backup_snapshot_restore.yaml +++ b/test/e2e/testdata/smoketest/postgresql/08_backup_snapshot_restore.yaml @@ -10,11 +10,11 @@ spec: topologyKeys: - kubernetes.io/hostname componentSpecs: - - name: mysql - componentDefRef: mysql + - name: postgresql + componentDefRef: pg-replication monitor: false - replicas: 3 - enabledLogs: [ "slow","error" ] + replicas: 2 + enabledLogs: ["running"] volumeClaimTemplates: - name: data spec: @@ -23,7 +23,7 @@ spec: - ReadWriteOnce resources: requests: - storage: 2Gi + storage: 11Gi dataSource: apiGroup: snapshot.storage.k8s.io kind: VolumeSnapshot diff --git a/test/e2e/testdata/smoketest/redis/00_rediscluster.yaml b/test/e2e/testdata/smoketest/redis/00_rediscluster.yaml index ed74fa027..dfe5e35a2 100644 --- a/test/e2e/testdata/smoketest/redis/00_rediscluster.yaml +++ b/test/e2e/testdata/smoketest/redis/00_rediscluster.yaml @@ -5,7 +5,7 @@ kind: ConfigMap metadata: name: redis7-config-template labels: - helm.sh/chart: redis-0.1.0 + helm.sh/chart: redis-0.5.0-alpha.3 app.kubernetes.io/name: redis app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" @@ -108,7 +108,7 @@ metadata: name: backup-policy-template-redis labels: clusterdefinition.kubeblocks.io/name: redis - helm.sh/chart: redis-0.1.0 + helm.sh/chart: redis-0.5.0-alpha.3 app.kubernetes.io/name: redis app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" @@ -124,7 +124,7 @@ kind: ClusterDefinition metadata: name: redis labels: - helm.sh/chart: redis-0.1.0 + helm.sh/chart: redis-0.5.0-alpha.3 app.kubernetes.io/name: redis app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" @@ -298,7 +298,7 @@ kind: ClusterVersion metadata: name: redis-7.0.5 labels: - helm.sh/chart: redis-0.1.0 + helm.sh/chart: redis-0.5.0-alpha.3 app.kubernetes.io/name: redis app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" @@ -319,7 +319,7 @@ kind: ConfigConstraint metadata: name: redis7-config-constraints labels: - helm.sh/chart: redis-0.1.0 + helm.sh/chart: redis-0.5.0-alpha.3 app.kubernetes.io/name: redis app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" @@ -500,7 +500,7 @@ kind: Cluster metadata: name: mycluster labels: - helm.sh/chart: redis-cluster-0.1.0 + helm.sh/chart: redis-cluster-0.5.0-alpha.3 app.kubernetes.io/name: redis-cluster app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "7.0.5" diff --git a/test/e2e/testdata/smoketest/smoketestrun.go b/test/e2e/testdata/smoketest/smoketestrun.go index 72de3a789..3429ef371 100644 --- a/test/e2e/testdata/smoketest/smoketestrun.go +++ b/test/e2e/testdata/smoketest/smoketestrun.go @@ -24,11 +24,11 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/sirupsen/logrus" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/dynamic" - "k8s.io/klog/v2" extensionsv1alpha1 "github.com/apecloud/kubeblocks/apis/extensions/v1alpha1" "github.com/apecloud/kubeblocks/internal/cli/types" @@ -40,6 +40,10 @@ const ( interval time.Duration = time.Second * 1 ) +type Options struct { + Dynamic dynamic.Interface +} + func SmokeTest() { BeforeEach(func() { }) @@ -48,14 +52,43 @@ func SmokeTest() { }) Context("KubeBlocks smoke test", func() { - - It("check addon auto-install", func() { - allEnabled, err := checkAddons() + It("check addon", func() { + cfg, err := e2eutil.GetConfig() if err != nil { + logrus.WithError(err).Fatal("could not get config") + } + dynamic, err := dynamic.NewForConfig(cfg) + if err != nil { + logrus.WithError(err).Fatal("could not generate dynamic client for config") + } + objects, err := dynamic.Resource(types.AddonGVR()).List(context.TODO(), metav1.ListOptions{ + LabelSelector: e2eutil.BuildAddonLabelSelector(), + }) + if err != nil && !apierrors.IsNotFound(err) { log.Println(err) } - Expect(allEnabled).Should(BeTrue()) - + if objects == nil || len(objects.Items) == 0 { + log.Println("No Addons found") + } + if len(objects.Items) > 0 { + for _, obj := range objects.Items { + addon := extensionsv1alpha1.Addon{} + if err = runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &addon); err != nil { + log.Println(err) + } + if addon.Status.ObservedGeneration == 0 { + log.Printf("Addon %s is not observed yet", addon.Name) + } + log.Printf("Addon: %s, enabled: %v, status: %s", + addon.Name, addon.Spec.InstallSpec.GetEnabled(), addon.Status.Phase) + // addon is enabled, then check its status + if addon.Spec.InstallSpec.GetEnabled() { + if addon.Status.Phase != extensionsv1alpha1.AddonEnabled { + log.Printf("Addon %s is not enabled yet", addon.Name) + } + } + } + } }) It("run test cases", func() { dir, err := os.Getwd() @@ -69,8 +102,7 @@ func SmokeTest() { } log.Println("folder: " + folder) files, _ := e2eutil.GetFiles(folder) - e2eutil.WaitTime(400000000) - clusterVersions := e2eutil.GetClusterVersion(folder) + var clusterVersions []string if len(clusterVersions) > 1 { for _, clusterVersion := range clusterVersions { if len(files) > 0 { @@ -92,70 +124,20 @@ func runTestCases(files []string) { By("test " + file) b := e2eutil.OpsYaml(file, "apply") Expect(b).Should(BeTrue()) - podStatusResult := e2eutil.CheckPodStatus() - log.Println(podStatusResult) - for _, result := range podStatusResult { - Eventually(func(g Gomega) { + Eventually(func(g Gomega) { + podStatusResult := e2eutil.CheckPodStatus() + for _, result := range podStatusResult { g.Expect(result).Should(BeTrue()) - }).Should(Succeed()) - } - clusterStatusResult := e2eutil.CheckClusterStatus() + } + }, time.Second*180, time.Second*1).Should(Succeed()) Eventually(func(g Gomega) { + clusterStatusResult := e2eutil.CheckClusterStatus() g.Expect(clusterStatusResult).Should(BeTrue()) - }).Should(Succeed()) + }, time.Second*180, time.Second*1).Should(Succeed()) + } if len(files) > 0 { file := e2eutil.GetClusterCreateYaml(files) e2eutil.OpsYaml(file, "delete") } } - -func checkAddons() (bool, error) { - e2eutil.WaitTime(400000000) - var Dynamic dynamic.Interface - addons := make(map[string]bool) - allEnabled := true - objects, err := Dynamic.Resource(types.AddonGVR()).List(context.TODO(), metav1.ListOptions{ - LabelSelector: e2eutil.BuildAddonLabelSelector(), - }) - if err != nil && !apierrors.IsNotFound(err) { - return false, err - } - if objects == nil || len(objects.Items) == 0 { - klog.V(1).Info("No Addons found") - return false, nil - } - for _, obj := range objects.Items { - addon := extensionsv1alpha1.Addon{} - if err = runtime.DefaultUnstructuredConverter.FromUnstructured(obj.Object, &addon); err != nil { - return false, err - } - - if addon.Status.ObservedGeneration == 0 { - klog.V(1).Infof("Addon %s is not observed yet", addon.Name) - allEnabled = false - continue - } - - installable := false - if addon.Spec.InstallSpec != nil { - installable = addon.Spec.Installable.AutoInstall - } - if addon.Spec.InstallSpec != nil { - installable = addon.Spec.Installable.AutoInstall - } - - klog.V(1).Infof("Addon: %s, enabled: %v, status: %s, auto-install: %v", - addon.Name, addon.Spec.InstallSpec.GetEnabled(), addon.Status.Phase, installable) - // addon is enabled, then check its status - if addon.Spec.InstallSpec.GetEnabled() { - addons[addon.Name] = true - if addon.Status.Phase != extensionsv1alpha1.AddonEnabled { - klog.V(1).Infof("Addon %s is not enabled yet", addon.Name) - addons[addon.Name] = false - allEnabled = false - } - } - } - return allEnabled, nil -} diff --git a/test/e2e/testdata/smoketest/wesql/00_wesqlcluster.yaml b/test/e2e/testdata/smoketest/wesql/00_wesqlcluster.yaml index 42780d13e..0de7e31dd 100644 --- a/test/e2e/testdata/smoketest/wesql/00_wesqlcluster.yaml +++ b/test/e2e/testdata/smoketest/wesql/00_wesqlcluster.yaml @@ -5,7 +5,7 @@ kind: Cluster metadata: name: mycluster labels: - helm.sh/chart: apecloud-mysql-cluster-0.1.3 + helm.sh/chart: apecloud-mysql-cluster-0.5.0-alpha.3 app.kubernetes.io/name: apecloud-mysql-cluster app.kubernetes.io/instance: mycluster app.kubernetes.io/version: "8.0.30" diff --git a/test/e2e/testdata/smoketest/wesql/04_cv.yaml b/test/e2e/testdata/smoketest/wesql/04_cv.yaml index 88bbf5b17..54a786242 100644 --- a/test/e2e/testdata/smoketest/wesql/04_cv.yaml +++ b/test/e2e/testdata/smoketest/wesql/04_cv.yaml @@ -3,10 +3,10 @@ kind: ClusterVersion metadata: name: ac-mysql-8.0.30-latest spec: - clusterDefinitionRef: apecloud-postgresql - componentVersions: - - componentDefRef: mysql - versionsContext: - containers: - - name: mysql - image: docker.io/apecloud/apecloud-mysql-server:latest + clusterDefinitionRef: apecloud-mysql + componentVersions: + - componentDefRef: mysql + versionsContext: + containers: + - name: mysql + image: docker.io/apecloud/apecloud-mysql-server:latest diff --git a/test/e2e/testdata/smoketest/wesql/10_reconfigure.yaml b/test/e2e/testdata/smoketest/wesql/10_reconfigure.yaml index c2bcb87ff..e8aa63d62 100644 --- a/test/e2e/testdata/smoketest/wesql/10_reconfigure.yaml +++ b/test/e2e/testdata/smoketest/wesql/10_reconfigure.yaml @@ -14,5 +14,5 @@ spec: value: "OFF" - key: max_connections value: "2000" - name: mysql-3node-tpl + name: mysql-consensusset-config type: Reconfiguring \ No newline at end of file diff --git a/test/e2e/util/client.go b/test/e2e/util/client.go index ba654b309..da3c368d4 100644 --- a/test/e2e/util/client.go +++ b/test/e2e/util/client.go @@ -17,8 +17,12 @@ limitations under the License. package util import ( + "os" + "github.com/vmware-tanzu/velero/pkg/client" "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" kbclient "sigs.k8s.io/controller-runtime/pkg/client" ) @@ -60,3 +64,15 @@ func InitTestClient(kubecontext string) (TestClient, error) { dynamicFactory: factory, }, nil } + +func GetConfig() (*rest.Config, error) { + kubeConfigPath, exists := os.LookupEnv("KUBECONFIG") + if !exists { + kubeConfigPath = os.ExpandEnv("$HOME/.kube/config") + } + config, err := clientcmd.BuildConfigFromFlags("", kubeConfigPath) + if err != nil { + return nil, err + } + return config, nil +} diff --git a/test/e2e/util/smoke_util.go b/test/e2e/util/smoke_util.go index 4b4316538..2ebc2ca72 100644 --- a/test/e2e/util/smoke_util.go +++ b/test/e2e/util/smoke_util.go @@ -74,7 +74,6 @@ func GetFolders(path string) ([]string, error) { } func CheckClusterStatus() bool { - WaitTime(400000000) cmd := "kubectl get cluster " + name + " -n " + namespace + " | grep " + name + " | awk '{print $5}'" log.Println(cmd) clusterStatus := ExecCommand(cmd) @@ -83,7 +82,6 @@ func CheckClusterStatus() bool { } func CheckPodStatus() map[string]bool { - WaitTime(400000000) var podStatusResult = make(map[string]bool) cmd := "kubectl get pod -n " + namespace + " -l '" + label + "=" + name + "'| grep " + name + " | awk '{print $1}'" log.Println(cmd) @@ -278,32 +276,13 @@ func ReplaceClusterVersionRef(fileName string, clusterVersionRef string) { } } -func KubernetesEnv() string { - cmd := "kbcli version | grep Kubernetes" - kubernetes := ExecCommand(cmd) - log.Println(kubernetes) - return kubernetes -} - -func CheckAddonsInstall(addonName string) string { - cmd := "kbcli addon list | grep " + addonName + " | awk '{print $3}'" - log.Println(cmd) - addonsStatus := ExecCommand(cmd) - log.Println(addonsStatus) - return StringStrip(addonsStatus) -} - -func OpsAddon(addonOps string, addonName string) string { - cmd := "kbcli addon " + addonOps + " " + addonName - log.Println(cmd) - enableAddon := ExecCommand(cmd) - log.Println(enableAddon) - addonsStatus := CheckAddonsInstall(addonName) - return StringStrip(addonsStatus) -} - func StringStrip(str string) string { str = strings.ReplaceAll(str, " ", "") str = strings.ReplaceAll(str, "\n", "") return str } + +func CheckKbcliExists() error { + _, err := exec.New().LookPath("kbcli") + return err +} From 75f81b81cf723b82c840338b427551a6d7684f92 Mon Sep 17 00:00:00 2001 From: shanshanying Date: Mon, 3 Apr 2023 14:23:18 +0800 Subject: [PATCH 33/80] feat: support user-accounts management for redis (#2335) --- Makefile | 2 - cmd/cmd.mk | 7 + cmd/probe/internal/binding/base.go | 1 + cmd/probe/internal/binding/mysql/mysql.go | 189 +++---- .../internal/binding/postgres/postgres.go | 148 +++-- cmd/probe/internal/binding/redis/redis.go | 471 +++++++++++++-- .../internal/binding/redis/redis_test.go | 534 +++++++++++++++--- cmd/probe/internal/binding/utils.go | 207 ++++--- cmd/probe/internal/component/redis/redis.go | 89 ++- config/probe/components/binding_redis.yaml | 12 + .../templates/clusterdefinition.yaml | 4 + go.mod | 18 +- go.sum | 36 +- 13 files changed, 1287 insertions(+), 431 deletions(-) create mode 100644 config/probe/components/binding_redis.yaml diff --git a/Makefile b/Makefile index 515662168..345cebf01 100644 --- a/Makefile +++ b/Makefile @@ -301,8 +301,6 @@ ARGUMENTS= DEBUG_PORT=2345 run-delve: manifests generate fmt vet ## Run Delve debugger. dlv --listen=:$(DEBUG_PORT) --headless=true --api-version=2 --accept-multiclient debug $(GO_PACKAGE) -- $(ARGUMENTS) - - ##@ Deployment ifndef ignore-not-found diff --git a/cmd/cmd.mk b/cmd/cmd.mk index 62e9324fe..3e70d72c3 100644 --- a/cmd/cmd.mk +++ b/cmd/cmd.mk @@ -64,6 +64,13 @@ probe: test-go-generate build-checks ## Build probe related binaries $(MAKE) bin/probe.${OS}.${ARCH} mv bin/probe.${OS}.${ARCH} bin/probe +.PHONY: probe-fast +probe-fast: OS=$(shell $(GO) env GOOS) +probe-fast: ARCH=$(shell $(GO) env GOARCH) +probe-fast: + $(MAKE) bin/probe.${OS}.${ARCH} + mv bin/probe.${OS}.${ARCH} bin/probe + .PHONY: clean-probe clean-probe: ## Clean bin/probe. rm -f bin/probe diff --git a/cmd/probe/internal/binding/base.go b/cmd/probe/internal/binding/base.go index 46edec794..a61e84a0c 100644 --- a/cmd/probe/internal/binding/base.go +++ b/cmd/probe/internal/binding/base.go @@ -45,6 +45,7 @@ type BaseInternalOps interface { InternalQuery(ctx context.Context, sql string) ([]byte, error) InternalExec(ctx context.Context, sql string) (int64, error) GetLogger() logger.Logger + GetRunningPort() int } type BaseOperations struct { diff --git a/cmd/probe/internal/binding/mysql/mysql.go b/cmd/probe/internal/binding/mysql/mysql.go index 25bbf472f..a52c5fb91 100644 --- a/cmd/probe/internal/binding/mysql/mysql.go +++ b/cmd/probe/internal/binding/mysql/mysql.go @@ -36,7 +36,6 @@ import ( "github.com/go-sql-driver/mysql" "github.com/pkg/errors" "github.com/spf13/viper" - "golang.org/x/exp/slices" . "github.com/apecloud/kubeblocks/cmd/probe/internal/binding" . "github.com/apecloud/kubeblocks/cmd/probe/util" @@ -137,7 +136,11 @@ func (mysqlOps *MysqlOperations) initIfNeed() bool { if mysqlOps.db == nil { go func() { err := mysqlOps.InitDelay() - mysqlOps.Logger.Errorf("MySQl connection init failed: %v", err) + if err != nil { + mysqlOps.Logger.Errorf("MySQL connection init failed: %v", err) + } else { + mysqlOps.Logger.Info("MySQL connection init success.") + } }() return true } @@ -496,145 +499,141 @@ func (mysqlOps *MysqlOperations) GetLogger() logger.Logger { } func (mysqlOps *MysqlOperations) listUsersOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = ListUsersOp - ) sqlTplRend := func(user UserInfo) string { return listUserTpl } - return QueryObject(ctx, mysqlOps, req, opsKind, nil, sqlTplRend, nil) + + return QueryObject(ctx, mysqlOps, req, ListUsersOp, sqlTplRend, nil, UserInfo{}) } func (mysqlOps *MysqlOperations) describeUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = DescribeUserOp - ) + var ( + object = UserInfo{} - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName + // get user grants + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(showGrantTpl, user.UserName) } - return nil - } - // get user grants - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(showGrantTpl, user.UserName) - } - - dataProcessor := func(data []byte) (interface{}, error) { - roles := make([]map[string]string, 0) - err := json.Unmarshal(data, &roles) - if err != nil { - return nil, err - } - user := UserInfo{} - userRoles := make([]string, 0) - for _, roleMap := range roles { - for k, v := range roleMap { - if len(user.UserName) == 0 { - user.UserName = strings.TrimPrefix(strings.TrimSuffix(k, "@%"), "Grants for ") + dataProcessor = func(data interface{}) (interface{}, error) { + roles := make([]map[string]string, 0) + err := json.Unmarshal(data.([]byte), &roles) + if err != nil { + return nil, err + } + user := UserInfo{} + userRoles := make([]string, 0) + for _, roleMap := range roles { + for k, v := range roleMap { + if len(user.UserName) == 0 { + user.UserName = strings.TrimPrefix(strings.TrimSuffix(k, "@%"), "Grants for ") + } + userRoles = append(userRoles, mysqlOps.inferRoleFromPriv(strings.TrimPrefix(v, "GRANT "))) } - userRoles = append(userRoles, mysqlOps.inferRoleFromPriv(strings.TrimPrefix(v, "GRANT "))) + } + user.RoleName = strings.Join(userRoles, ",") + if jsonData, err := json.Marshal([]UserInfo{user}); err != nil { + return nil, err + } else { + return string(jsonData), nil } } - user.RoleName = strings.Join(userRoles, ",") - if jsonData, err := json.Marshal([]UserInfo{user}); err != nil { - return nil, err - } else { - return string(jsonData), nil - } + ) + + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return QueryObject(ctx, mysqlOps, req, opsKind, validFn, sqlTplRend, dataProcessor) + return QueryObject(ctx, mysqlOps, req, DescribeUserOp, sqlTplRend, dataProcessor, object) } func (mysqlOps *MysqlOperations) createUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = CreateUserOp - ) + var ( + object = UserInfo{} - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(createUserTpl, user.UserName, user.Password) } - if len(user.Password) == 0 { - return ErrNoPassword - } - return nil - } - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(createUserTpl, user.UserName, user.Password) - } + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("created user: %s, with password: %s", user.UserName, user.Password) + } + ) - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf("created user: %s, with password: %s", user.UserName, user.Password) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndPasswdValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return ExecuteObject(ctx, mysqlOps, req, opsKind, validFn, sqlTplRend, msgTplRend) + return ExecuteObject(ctx, mysqlOps, req, CreateUserOp, sqlTplRend, msgTplRend, object) } func (mysqlOps *MysqlOperations) deleteUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = DeleteUserOp - ) - - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName + var ( + object = UserInfo{} + validFn = func(user UserInfo) error { + if len(user.UserName) == 0 { + return ErrNoUserName + } + return nil } - return nil - } - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(deleteUserTpl, user.UserName) - } - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf("deleted user: %s", user.UserName) + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(deleteUserTpl, user.UserName) + } + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("deleted user: %s", user.UserName) + } + ) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, validFn, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return ExecuteObject(ctx, mysqlOps, req, opsKind, validFn, sqlTplRend, msgTplRend) + + return ExecuteObject(ctx, mysqlOps, req, DeleteUserOp, sqlTplRend, msgTplRend, object) } func (mysqlOps *MysqlOperations) grantUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( succMsgTpl = "role %s granted to user: %s" ) return mysqlOps.managePrivillege(ctx, req, GrantUserRoleOp, grantTpl, succMsgTpl) } func (mysqlOps *MysqlOperations) revokeUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( succMsgTpl = "role %s revoked from user: %s" ) return mysqlOps.managePrivillege(ctx, req, RevokeUserRoleOp, revokeTpl, succMsgTpl) } func (mysqlOps *MysqlOperations) managePrivillege(ctx context.Context, req *bindings.InvokeRequest, op bindings.OperationKind, sqlTpl string, succMsgTpl string) (OpsResult, error) { - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName - } - if len(user.RoleName) == 0 { - return ErrNoRoleName + var ( + object = UserInfo{} + sqlTplRend = func(user UserInfo) string { + // render sql stmts + roleDesc, _ := mysqlOps.renderRoleByName(user.RoleName) + // update privilege + sql := fmt.Sprintf(sqlTpl, roleDesc, user.UserName) + return sql } - roles := []string{ReadOnlyRole, ReadWriteRole, SuperUserRole} - if !slices.Contains(roles, strings.ToLower(user.RoleName)) { - return ErrInvalidRoleName + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf(succMsgTpl, user.RoleName, user.UserName) } - return nil - } - sqlTplRend := func(user UserInfo) string { - // render sql stmts - roleDesc, _ := mysqlOps.renderRoleByName(user.RoleName) - // update privilege - sql := fmt.Sprintf(sqlTpl, roleDesc, user.UserName) - return sql - } - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf(succMsgTpl, user.RoleName, user.UserName) + ) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndRoleValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - - return ExecuteObject(ctx, mysqlOps, req, op, validFn, sqlTplRend, msgTplRend) + return ExecuteObject(ctx, mysqlOps, req, op, sqlTplRend, msgTplRend, object) } func (mysqlOps *MysqlOperations) renderRoleByName(roleName string) (string, error) { diff --git a/cmd/probe/internal/binding/postgres/postgres.go b/cmd/probe/internal/binding/postgres/postgres.go index e8c7ca8d1..14b55a7ff 100644 --- a/cmd/probe/internal/binding/postgres/postgres.go +++ b/cmd/probe/internal/binding/postgres/postgres.go @@ -30,7 +30,6 @@ import ( "github.com/jackc/pgx/v5/pgxpool" "github.com/pkg/errors" "github.com/spf13/viper" - "golang.org/x/exp/slices" . "github.com/apecloud/kubeblocks/cmd/probe/internal/binding" . "github.com/apecloud/kubeblocks/cmd/probe/util" @@ -369,128 +368,127 @@ func (pgOps *PostgresOperations) GetLogger() logger.Logger { } func (pgOps *PostgresOperations) createUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( + object = UserInfo{} opsKind = CreateUserOp - ) - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(createUserTpl, user.UserName, user.Password) } - if len(user.Password) == 0 { - return ErrNoPassword + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("created user: %s, with password: %s", user.UserName, user.Password) } - return nil - } - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(createUserTpl, user.UserName, user.Password) - } - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf("created user: %s, with password: %s", user.UserName, user.Password) + ) + + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndPasswdValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return ExecuteObject(ctx, pgOps, req, opsKind, validFn, sqlTplRend, msgTplRend) + + return ExecuteObject(ctx, pgOps, req, opsKind, sqlTplRend, msgTplRend, object) } func (pgOps *PostgresOperations) deleteUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = CreateUserOp + var ( + object = UserInfo{} + opsKind = CreateUserOp + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(dropUserTpl, user.UserName) + } + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("deleted user: %s", user.UserName) + } ) - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName - } - return nil - } - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(dropUserTpl, user.UserName) - } - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf("deleted user: %s", user.UserName) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return ExecuteObject(ctx, pgOps, req, opsKind, validFn, sqlTplRend, msgTplRend) + return ExecuteObject(ctx, pgOps, req, opsKind, sqlTplRend, msgTplRend, object) } func (pgOps *PostgresOperations) grantUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( succMsgTpl = "role %s granted to user: %s" ) return pgOps.managePrivillege(ctx, req, GrantUserRoleOp, grantTpl, succMsgTpl) } func (pgOps *PostgresOperations) revokeUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( succMsgTpl = "role %s revoked from user: %s" ) return pgOps.managePrivillege(ctx, req, RevokeUserRoleOp, revokeTpl, succMsgTpl) } func (pgOps *PostgresOperations) managePrivillege(ctx context.Context, req *bindings.InvokeRequest, op bindings.OperationKind, sqlTpl string, succMsgTpl string) (OpsResult, error) { - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName - } - if len(user.RoleName) == 0 { - return ErrNoRoleName - } - roles := []string{ReadOnlyRole, ReadWriteRole, SuperUserRole} - if !slices.Contains(roles, strings.ToLower(user.RoleName)) { - return ErrInvalidRoleName + var ( + object = UserInfo{} + + sqlTplRend = func(user UserInfo) string { + if user.RoleName == SuperUserRole { + if op == GrantUserRoleOp { + return "ALTER USER " + user.UserName + " WITH SUPERUSER;" + } else { + return "ALTER USER " + user.UserName + " WITH NOSUPERUSER;" + } + } + roleDesc, _ := pgOps.renderRoleByName(user.RoleName) + return fmt.Sprintf(sqlTpl, roleDesc, user.UserName) } - return nil - } - sqlTplRend := func(user UserInfo) string { - if user.RoleName == SuperUserRole { - if op == GrantUserRoleOp { - return "ALTER USER " + user.UserName + " WITH SUPERUSER;" - } else { - return "ALTER USER " + user.UserName + " WITH NOSUPERUSER;" - } + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf(succMsgTpl, user.RoleName, user.UserName) } - roleDesc, _ := pgOps.renderRoleByName(user.RoleName) - return fmt.Sprintf(sqlTpl, roleDesc, user.UserName) - } + ) - msgTplRend := func(user UserInfo) string { - return fmt.Sprintf(succMsgTpl, user.RoleName, user.UserName) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndRoleValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return ExecuteObject(ctx, pgOps, req, op, validFn, sqlTplRend, msgTplRend) + return ExecuteObject(ctx, pgOps, req, op, sqlTplRend, msgTplRend, object) } func (pgOps *PostgresOperations) listUsersOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( - opsKind = ListUsersOp + var ( + opsKind = ListUsersOp + sqlTplRend = func(user UserInfo) string { + return listUserTpl + } ) - sqlTplRend := func(user UserInfo) string { - return listUserTpl - } - return QueryObject(ctx, pgOps, req, opsKind, nil, sqlTplRend, pgUserRolesProcessor) + return QueryObject(ctx, pgOps, req, opsKind, sqlTplRend, pgUserRolesProcessor, UserInfo{}) } func (pgOps *PostgresOperations) describeUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { - const ( + var ( + object = UserInfo{} opsKind = DescribeUserOp - ) - validFn := func(user UserInfo) error { - if len(user.UserName) == 0 { - return ErrNoUserName + sqlTplRend = func(user UserInfo) string { + return fmt.Sprintf(descUserTpl, user.UserName) } - return nil - } + ) - sqlTplRend := func(user UserInfo) string { - return fmt.Sprintf(descUserTpl, user.UserName) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil } - return QueryObject(ctx, pgOps, req, opsKind, validFn, sqlTplRend, pgUserRolesProcessor) + return QueryObject(ctx, pgOps, req, opsKind, sqlTplRend, pgUserRolesProcessor, object) } // post-processing -func pgUserRolesProcessor(data []byte) (interface{}, error) { +func pgUserRolesProcessor(data interface{}) (interface{}, error) { type pgUserInfo struct { UserName string `json:"username"` Expired bool `json:"expired"` @@ -499,7 +497,7 @@ func pgUserRolesProcessor(data []byte) (interface{}, error) { } // parse data to struct var pgUsers []pgUserInfo - err := json.Unmarshal(data, &pgUsers) + err := json.Unmarshal(data.([]byte), &pgUsers) if err != nil { return nil, err } diff --git a/cmd/probe/internal/binding/redis/redis.go b/cmd/probe/internal/binding/redis/redis.go index 9f521b2c0..16bac29e0 100644 --- a/cmd/probe/internal/binding/redis/redis.go +++ b/cmd/probe/internal/binding/redis/redis.go @@ -18,47 +18,126 @@ package redis import ( "context" + "encoding/json" "fmt" + "strconv" + "strings" + "sync" - "github.com/go-redis/redis/v8" - "github.com/pkg/errors" + "github.com/redis/go-redis/v9" + "golang.org/x/exp/slices" - "github.com/dapr/components-contrib/bindings" + bindings "github.com/dapr/components-contrib/bindings" "github.com/dapr/kit/logger" + . "github.com/apecloud/kubeblocks/cmd/probe/internal/binding" rediscomponent "github.com/apecloud/kubeblocks/cmd/probe/internal/component/redis" + . "github.com/apecloud/kubeblocks/cmd/probe/util" +) + +var ( + redisPreDefinedUsers = []string{ + "default", + "kbadmin", + "kbdataprotection", + "kbmonitoring", + "kbprobe", + "kbreplicator", + } ) // Redis is a redis output binding. type Redis struct { client redis.UniversalClient clientSettings *rediscomponent.Settings - logger logger.Logger + mu sync.Mutex ctx context.Context cancel context.CancelFunc + + BaseOperations } +var _ BaseInternalOps = &Redis{} + // NewRedis returns a new redis bindings instance. func NewRedis(logger logger.Logger) bindings.OutputBinding { - return &Redis{logger: logger} + return &Redis{BaseOperations: BaseOperations{Logger: logger}} } // Init performs metadata parsing and connection creation. func (r *Redis) Init(meta bindings.Metadata) (err error) { - r.client, r.clientSettings, err = rediscomponent.ParseClientFromProperties(meta.Properties, nil) + r.BaseOperations.Init(meta) + + r.Logger.Debug("Initializing Redis binding") + r.DBType = "redis" + r.InitIfNeed = r.initIfNeed + r.BaseOperations.GetRole = r.GetRole + + // register redis operations + r.RegisterOperation(bindings.CreateOperation, r.createOps) + r.RegisterOperation(bindings.DeleteOperation, r.deleteOps) + r.RegisterOperation(bindings.GetOperation, r.getOps) + + // following are ops for account management + r.RegisterOperation(ListUsersOp, r.listUsersOps) + r.RegisterOperation(CreateUserOp, r.createUserOps) + r.RegisterOperation(DeleteUserOp, r.deleteUserOps) + r.RegisterOperation(DescribeUserOp, r.describeUserOps) + r.RegisterOperation(GrantUserRoleOp, r.grantUserRoleOps) + r.RegisterOperation(RevokeUserRoleOp, r.revokeUserRoleOps) + + return nil +} + +func (r *Redis) GetRunningPort() int { + // parse port from host + if r.clientSettings != nil { + host := r.clientSettings.Host + if strings.Contains(host, ":") { + parts := strings.Split(host, ":") + if len(parts) == 2 { + port, _ := strconv.Atoi(parts[1]) + return port + } + } + } + return 0 +} + +func (r *Redis) initIfNeed() bool { + if r.client == nil { + go func() { + if err := r.initDelay(); err != nil { + r.Logger.Errorf("redis connection init failed: %v", err) + } else { + r.Logger.Info("redis connection init succeed.") + } + }() + return true + } + return false +} + +func (r *Redis) initDelay() error { + r.mu.Lock() + defer r.mu.Unlock() + if r.client != nil { + return nil + } + var err error + r.client, r.clientSettings, err = rediscomponent.ParseClientFromProperties(r.Metadata.Properties, nil) if err != nil { return err } r.ctx, r.cancel = context.WithCancel(context.Background()) - - _, err = r.client.Ping(r.ctx).Result() + err = r.Ping() if err != nil { return fmt.Errorf("redis binding: error connecting to redis at %s: %s", r.clientSettings.Host, err) } - - return err + r.DBPort = r.GetRunningPort() + return nil } func (r *Redis) Ping() error { @@ -69,41 +148,314 @@ func (r *Redis) Ping() error { return nil } -func (r *Redis) Operations() []bindings.OperationKind { - return []bindings.OperationKind{ - bindings.CreateOperation, - bindings.DeleteOperation, - bindings.GetOperation, +// GetLogger returns the logger, implements BaseInternalOps interface. +func (r *Redis) GetLogger() logger.Logger { + return r.Logger +} + +// InternalQuery is used for internal query, implement BaseInternalOps interface. +func (r *Redis) InternalQuery(ctx context.Context, cmd string) ([]byte, error) { + redisArgs := tokenizeCmd2Args(cmd) + result, err := r.query(ctx, redisArgs...) + if err != nil { + return nil, err } + return json.Marshal(result) } -func (r *Redis) Invoke(ctx context.Context, req *bindings.InvokeRequest) (*bindings.InvokeResponse, error) { - if key, ok := req.Metadata["key"]; ok && key != "" { - switch req.Operation { - case bindings.DeleteOperation: - err := r.client.Del(ctx, key).Err() - if err != nil { - return nil, err +// InternalExec is used for internal execution, implement BaseInternalOps interface. +func (r *Redis) InternalExec(ctx context.Context, cmd string) (int64, error) { + // split command into array of args + redisArgs := tokenizeCmd2Args(cmd) + return 0, r.exec(ctx, redisArgs...) +} + +func (r *Redis) exec(ctx context.Context, args ...interface{}) error { + return r.client.Do(ctx, args...).Err() +} + +func (r *Redis) query(ctx context.Context, args ...interface{}) (interface{}, error) { + // parse result into an slice of string + return r.client.Do(ctx, args...).Result() +} + +func (r *Redis) createOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = RedisEntry{} + + cmdRender = func(redis RedisEntry) string { + return fmt.Sprintf("SET %s %s", redis.Key, redis.Data) + } + msgRender = func(redis RedisEntry) string { + return fmt.Sprintf("SET key : %s", redis.Key) + } + ) + + if err := ParseObjFromRequest(req, defaultRedisEntryParser, defaultRedisEntryValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + return ExecuteObject(ctx, r, req, bindings.CreateOperation, cmdRender, msgRender, object) +} + +func (r *Redis) deleteOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = RedisEntry{} + cmdRender = func(redis RedisEntry) string { + return fmt.Sprintf("DEL %s", redis.Key) + } + msgRender = func(redis RedisEntry) string { + return fmt.Sprintf("deleted key: %s", redis.Key) + } + ) + if err := ParseObjFromRequest(req, defaultRedisEntryParser, defaultRedisEntryValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + + return ExecuteObject(ctx, r, req, bindings.DeleteOperation, cmdRender, msgRender, object) +} + +func (r *Redis) getOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = RedisEntry{} + cmdRender = func(redis RedisEntry) string { + return fmt.Sprintf("GET %s", redis.Key) + } + ) + if err := ParseObjFromRequest(req, defaultRedisEntryParser, defaultRedisEntryValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + return QueryObject(ctx, r, req, bindings.GetOperation, cmdRender, nil, object) +} + +func (r *Redis) listUsersOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + dataProcessor := func(data interface{}) (interface{}, error) { + // data is an array of interface{} of string + // parse redis user name and roles + results := make([]string, 0) + err := json.Unmarshal(data.([]byte), &results) + if err != nil { + return nil, err + } + users := make([]UserInfo, 0) + for _, userInfo := range results { + userName := strings.TrimSpace(userInfo) + if slices.Contains(redisPreDefinedUsers, userName) { + continue } - case bindings.GetOperation: - data, err := r.client.Get(ctx, key).Result() + user := UserInfo{UserName: userName} + users = append(users, user) + } + if jsonData, err := json.Marshal(users); err != nil { + return nil, err + } else { + return string(jsonData), nil + } + } + + cmdRender := func(user UserInfo) string { + return "ACL USERS" + } + + return QueryObject(ctx, r, req, ListUsersOp, cmdRender, dataProcessor, UserInfo{}) +} + +func (r *Redis) describeUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = UserInfo{} + dataProcessor = func(data interface{}) (interface{}, error) { + redisUserPrivContxt := []string{"commands", "keys", "channels", "selectors"} + redisUserInfoContext := []string{"flags", "passwords"} + + profile := make(map[string]string, 0) + results := make([]interface{}, 0) + err := json.Unmarshal(data.([]byte), &results) if err != nil { return nil, err } - rep := &bindings.InvokeResponse{} - rep.Data = []byte(data) - return rep, nil - case bindings.CreateOperation: - _, err := r.client.Do(ctx, "SET", key, req.Data).Result() - if err != nil { + + var context string + for i := 0; i < len(results); i++ { + result := results[i] + switch result := result.(type) { + case string: + strVal := strings.TrimSpace(result) + if len(strVal) == 0 { + continue + } + if slices.Contains(redisUserInfoContext, strVal) { + i++ + continue + } + if slices.Contains(redisUserPrivContxt, strVal) { + context = strVal + } else { + profile[context] = strVal + } + case []interface{}: + selectors := make([]string, 0) + for _, sel := range result { + selectors = append(selectors, sel.(string)) + } + profile[context] = strings.Join(selectors, " ") + } + } + + users := make([]UserInfo, 0) + user := UserInfo{ + UserName: object.UserName, + RoleName: redisPriv2RoleName(profile["commands"] + " " + profile["keys"]), + } + users = append(users, user) + if jsonData, err := json.Marshal(users); err != nil { return nil, err + } else { + return string(jsonData), nil } - default: - return nil, fmt.Errorf("invalid operation type: %s", req.Operation) } - return nil, nil + cmdRender = func(user UserInfo) string { + return fmt.Sprintf("ACL GETUSER %s", user.UserName) + } + ) + + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + + return QueryObject(ctx, r, req, DescribeUserOp, cmdRender, dataProcessor, object) +} + +func (r *Redis) createUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = UserInfo{} + + cmdRender = func(user UserInfo) string { + return fmt.Sprintf("ACL SETUSER %s >%s", user.UserName, user.Password) + } + + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("created user: %s, with password: %s", user.UserName, user.Password) + } + ) + + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndPasswdValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + + return ExecuteObject(ctx, r, req, CreateUserOp, cmdRender, msgTplRend, object) +} + +func (r *Redis) deleteUserOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + object = UserInfo{} + cmdRender = func(user UserInfo) string { + return fmt.Sprintf("ACL DELUSER %s", user.UserName) + } + + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf("deleted user: %s", user.UserName) + } + ) + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + + return ExecuteObject(ctx, r, req, DeleteUserOp, cmdRender, msgTplRend, object) +} + +func (r *Redis) grantUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + succMsgTpl = "role %s granted to user: %s" + ) + return r.managePrivillege(ctx, req, GrantUserRoleOp, succMsgTpl) +} + +func (r *Redis) revokeUserRoleOps(ctx context.Context, req *bindings.InvokeRequest, resp *bindings.InvokeResponse) (OpsResult, error) { + var ( + succMsgTpl = "role %s revoked from user: %s" + ) + return r.managePrivillege(ctx, req, RevokeUserRoleOp, succMsgTpl) +} + +func (r *Redis) managePrivillege(ctx context.Context, req *bindings.InvokeRequest, op bindings.OperationKind, succMsgTpl string) (OpsResult, error) { + var ( + object = UserInfo{} + + cmdRend = func(user UserInfo) string { + command := roleName2RedisPriv(op, user.RoleName) + return fmt.Sprintf("ACL SETUSER %s %s", user.UserName, command) + } + + msgTplRend = func(user UserInfo) string { + return fmt.Sprintf(succMsgTpl, user.RoleName, user.UserName) + } + ) + + if err := ParseObjFromRequest(req, DefaultUserInfoParser, UserNameAndRoleValidator, &object); err != nil { + result := OpsResult{} + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + return result, nil + } + + return ExecuteObject(ctx, r, req, op, cmdRend, msgTplRend, object) +} + +func roleName2RedisPriv(op bindings.OperationKind, roleName string) string { + const ( + grantPrefix = "+" + revokePrefix = "-" + ) + var prefix string + if op == GrantUserRoleOp { + prefix = grantPrefix + } else { + prefix = revokePrefix + } + var command string + switch roleName { + case ReadOnlyRole: + command = fmt.Sprintf("-@all %s@read allkeys", prefix) + case ReadWriteRole: + command = fmt.Sprintf("-@all %s@write %s@read allkeys", prefix, prefix) + case SuperUserRole: + command = fmt.Sprintf("%s@all allkeys", prefix) + } + return command +} + +func redisPriv2RoleName(commands string) string { + if commands == "-@all" { + return NoPrivileges + } + + switch commands { + case "-@all +@read ~*": + return ReadOnlyRole + case "-@all +@write +@read ~*": + return ReadWriteRole + case "+@all ~*": + return SuperUserRole + default: + return CustomizedRole } - return nil, errors.New("redis binding: missing key in request metadata") } func (r *Redis) Close() error { @@ -111,3 +463,54 @@ func (r *Redis) Close() error { return r.client.Close() } + +func (r *Redis) GetRole(ctx context.Context, request *bindings.InvokeRequest, response *bindings.InvokeResponse) (string, error) { + // sql exec timeout need to be less than httpget's timeout which default is 1s. + // ctx1, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond) + // defer cancel() + ctx1 := ctx + section := "Replication" + + var role string + result, err := r.client.Info(ctx1, section).Result() + if err != nil { + r.Logger.Errorf("Role query error: %v", err) + return role, err + } else { + // split the result into lines + lines := strings.Split(result, "\r\n") + // find the line with role + for _, line := range lines { + if strings.HasPrefix(line, "role:") { + role = strings.Split(line, ":")[1] + break + } + } + } + return role, nil +} + +func defaultRedisEntryParser(req *bindings.InvokeRequest, object *RedisEntry) error { + if req == nil || req.Metadata == nil { + return fmt.Errorf("no metadata provided") + } + object.Key = req.Metadata["key"] + object.Data = req.Data + return nil +} + +func defaultRedisEntryValidator(redis RedisEntry) error { + if len(redis.Key) == 0 { + return fmt.Errorf("redis binding: missing key in request metadata") + } + return nil +} + +func tokenizeCmd2Args(cmd string) []interface{} { + args := strings.Split(cmd, " ") + redisArgs := make([]interface{}, 0, len(args)) + for _, arg := range args { + redisArgs = append(redisArgs, arg) + } + return redisArgs +} diff --git a/cmd/probe/internal/binding/redis/redis_test.go b/cmd/probe/internal/binding/redis/redis_test.go index fede4403d..5280295ad 100644 --- a/cmd/probe/internal/binding/redis/redis_test.go +++ b/cmd/probe/internal/binding/redis/redis_test.go @@ -18,106 +18,512 @@ package redis import ( "context" + "encoding/json" + "fmt" + "strings" "testing" - "github.com/alicebob/miniredis/v2" - "github.com/go-redis/redis/v8" "github.com/stretchr/testify/assert" "github.com/dapr/components-contrib/bindings" "github.com/dapr/kit/logger" + + redismock "github.com/go-redis/redismock/v9" + + . "github.com/apecloud/kubeblocks/cmd/probe/internal/binding" + . "github.com/apecloud/kubeblocks/cmd/probe/util" ) const ( - testData = `{"data":"data"}` - testKey = "test" -) + testData = `{"data":"data"}` + testKey = "test" + redisHost = "127.0.0.1:6379" -func TestInvokeCreate(t *testing.T) { - s, c := setupMiniredis() - defer s.Close() + userName = "kiminonawa" + password = "moss" + roleName = ReadWriteRole +) - bind := &Redis{ - client: c, - logger: logger.NewLogger("test"), - } - bind.ctx, bind.cancel = context.WithCancel(context.Background()) +type redisTestCase struct { + testName string + testMetaData map[string]string + expectEveType string + expectEveMsg string +} - _, err := c.Do(context.Background(), "GET", testKey).Result() - assert.Equal(t, redis.Nil, err) +func TestRedisInit(t *testing.T) { + r, _ := mockRedisOps(t) + defer r.Close() + // make sure operations are inited + assert.NotNil(t, r.client) + assert.NotNil(t, r.OperationMap[ListUsersOp]) + assert.NotNil(t, r.OperationMap[CreateUserOp]) + assert.NotNil(t, r.OperationMap[DeleteUserOp]) + assert.NotNil(t, r.OperationMap[DescribeUserOp]) + assert.NotNil(t, r.OperationMap[GrantUserRoleOp]) + assert.NotNil(t, r.OperationMap[RevokeUserRoleOp]) +} +func TestRedisInvokeCreate(t *testing.T) { + r, mock := mockRedisOps(t) + defer r.Close() - bindingRes, err := bind.Invoke(context.TODO(), &bindings.InvokeRequest{ + result := OpsResult{} + request := &bindings.InvokeRequest{ Data: []byte(testData), Metadata: map[string]string{"key": testKey}, Operation: bindings.CreateOperation, - }) - assert.Equal(t, nil, err) - assert.Equal(t, true, bindingRes == nil) + } + // mock expectation + mock.ExpectDo("SET", testKey, testData).SetVal("ok") - getRes, err := c.Do(context.Background(), "GET", testKey).Result() + // invoke + bindingRes, err := r.Invoke(context.TODO(), request) assert.Equal(t, nil, err) - assert.Equal(t, true, getRes == testData) + assert.NotNil(t, bindingRes) + assert.NotNil(t, bindingRes.Data) + + err = json.Unmarshal(bindingRes.Data, &result) + assert.Nil(t, err) + assert.Equal(t, RespEveSucc, result[RespTypEve], result[RespTypMsg]) } -func TestInvokeGet(t *testing.T) { - s, c := setupMiniredis() - defer s.Close() +func TestRedisInvokeGet(t *testing.T) { + r, mock := mockRedisOps(t) + defer r.Close() - bind := &Redis{ - client: c, - logger: logger.NewLogger("test"), + opsResult := OpsResult{} + request := &bindings.InvokeRequest{ + Metadata: map[string]string{"key": testKey}, + Operation: bindings.GetOperation, } - bind.ctx, bind.cancel = context.WithCancel(context.Background()) + // mock expectation, set to nil + mock.ExpectDo("GET", testKey).RedisNil() + mock.ExpectDo("GET", testKey).SetVal(testData) - _, err := c.Do(context.Background(), "SET", testKey, testData).Result() - assert.Equal(t, nil, err) + // invoke create + bindingRes, err := r.Invoke(context.TODO(), request) + assert.Nil(t, err) + assert.NotNil(t, bindingRes) + assert.NotNil(t, bindingRes.Data) + err = json.Unmarshal(bindingRes.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, RespEveFail, opsResult[RespTypEve]) - bindingRes, err := bind.Invoke(context.TODO(), &bindings.InvokeRequest{ + // invoke one more time + bindingRes, err = r.Invoke(context.TODO(), request) + assert.Nil(t, err) + assert.NotNil(t, bindingRes.Data) + err = json.Unmarshal(bindingRes.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, RespEveSucc, opsResult[RespTypEve]) + var o1 interface{} + _ = json.Unmarshal([]byte(opsResult[RespTypMsg].(string)), &o1) + assert.Equal(t, testData, o1) +} + +func TestRedisInvokeDelete(t *testing.T) { + r, mock := mockRedisOps(t) + defer r.Close() + + opsResult := OpsResult{} + request := &bindings.InvokeRequest{ Metadata: map[string]string{"key": testKey}, - Operation: bindings.GetOperation, - }) - assert.Equal(t, nil, err) - assert.Equal(t, true, string(bindingRes.Data) == testData) + Operation: bindings.DeleteOperation, + } + // mock expectation, set to err + mock.ExpectDo("DEL", testKey).SetVal("ok") + + // invoke delete + bindingRes, err := r.Invoke(context.TODO(), request) + assert.Nil(t, err) + assert.NotNil(t, bindingRes) + assert.NotNil(t, bindingRes.Data) + err = json.Unmarshal(bindingRes.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, RespEveSucc, opsResult[RespTypEve]) } -func TestInvokeDelete(t *testing.T) { - s, c := setupMiniredis() - defer s.Close() +func TestRedisGetRoles(t *testing.T) { + r, mock := mockRedisOps(t) + defer r.Close() - bind := &Redis{ - client: c, - logger: logger.NewLogger("test"), + opsResult := OpsResult{} + request := &bindings.InvokeRequest{ + Operation: GetRoleOperation, } - bind.ctx, bind.cancel = context.WithCancel(context.Background()) - _, err := c.Do(context.Background(), "SET", testKey, testData).Result() - assert.Equal(t, nil, err) + // mock expectation, set to err + mock.ExpectInfo("Replication").SetVal("role:master\r\nconnected_slaves:1") + mock.ExpectInfo("Replication").SetVal("role:slave\r\nmaster_port:6379") + // invoke request + bindingRes, err := r.Invoke(context.TODO(), request) + assert.Nil(t, err) + assert.NotNil(t, bindingRes) + assert.NotNil(t, bindingRes.Data) + err = json.Unmarshal(bindingRes.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, RespEveSucc, opsResult[RespTypEve]) + assert.Equal(t, "master", opsResult["role"]) - getRes, err := c.Do(context.Background(), "GET", testKey).Result() - assert.Equal(t, nil, err) - assert.Equal(t, true, getRes == testData) + // invoke one more time + bindingRes, err = r.Invoke(context.TODO(), request) + assert.Nil(t, err) + err = json.Unmarshal(bindingRes.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, RespEveSucc, opsResult[RespTypEve]) + assert.Equal(t, "slave", opsResult["role"]) +} - _, err = bind.Invoke(context.TODO(), &bindings.InvokeRequest{ - Metadata: map[string]string{"key": testKey}, - Operation: bindings.DeleteOperation, +func TestRedisAccounts(t *testing.T) { + // prepare + r, mock := mockRedisOps(t) + defer r.Close() + + ctx := context.TODO() + // list accounts + t.Run("List Accounts", func(t *testing.T) { + mock.ExpectDo("ACL", "USERS").SetVal([]string{"ape", "default", "kbadmin"}) + + response, err := r.Invoke(ctx, &bindings.InvokeRequest{ + Operation: ListUsersOp, + }) + + assert.Nil(t, err) + assert.NotNil(t, response) + assert.NotNil(t, response.Data) + // parse result + opsResult := OpsResult{} + _ = json.Unmarshal(response.Data, &opsResult) + assert.Equal(t, RespEveSucc, opsResult[RespTypEve], opsResult[RespTypMsg]) + + users := make([]UserInfo, 0) + err = json.Unmarshal([]byte(opsResult[RespTypMsg].(string)), &users) + assert.Nil(t, err) + assert.NotEmpty(t, users) + user := users[0] + assert.Equal(t, "ape", user.UserName) + mock.ClearExpect() }) - assert.Equal(t, nil, err) + // create accounts + t.Run("Create Accounts", func(t *testing.T) { + + var ( + err error + opsResult = OpsResult{} + response *bindings.InvokeResponse + request = &bindings.InvokeRequest{ + Operation: CreateUserOp, + } + ) + + testCases := []redisTestCase{ + { + testName: "emptymeta", + testMetaData: map[string]string{}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "nousername", + testMetaData: map[string]string{"password": "moli"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "nopasswd", + testMetaData: map[string]string{"userName": "namae"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoPassword.Error(), + }, + { + testName: "validInput", + testMetaData: map[string]string{ + "userName": userName, + "password": password, + }, + expectEveType: RespEveSucc, + expectEveMsg: fmt.Sprintf("created user: %s", userName), + }, + } + // mock a user + mock.ExpectDo("ACL", "SETUSER", userName, ">"+password).SetVal("ok") + + for _, accTest := range testCases { + request.Metadata = accTest.testMetaData + response, err = r.Invoke(ctx, request) + assert.Nil(t, err) + assert.NotNil(t, response.Data) + err = json.Unmarshal(response.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, accTest.expectEveType, opsResult[RespTypEve], opsResult[RespTypMsg]) + assert.Contains(t, opsResult[RespTypMsg], accTest.expectEveMsg) + } + mock.ClearExpect() + }) + // grant and revoke role + t.Run("Grant Accounts", func(t *testing.T) { + + var ( + err error + opsResult = OpsResult{} + response *bindings.InvokeResponse + ) + + testCases := []redisTestCase{ + { + testName: "emptymeta", + testMetaData: map[string]string{}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "nousername", + testMetaData: map[string]string{"password": "moli"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "norolename", + testMetaData: map[string]string{"userName": "namae"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoRoleName.Error(), + }, + { + testName: "invalidRoleName", + testMetaData: map[string]string{"userName": "namae", "roleName": "superman"}, + expectEveType: RespEveFail, + expectEveMsg: ErrInvalidRoleName.Error(), + }, + { + testName: "validInput", + testMetaData: map[string]string{ + "userName": userName, + "roleName": roleName, + }, + expectEveType: RespEveSucc, + }, + } + + for _, ops := range []bindings.OperationKind{GrantUserRoleOp, RevokeUserRoleOp} { + // mock exepctation + args := tokenizeCmd2Args(fmt.Sprintf("ACL SETUSER %s %s", userName, roleName2RedisPriv(ops, roleName))) + mock.ExpectDo(args...).SetVal("ok") + + request := &bindings.InvokeRequest{ + Operation: ops, + } + for _, accTest := range testCases { + request.Metadata = accTest.testMetaData + response, err = r.Invoke(ctx, request) + assert.Nil(t, err) + assert.NotNil(t, response.Data) + err = json.Unmarshal(response.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, accTest.expectEveType, opsResult[RespTypEve], opsResult[RespTypMsg]) + if len(accTest.expectEveMsg) > 0 { + assert.Contains(t, accTest.expectEveMsg, opsResult[RespTypMsg]) + } + } + } + mock.ClearExpect() + }) + + // desc accounts + t.Run("Desc Accounts", func(t *testing.T) { + var ( + err error + opsResult = OpsResult{} + response *bindings.InvokeResponse + request = &bindings.InvokeRequest{ + Operation: DescribeUserOp, + } + // mock a user, describing it as an array of interface{} + userInfo = []interface{}{ + "flags", + []interface{}{"on"}, + "passwords", + []interface{}{"mock-password"}, + "commands", + "+@all", + "keys", + "~*", + "channels", + "", + "selectors", + []interface{}{}, + } + ) - rgetRep, err := c.Do(context.Background(), "GET", testKey).Result() - assert.Equal(t, redis.Nil, err) - assert.Equal(t, nil, rgetRep) + testCases := []redisTestCase{ + { + testName: "emptymeta", + testMetaData: map[string]string{}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "nousername", + testMetaData: map[string]string{"password": "moli"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "validInputButNil", + testMetaData: map[string]string{ + "userName": userName, + }, + expectEveType: RespEveFail, + expectEveMsg: "redis: nil", + }, + { + testName: "validInput", + testMetaData: map[string]string{ + "userName": userName, + }, + expectEveType: RespEveSucc, + }, + } + + mock.ExpectDo("ACL", "GETUSER", userName).RedisNil() + mock.ExpectDo("ACL", "GETUSER", userName).SetVal(userInfo) + + for _, accTest := range testCases { + request.Metadata = accTest.testMetaData + response, err = r.Invoke(ctx, request) + assert.Nil(t, err) + assert.NotNil(t, response.Data) + err = json.Unmarshal(response.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, accTest.expectEveType, opsResult[RespTypEve], opsResult[RespTypMsg]) + if len(accTest.expectEveMsg) > 0 { + assert.Contains(t, opsResult[RespTypMsg], accTest.expectEveMsg) + } + if RespEveSucc == opsResult[RespTypEve] { + // parse user info + users := make([]UserInfo, 0) + err = json.Unmarshal([]byte(opsResult[RespTypMsg].(string)), &users) + assert.Nil(t, err) + assert.Len(t, users, 1) + user := users[0] + assert.Equal(t, userName, user.UserName) + assert.Equal(t, SuperUserRole, user.RoleName) + } + } + mock.ClearExpect() + }) + // delete accounts + t.Run("Delete Accounts", func(t *testing.T) { + + var ( + err error + opsResult = OpsResult{} + response *bindings.InvokeResponse + request = &bindings.InvokeRequest{ + Operation: DeleteUserOp, + } + ) + + testCases := []redisTestCase{ + { + testName: "emptymeta", + testMetaData: map[string]string{}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "nousername", + testMetaData: map[string]string{"password": "moli"}, + expectEveType: RespEveFail, + expectEveMsg: ErrNoUserName.Error(), + }, + { + testName: "validInput", + testMetaData: map[string]string{ + "userName": userName, + }, + expectEveType: RespEveSucc, + expectEveMsg: fmt.Sprintf("deleted user: %s", userName), + }, + } + // mock a user + mock.ExpectDo("ACL", "DELUSER", userName).SetVal("ok") + + for _, accTest := range testCases { + request.Metadata = accTest.testMetaData + response, err = r.Invoke(ctx, request) + assert.Nil(t, err) + assert.NotNil(t, response.Data) + err = json.Unmarshal(response.Data, &opsResult) + assert.Nil(t, err) + assert.Equal(t, accTest.expectEveType, opsResult[RespTypEve], opsResult[RespTypMsg]) + assert.Contains(t, opsResult[RespTypMsg], accTest.expectEveMsg) + } + mock.ClearExpect() + }) + + t.Run("RoleName Conversion", func(t *testing.T) { + type roleTestCase struct { + roleName string + redisPrivs string + } + grantTestCases := []roleTestCase{ + { + SuperUserRole, + "+@all allkeys", + }, + { + ReadWriteRole, + "-@all +@write +@read allkeys", + }, + { + ReadOnlyRole, + "-@all +@read allkeys", + }, + } + for _, test := range grantTestCases { + cmd := roleName2RedisPriv(GrantUserRoleOp, test.roleName) + assert.Equal(t, test.redisPrivs, cmd) + + // allkeys -> ~* + cmd = strings.Replace(cmd, "allkeys", "~*", 1) + inferredRole := redisPriv2RoleName(cmd) + assert.Equal(t, test.roleName, inferredRole) + } + + revokeTestCases := []roleTestCase{ + { + SuperUserRole, + "-@all allkeys", + }, + { + ReadWriteRole, + "-@all -@write -@read allkeys", + }, + { + ReadOnlyRole, + "-@all -@read allkeys", + }, + } + for _, test := range revokeTestCases { + cmd := roleName2RedisPriv(RevokeUserRoleOp, test.roleName) + assert.Equal(t, test.redisPrivs, cmd) + } + }) } -func setupMiniredis() (*miniredis.Miniredis, *redis.Client) { - s, err := miniredis.Run() - if err != nil { - panic(err) - } - opts := &redis.Options{ - Addr: s.Addr(), - DB: 0, - } +func mockRedisOps(t *testing.T) (*Redis, redismock.ClientMock) { + client, mock := redismock.NewClientMock() - return s, redis.NewClient(opts) + if client == nil || mock == nil { + t.Fatalf("failed to mock a redis client") + return nil, nil + } + r := &Redis{} + r.Logger = logger.NewLogger("test") + r.client = client + r.ctx, r.cancel = context.WithCancel(context.Background()) + _ = r.Init(bindings.Metadata{}) + r.DBPort = 6379 + return r, mock } diff --git a/cmd/probe/internal/binding/utils.go b/cmd/probe/internal/binding/utils.go index 15ffc8fb5..118928fa6 100644 --- a/cmd/probe/internal/binding/utils.go +++ b/cmd/probe/internal/binding/utils.go @@ -20,20 +20,26 @@ import ( "context" "encoding/json" "fmt" + "strings" "time" "github.com/dapr/components-contrib/bindings" + "golang.org/x/exp/slices" ) type UserInfo struct { - UserName string `json:"userName"` - Password string `json:"password,omitempty"` - Expired string `json:"expired,omitempty"` - ExpireAt time.Duration `json:"expireAt,omitempty"` - RoleName string `json:"roleName,omitempty"` + UserName string `json:"userName"` + Password string `json:"password,omitempty"` + Expired string `json:"expired,omitempty"` + RoleName string `json:"roleName,omitempty"` } -type OpsMetadata struct { +type RedisEntry struct { + Key string `json:"key"` + Data []byte `json:"data,omitempty"` +} + +type opsMetadata struct { Operation bindings.OperationKind `json:"operation,omitempty"` StartTime string `json:"startTime,omitempty"` EndTime string `json:"endTime,omitempty"` @@ -41,112 +47,145 @@ type OpsMetadata struct { } // UserDefinedObjectType defines the interface for User Defined Objects. -type UserDefinedObjectType interface { - UserInfo +type customizedObjType interface { + UserInfo | RedisEntry } -// SQLRender defines the interface to render a SQL statement for given object. -type SQLRender[T UserDefinedObjectType] func(object T) string - -// SQLPostProcessor defines what to do after retrieving results from database. -type SQLPostProcessor[T UserDefinedObjectType] func(object T) error +// CmdRender defines the interface to render a statement from given object. +type cmdRender[T customizedObjType] func(object T) string -// UserDefinedObjectValidator defines the interface to validate the User Defined Object. -type UserDefinedObjectValidator[T UserDefinedObjectType] func(object T) error +// resultRender defines the interface to render the data from database. +type resultRender[T customizedObjType] func(interface{}) (interface{}, error) -// DataRender defines the interface to render the data from database. -type DataRender func([]byte) (interface{}, error) +// objectValidator defines the interface to validate the User Defined Object. +type objectValidator[T customizedObjType] func(object T) error -func ParseObjectFromMetadata[T UserDefinedObjectType](metadata map[string]string, object *T, fn UserDefinedObjectValidator[T]) error { - if metadata == nil { - return fmt.Errorf("no metadata provided") - } else if jsonData, err := json.Marshal(metadata); err != nil { - return err - } else if err = json.Unmarshal(jsonData, object); err != nil { - return err - } else if fn != nil { - return fn(*object) - } - return nil -} - -func GetAndFormatNow() string { - return time.Now().Format(time.RFC3339Nano) -} - -func OpsTerminateOnSucc(result OpsResult, metadata OpsMetadata, msg interface{}) (OpsResult, error) { - metadata.EndTime = GetAndFormatNow() - result[RespTypEve] = RespEveSucc - result[RespTypMsg] = msg - result[RespTypMeta] = metadata - return result, nil -} +// objectParser defines the interface to parse the User Defined Object from request. +type objectParser[T customizedObjType] func(req *bindings.InvokeRequest, object *T) error -func OpsTerminateOnErr(result OpsResult, metadata OpsMetadata, err error) (OpsResult, error) { - metadata.EndTime = GetAndFormatNow() - result[RespTypEve] = RespEveFail - result[RespTypMsg] = err.Error() - result[RespTypMeta] = metadata - return result, nil -} - -func ExecuteObject[T UserDefinedObjectType](ctx context.Context, ops BaseInternalOps, req *bindings.InvokeRequest, - opsKind bindings.OperationKind, - validFn UserDefinedObjectValidator[T], - sqlTplRend SQLRender[T], msgTplRend SQLRender[T]) (OpsResult, error) { +func ExecuteObject[T customizedObjType](ctx context.Context, ops BaseInternalOps, req *bindings.InvokeRequest, + opsKind bindings.OperationKind, sqlTplRend cmdRender[T], msgTplRend cmdRender[T], object T) (OpsResult, error) { var ( - result = OpsResult{} - userInfo = T{} - err error + result = OpsResult{} + err error ) - metadata := OpsMetadata{Operation: opsKind, StartTime: GetAndFormatNow()} - // parser userinfo from metadata - if err = ParseObjectFromMetadata(req.Metadata, &userInfo, validFn); err != nil { - return OpsTerminateOnErr(result, metadata, err) - } + metadata := opsMetadata{Operation: opsKind, StartTime: getAndFormatNow()} - sql := sqlTplRend(userInfo) + sql := sqlTplRend(object) metadata.Extra = sql - ops.GetLogger().Debugf("MysqlOperations.execUser() with sql: %s", sql) + ops.GetLogger().Debugf("ExecObject with cmd: %s", sql) + if _, err = ops.InternalExec(ctx, sql); err != nil { - return OpsTerminateOnErr(result, metadata, err) + return opsTerminateOnErr(result, metadata, err) } - return OpsTerminateOnSucc(result, metadata, msgTplRend(userInfo)) + return opsTerminateOnSucc(result, metadata, msgTplRend(object)) } -func QueryObject[T UserDefinedObjectType](ctx context.Context, ops BaseInternalOps, req *bindings.InvokeRequest, - opsKind bindings.OperationKind, - validFn UserDefinedObjectValidator[T], - sqlTplRend SQLRender[T], - dataProcessor DataRender) (OpsResult, error) { +func QueryObject[T customizedObjType](ctx context.Context, ops BaseInternalOps, req *bindings.InvokeRequest, + opsKind bindings.OperationKind, sqlTplRend cmdRender[T], dataProcessor resultRender[T], object T) (OpsResult, error) { var ( - result = OpsResult{} - userInfo = T{} - err error + result = OpsResult{} + err error ) - metadata := OpsMetadata{Operation: opsKind, StartTime: GetAndFormatNow()} - // parser userinfo from metadata - if err := ParseObjectFromMetadata(req.Metadata, &userInfo, validFn); err != nil { - return OpsTerminateOnErr(result, metadata, err) - } + metadata := opsMetadata{Operation: opsKind, StartTime: getAndFormatNow()} - sql := sqlTplRend(userInfo) + sql := sqlTplRend(object) metadata.Extra = sql - ops.GetLogger().Debugf("MysqlOperations.queryUser() with sql: %s", sql) + ops.GetLogger().Debugf("QueryObject() with cmd: %s", sql) jsonData, err := ops.InternalQuery(ctx, sql) if err != nil { - return OpsTerminateOnErr(result, metadata, err) + return opsTerminateOnErr(result, metadata, err) } if dataProcessor == nil { - return OpsTerminateOnSucc(result, metadata, string(jsonData)) + return opsTerminateOnSucc(result, metadata, string(jsonData)) } + if ret, err := dataProcessor(jsonData); err != nil { - return OpsTerminateOnErr(result, metadata, err) + return opsTerminateOnErr(result, metadata, err) } else { - return OpsTerminateOnSucc(result, metadata, ret) + return opsTerminateOnSucc(result, metadata, ret) + } +} + +func ParseObjFromRequest[T customizedObjType](req *bindings.InvokeRequest, parse objectParser[T], validator objectValidator[T], object *T) error { + if req == nil { + return fmt.Errorf("no request provided") + } + if parse != nil { + if err := parse(req, object); err != nil { + return err + } } + if validator != nil { + if err := validator(*object); err != nil { + return err + } + } + return nil +} + +func DefaultUserInfoParser(req *bindings.InvokeRequest, object *UserInfo) error { + if req == nil || req.Metadata == nil { + return fmt.Errorf("no metadata provided") + } else if jsonData, err := json.Marshal(req.Metadata); err != nil { + return err + } else if err = json.Unmarshal(jsonData, object); err != nil { + return err + } + return nil +} + +func UserNameValidator(user UserInfo) error { + if len(user.UserName) == 0 { + return ErrNoUserName + } + return nil +} + +func UserNameAndPasswdValidator(user UserInfo) error { + if len(user.UserName) == 0 { + return ErrNoUserName + } + if len(user.Password) == 0 { + return ErrNoPassword + } + return nil +} + +func UserNameAndRoleValidator(user UserInfo) error { + if len(user.UserName) == 0 { + return ErrNoUserName + } + if len(user.RoleName) == 0 { + return ErrNoRoleName + } + roles := []string{ReadOnlyRole, ReadWriteRole, SuperUserRole} + if !slices.Contains(roles, strings.ToLower(user.RoleName)) { + return ErrInvalidRoleName + } + return nil +} + +func getAndFormatNow() string { + return time.Now().Format(time.RFC3339Nano) +} + +func opsTerminateOnSucc(result OpsResult, metadata opsMetadata, msg interface{}) (OpsResult, error) { + metadata.EndTime = getAndFormatNow() + result[RespTypEve] = RespEveSucc + result[RespTypMsg] = msg + result[RespTypMeta] = metadata + return result, nil +} + +func opsTerminateOnErr(result OpsResult, metadata opsMetadata, err error) (OpsResult, error) { + metadata.EndTime = getAndFormatNow() + result[RespTypEve] = RespEveFail + result[RespTypMsg] = err.Error() + result[RespTypMeta] = metadata + return result, nil } diff --git a/cmd/probe/internal/component/redis/redis.go b/cmd/probe/internal/component/redis/redis.go index 579254224..4c69272fe 100644 --- a/cmd/probe/internal/component/redis/redis.go +++ b/cmd/probe/internal/component/redis/redis.go @@ -22,7 +22,7 @@ import ( "strings" "time" - "github.com/go-redis/redis/v8" + "github.com/redis/go-redis/v9" ) const ( @@ -52,23 +52,20 @@ func newFailoverClient(s *Settings) redis.UniversalClient { return nil } opts := &redis.FailoverOptions{ - DB: s.DB, - MasterName: s.SentinelMasterName, - SentinelAddrs: []string{s.Host}, - Password: s.Password, - Username: s.Username, - MaxRetries: s.RedisMaxRetries, - MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), - MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), - DialTimeout: time.Duration(s.DialTimeout), - ReadTimeout: time.Duration(s.ReadTimeout), - WriteTimeout: time.Duration(s.WriteTimeout), - PoolSize: s.PoolSize, - MaxConnAge: time.Duration(s.MaxConnAge), - MinIdleConns: s.MinIdleConns, - PoolTimeout: time.Duration(s.PoolTimeout), - IdleCheckFrequency: time.Duration(s.IdleCheckFrequency), - IdleTimeout: time.Duration(s.IdleTimeout), + DB: s.DB, + MasterName: s.SentinelMasterName, + SentinelAddrs: []string{s.Host}, + Password: s.Password, + Username: s.Username, + MaxRetries: s.RedisMaxRetries, + MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), + MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), + DialTimeout: time.Duration(s.DialTimeout), + ReadTimeout: time.Duration(s.ReadTimeout), + WriteTimeout: time.Duration(s.WriteTimeout), + PoolSize: s.PoolSize, + MinIdleConns: s.MinIdleConns, + PoolTimeout: time.Duration(s.PoolTimeout), } /* #nosec */ @@ -93,21 +90,18 @@ func newClient(s *Settings) redis.UniversalClient { } if s.RedisType == ClusterType { options := &redis.ClusterOptions{ - Addrs: strings.Split(s.Host, ","), - Password: s.Password, - Username: s.Username, - MaxRetries: s.RedisMaxRetries, - MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), - MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), - DialTimeout: time.Duration(s.DialTimeout), - ReadTimeout: time.Duration(s.ReadTimeout), - WriteTimeout: time.Duration(s.WriteTimeout), - PoolSize: s.PoolSize, - MaxConnAge: time.Duration(s.MaxConnAge), - MinIdleConns: s.MinIdleConns, - PoolTimeout: time.Duration(s.PoolTimeout), - IdleCheckFrequency: time.Duration(s.IdleCheckFrequency), - IdleTimeout: time.Duration(s.IdleTimeout), + Addrs: strings.Split(s.Host, ","), + Password: s.Password, + Username: s.Username, + MaxRetries: s.RedisMaxRetries, + MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), + MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), + DialTimeout: time.Duration(s.DialTimeout), + ReadTimeout: time.Duration(s.ReadTimeout), + WriteTimeout: time.Duration(s.WriteTimeout), + PoolSize: s.PoolSize, + MinIdleConns: s.MinIdleConns, + PoolTimeout: time.Duration(s.PoolTimeout), } /* #nosec */ if s.EnableTLS { @@ -120,22 +114,19 @@ func newClient(s *Settings) redis.UniversalClient { } options := &redis.Options{ - Addr: s.Host, - Password: s.Password, - Username: s.Username, - DB: s.DB, - MaxRetries: s.RedisMaxRetries, - MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), - MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), - DialTimeout: time.Duration(s.DialTimeout), - ReadTimeout: time.Duration(s.ReadTimeout), - WriteTimeout: time.Duration(s.WriteTimeout), - PoolSize: s.PoolSize, - MaxConnAge: time.Duration(s.MaxConnAge), - MinIdleConns: s.MinIdleConns, - PoolTimeout: time.Duration(s.PoolTimeout), - IdleCheckFrequency: time.Duration(s.IdleCheckFrequency), - IdleTimeout: time.Duration(s.IdleTimeout), + Addr: s.Host, + Password: s.Password, + Username: s.Username, + DB: s.DB, + MaxRetries: s.RedisMaxRetries, + MaxRetryBackoff: time.Duration(s.RedisMaxRetryInterval), + MinRetryBackoff: time.Duration(s.RedisMinRetryInterval), + DialTimeout: time.Duration(s.DialTimeout), + ReadTimeout: time.Duration(s.ReadTimeout), + WriteTimeout: time.Duration(s.WriteTimeout), + PoolSize: s.PoolSize, + MinIdleConns: s.MinIdleConns, + PoolTimeout: time.Duration(s.PoolTimeout), } /* #nosec */ diff --git a/config/probe/components/binding_redis.yaml b/config/probe/components/binding_redis.yaml new file mode 100644 index 000000000..a0d64f659 --- /dev/null +++ b/config/probe/components/binding_redis.yaml @@ -0,0 +1,12 @@ +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: redis +spec: + type: bindings.redis + version: v1 + metadata: + - name: redisHost + value: 127.0.0.1:6379 + - name: enableTLS + value: false \ No newline at end of file diff --git a/deploy/postgresql/templates/clusterdefinition.yaml b/deploy/postgresql/templates/clusterdefinition.yaml index 55d89e080..2d37e3035 100644 --- a/deploy/postgresql/templates/clusterdefinition.yaml +++ b/deploy/postgresql/templates/clusterdefinition.yaml @@ -16,6 +16,10 @@ spec: workloadType: Replication characterType: postgresql probes: + roleChangedProbe: + failureThreshold: 2 + periodSeconds: 1 + timeoutSeconds: 1 monitor: builtIn: false exporterConfig: diff --git a/go.mod b/go.mod index 54dec3e09..401480ae3 100644 --- a/go.mod +++ b/go.mod @@ -9,7 +9,6 @@ require ( github.com/Masterminds/sprig/v3 v3.2.3 github.com/StudioSol/set v1.0.0 github.com/ahmetalpbalkan/go-cursor v0.0.0-20131010032410-8136607ea412 - github.com/alicebob/miniredis/v2 v2.13.3 github.com/authzed/controller-idioms v0.7.0 github.com/aws/aws-sdk-go v1.44.122 github.com/bhmj/jsonslice v1.1.2 @@ -32,7 +31,7 @@ require ( github.com/go-git/go-git/v5 v5.5.2 github.com/go-logr/logr v1.2.3 github.com/go-logr/zapr v1.2.3 - github.com/go-redis/redis/v8 v8.11.5 + github.com/go-redis/redismock/v9 v9.0.2 github.com/go-sql-driver/mysql v1.7.0 github.com/golang/mock v1.6.0 github.com/google/go-cmp v0.5.9 @@ -47,11 +46,12 @@ require ( github.com/leaanthony/debme v1.2.1 github.com/manifoldco/promptui v0.9.0 github.com/mitchellh/mapstructure v1.5.1-0.20220423185008-bf980b35cac4 - github.com/onsi/ginkgo/v2 v2.6.1 - github.com/onsi/gomega v1.24.2 + github.com/onsi/ginkgo/v2 v2.7.0 + github.com/onsi/gomega v1.25.0 github.com/opencontainers/image-spec v1.1.0-rc2 github.com/pingcap/go-tpc v1.0.9 github.com/pkg/errors v0.9.1 + github.com/redis/go-redis/v9 v9.0.1 github.com/replicatedhq/termui/v3 v3.1.1-0.20200811145416-f40076d26851 github.com/replicatedhq/troubleshoot v0.57.0 github.com/sethvargo/go-password v0.2.0 @@ -73,7 +73,7 @@ require ( go.uber.org/zap v1.24.0 golang.org/x/crypto v0.5.0 golang.org/x/exp v0.0.0-20221028150844-83b7d23a625f - golang.org/x/net v0.5.0 + golang.org/x/net v0.7.0 golang.org/x/sync v0.1.0 golang.org/x/sys v0.5.0 google.golang.org/grpc v1.52.0 @@ -118,7 +118,6 @@ require ( github.com/PuerkitoBio/purell v1.1.1 // indirect github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect github.com/acomagu/bufpipe v1.0.3 // indirect - github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a // indirect github.com/andybalholm/brotli v1.0.4 // indirect github.com/antlr/antlr4/runtime/Go/antlr v1.4.10 // indirect github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect @@ -129,7 +128,7 @@ require ( github.com/c9s/goprocinfo v0.0.0-20170724085704-0010a05ce49f // indirect github.com/cenkalti/backoff v2.2.1+incompatible // indirect github.com/cenkalti/backoff/v4 v4.2.0 // indirect - github.com/cespare/xxhash/v2 v2.1.2 // indirect + github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/chai2010/gettext-go v1.0.2 // indirect github.com/chzyer/readline v1.5.1 // indirect github.com/cloudflare/circl v1.1.0 // indirect @@ -324,7 +323,6 @@ require ( github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect github.com/xlab/treeprint v1.1.0 // indirect github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect - github.com/yuin/gopher-lua v0.0.0-20200603152657-dc2b0ca8b37e // indirect github.com/yusufpapurcu/wmi v1.2.2 // indirect github.com/zclconf/go-cty v1.12.1 // indirect go.etcd.io/bbolt v1.3.6 // indirect @@ -351,8 +349,8 @@ require ( go4.org/unsafe/assume-no-moving-gc v0.0.0-20220617031537-928513b29760 // indirect golang.org/x/mod v0.7.0 // indirect golang.org/x/oauth2 v0.4.0 // indirect - golang.org/x/term v0.4.0 // indirect - golang.org/x/text v0.6.0 // indirect + golang.org/x/term v0.5.0 // indirect + golang.org/x/text v0.7.0 // indirect golang.org/x/time v0.3.0 // indirect golang.org/x/tools v0.4.0 // indirect golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect diff --git a/go.sum b/go.sum index f8df2af34..4aa950629 100644 --- a/go.sum +++ b/go.sum @@ -176,10 +176,6 @@ github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRF github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0= github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho= github.com/alexflint/go-filemutex v0.0.0-20171022225611-72bdc8eae2ae/go.mod h1:CgnQgUtFrFz9mxFNtED3jI5tLDjKlOM+oUF/sTk6ps0= -github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a h1:HbKu58rmZpUGpz5+4FfNmIU+FmZg2P3Xaj2v2bfNWmk= -github.com/alicebob/gopher-json v0.0.0-20200520072559-a9ecdc9d1d3a/go.mod h1:SGnFV6hVsYE877CKEZ6tDNTjaSXYUk6QqoIK6PrAtcc= -github.com/alicebob/miniredis/v2 v2.13.3 h1:kohgdtN58KW/r9ZDVmMJE3MrfbumwsDQStd0LPAGmmw= -github.com/alicebob/miniredis/v2 v2.13.3/go.mod h1:uS970Sw5Gs9/iK3yBg0l9Uj9s25wXxSpQUE9EaJ/Blg= github.com/andybalholm/brotli v1.0.0/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y= github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY= github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig= @@ -237,6 +233,8 @@ github.com/briandowns/spinner v1.23.0 h1:alDF2guRWqa/FOZZYWjlMIx2L6H0wyewPxo/CH4 github.com/briandowns/spinner v1.23.0/go.mod h1:rPG4gmXeN3wQV/TsAY4w8lPdIM6RX3yqeBQJSrbXjuE= github.com/bshuster-repo/logrus-logstash-hook v0.4.1/go.mod h1:zsTqEiSzDgAa/8GZR7E1qaXrhYNDKBYy5/dWPTIflbk= github.com/bshuster-repo/logrus-logstash-hook v1.0.0 h1:e+C0SB5R1pu//O4MQ3f9cFuPGoOVeF2fE4Og9otCc70= +github.com/bsm/ginkgo/v2 v2.5.0 h1:aOAnND1T40wEdAtkGSkvSICWeQ8L3UASX7YVCqQx+eQ= +github.com/bsm/gomega v1.20.0 h1:JhAwLmtRzXFTx2AkALSLa8ijZafntmhSoU63Ok18Uq8= github.com/buger/jsonparser v0.0.0-20180808090653-f4dd9f5a6b44/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s= github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0= github.com/bugsnag/bugsnag-go v0.0.0-20141110184014-b1d153021fcd/go.mod h1:2oa8nejYd4cQ/b0hMIopN0lCRxU0bueqREvZLWFrtK8= @@ -261,8 +259,9 @@ github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054 h1:uH66TXeswKn5P github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE= github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= +github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chai2010/gettext-go v1.0.2 h1:1Lwwip6Q2QGsAdl/ZKPCwTe9fe0CjlUbqj5bFNSjIRk= github.com/chai2010/gettext-go v1.0.2/go.mod h1:y+wnP2cHYaVj19NZhYKAwEMH2CI1gNHeQQ+5AjwawxA= github.com/checkpoint-restore/go-criu/v4 v4.1.0/go.mod h1:xUQBLp4RLc5zJtWY++yjOoMoB5lihDt7fai+75m+rGw= @@ -645,8 +644,8 @@ github.com/go-openapi/swag v0.22.3 h1:yMBqmnQ0gyZvEb/+KzuWZOXgllrXT4SADYbvDaXHv/ github.com/go-openapi/swag v0.22.3/go.mod h1:UzaqsxGiab7freDnrUUra0MwWfN/q7tE4j+VcZ0yl14= github.com/go-redis/redis/v7 v7.4.1 h1:PASvf36gyUpr2zdOUS/9Zqc80GbM+9BDyiJSJDDOrTI= github.com/go-redis/redis/v7 v7.4.1/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg= -github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI= -github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo= +github.com/go-redis/redismock/v9 v9.0.2 h1:1X51FovN18M9GXBdbi5xWiXoFPXAijdLdvA7VrYjoVA= +github.com/go-redis/redismock/v9 v9.0.2/go.mod h1:Ojrqw2Kut8BB8HZlXwNgfwhp5xvtVQTjgbIdIMi980g= github.com/go-sql-driver/mysql v1.3.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w= github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w= github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w= @@ -1247,8 +1246,8 @@ github.com/onsi/ginkgo v1.13.0/go.mod h1:+REjRxOmWfHCjfv9TTWB1jD1Frx4XydAD3zm1ls github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0= github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= -github.com/onsi/ginkgo/v2 v2.6.1 h1:1xQPCjcqYw/J5LchOcp4/2q/jzJFjiAOc25chhnDw+Q= -github.com/onsi/ginkgo/v2 v2.6.1/go.mod h1:yjiuMwPokqY1XauOgju45q3sJt6VzQ/Fict1LFVcsAo= +github.com/onsi/ginkgo/v2 v2.7.0 h1:/XxtEV3I3Eif/HobnVx9YmJgk8ENdRsuUmM+fLCFNow= +github.com/onsi/ginkgo/v2 v2.7.0/go.mod h1:yjiuMwPokqY1XauOgju45q3sJt6VzQ/Fict1LFVcsAo= github.com/onsi/gomega v0.0.0-20151007035656-2152b45fa28a/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= @@ -1258,8 +1257,8 @@ github.com/onsi/gomega v1.9.0/go.mod h1:Ho0h+IUsWyvy1OpqCwxlQ/21gkhVunqlU8fDGcoT github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= github.com/onsi/gomega v1.10.3/go.mod h1:V9xEwhxec5O8UDM77eCW8vLymOMltsqPVYWrpDsH8xc= github.com/onsi/gomega v1.16.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY= -github.com/onsi/gomega v1.24.2 h1:J/tulyYK6JwBldPViHJReihxxZ+22FHs0piGjQAvoUE= -github.com/onsi/gomega v1.24.2/go.mod h1:gs3J10IS7Z7r7eXRoNJIrNqU4ToQukCJhFtKrWgHWnk= +github.com/onsi/gomega v1.25.0 h1:Vw7br2PCDYijJHSfBOWhov+8cAnUf8MfMaIOV323l6Y= +github.com/onsi/gomega v1.25.0/go.mod h1:r+zV744Re+DiYCIPRlYOTxn0YkOLcAnW8k1xXdMPGhM= github.com/opencontainers/go-digest v0.0.0-20170106003457-a6d0ee40d420/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s= github.com/opencontainers/go-digest v0.0.0-20180430190053-c9281466c8b2/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s= github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQYUd2OVphdqWwCJHrFt2s= @@ -1388,6 +1387,8 @@ github.com/quasilyte/go-ruleguard v0.2.0/go.mod h1:2RT/tf0Ce0UDj5y243iWKosQogJd8 github.com/quasilyte/regex/syntax v0.0.0-20200407221936-30656e2c4a95/go.mod h1:rlzQ04UMyJXu/aOvhd8qT+hvDrFpiwqp8MRXDY9szc0= github.com/rabbitmq/amqp091-go v1.1.0/go.mod h1:ogQDLSOACsLPsIq0NpbtiifNZi2YOz0VTJ0kHRghqbM= github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4= +github.com/redis/go-redis/v9 v9.0.1 h1:L1B0L2Y7dQMnKxwfzSwemceGlQwVUsqJ1kjkdaoNhts= +github.com/redis/go-redis/v9 v9.0.1/go.mod h1:/xDTe9EF1LM61hek62Poq2nzQSGj0xSrEtEHbBQevps= github.com/replicatedhq/termui/v3 v3.1.1-0.20200811145416-f40076d26851 h1:eRlNDHxGfVkPCRXbA4BfQJvt5DHjFiTtWy3R/t4djyY= github.com/replicatedhq/termui/v3 v3.1.1-0.20200811145416-f40076d26851/go.mod h1:JDxG6+uubnk9/BZ2yUsyAJJwlptjrnmB2MPF5d2Xe/8= github.com/replicatedhq/troubleshoot v0.57.0 h1:m9B31Mhgiz4Lwz+W4RvFkqhfYZLCwAqRPUwiwmSAAps= @@ -1626,9 +1627,6 @@ github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9dec github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.0/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -github.com/yuin/gopher-lua v0.0.0-20191220021717-ab39c6098bdb/go.mod h1:gqRgreBUhTSL0GeU64rtZ3Uq3wtjOa/TB2YfrtkCbVQ= -github.com/yuin/gopher-lua v0.0.0-20200603152657-dc2b0ca8b37e h1:oIpIX9VKxSCFrfjsKpluGbNPBGq9iNnT9crH781j9wY= -github.com/yuin/gopher-lua v0.0.0-20200603152657-dc2b0ca8b37e/go.mod h1:gqRgreBUhTSL0GeU64rtZ3Uq3wtjOa/TB2YfrtkCbVQ= github.com/yusufpapurcu/wmi v1.2.2 h1:KBNDSne4vP5mbSWnJbO+51IMOXJB67QiYCSBrubbPRg= github.com/yusufpapurcu/wmi v1.2.2/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0= github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43 h1:+lm10QQTNSBd8DVTNGHx7o/IKu9HYDvLMffDhbyLccI= @@ -1880,8 +1878,9 @@ golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.0.0-20220906165146-f3363e06e74c/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= -golang.org/x/net v0.5.0 h1:GyT4nK/YDHSqa1c4753ouYCDajOYKTja9Xb/OHtgvSw= golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws= +golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -1921,7 +1920,6 @@ golang.org/x/sys v0.0.0-20181026203630-95b1ffbd15a5/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20181122145206-62eef0e2fa9b/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190204203706-41f3e6584952/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -2038,8 +2036,9 @@ golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9sn golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.0.0-20220722155259-a9ba230a4035/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= -golang.org/x/term v0.4.0 h1:O7UWfv5+A2qiuulQk30kVinPoMtoIPeVaKLEgLpVkvg= golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ= +golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -2050,8 +2049,9 @@ golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.6.0 h1:3XmdazWV+ubf7QgHSTWeykHOci5oeekaGJBLkrkaw4k= golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= From 4cb3f85572263b1f744a86769b47e594f524d967 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Mon, 3 Apr 2023 15:30:56 +0800 Subject: [PATCH 34/80] fix: fixed #2262, update addon enable sub-cmd usage help (#2372) Co-authored-by: nashtsai --- docs/user_docs/cli/kbcli_addon_enable.md | 6 +++--- internal/cli/cmd/addon/addon.go | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/user_docs/cli/kbcli_addon_enable.md b/docs/user_docs/cli/kbcli_addon_enable.md index c9811c053..01559bc56 100644 --- a/docs/user_docs/cli/kbcli_addon_enable.md +++ b/docs/user_docs/cli/kbcli_addon_enable.md @@ -46,9 +46,9 @@ kbcli addon enable ADDON_NAME [flags] --set stringArray set values on the command line (can specify multiple or separate values with commas: key1=val1,key2=val2), it's only being processed if addon's type is helm. --show-managed-fields If true, keep the managedFields when printing objects in JSON or YAML format. --storage stringArray Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items)). - Additional notes: for type=Helm addon and if the value mapped directly to a StatefulSet's volume claim template - the helm upgrade action will failed, to resolved this you will need to disable and re-enable the addon, also noted - that storage size can only be expanded by PVC resizing. + Additional notes for Helm type Addon, that resizing storage will fail if modified value is a storage request size + that belongs to StatefulSet's volume claim template, to resolve 'Failed' Addon status possible action is disable and + re-enable the addon (More info on how-to resize a PVC: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources). --storage-class stringArray Sets addon storage class name (--storage-class [extraName:]) (can specify multiple if has extra items)) --template string Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]. diff --git a/internal/cli/cmd/addon/addon.go b/internal/cli/cmd/addon/addon.go index 9a72cd7bb..63285fa88 100644 --- a/internal/cli/cmd/addon/addon.go +++ b/internal/cli/cmd/addon/addon.go @@ -199,9 +199,9 @@ func newEnableCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra "Sets addon CPU resource values (--cpu [extraName:]/) (can specify multiple if has extra items))") cmd.Flags().StringArrayVar(&o.addonEnableFlags.StorageSets, "storage", []string{}, `Sets addon storage size (--storage [extraName:]) (can specify multiple if has extra items)). -Additional notes: for type=Helm addon and if the value mapped directly to a StatefulSet's volume claim template -the helm upgrade action will failed, to resolved this you will need to disable and re-enable the addon, also noted -that storage size can only be expanded by PVC resizing. +Additional notes for Helm type Addon, that resizing storage will fail if modified value is a storage request size +that belongs to StatefulSet's volume claim template, to resolve 'Failed' Addon status possible action is disable and +re-enable the addon (More info on how-to resize a PVC: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources). `) cmd.Flags().StringArrayVar(&o.addonEnableFlags.ReplicaCountSets, "replicas", []string{}, "Sets addon component replica count (--replicas [extraName:]) (can specify multiple if has extra items))") From 791d91bba0a46cd234a679d0e0aaaef3d32b9680 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Mon, 3 Apr 2023 21:51:20 +0800 Subject: [PATCH 35/80] chore: tidy up Addon API, with strict validation rules (#2402) --- apis/extensions/v1alpha1/addon_types.go | 45 ++++++++---- apis/extensions/v1alpha1/addon_types_test.go | 2 +- apis/extensions/v1alpha1/type.go | 11 --- .../extensions.kubeblocks.io_addons.yaml | 71 ++++++++++++------- config/samples/extensions_v1alpha1_addon.yaml | 8 +-- .../crds/extensions.kubeblocks.io_addons.yaml | 71 ++++++++++++------- .../helm/templates/addons/nyancat-addon.yaml | 1 - go.mod | 7 +- go.sum | 5 -- test/testdata/addon/addon.yaml | 2 +- 10 files changed, 133 insertions(+), 90 deletions(-) diff --git a/apis/extensions/v1alpha1/addon_types.go b/apis/extensions/v1alpha1/addon_types.go index e7b32d28e..151744a4b 100644 --- a/apis/extensions/v1alpha1/addon_types.go +++ b/apis/extensions/v1alpha1/addon_types.go @@ -30,12 +30,14 @@ import ( ) // AddonSpec defines the desired state of Addon +// +kubebuilder:validation:XValidation:rule="has(self.type) && self.type == 'Helm' ? has(self.helm) : !has(self.helm)",message="spec.helm is required when spec.type is Helm, and forbidden otherwise" type AddonSpec struct { // Addon description. // +optional Description string `json:"description,omitempty"` // Addon type, valid value is helm. + // +unionDiscriminator // +kubebuilder:validation:Required Type AddonType `json:"type"` @@ -85,7 +87,6 @@ type InstallableSpec struct { AutoInstall bool `json:"autoInstall"` } -// SelectorRequirement is the installation selector requirement. type SelectorRequirement struct { // The selector key, valid values are KubeVersion, KubeGitVersion. // "KubeVersion" the semver expression of Kubernetes versions, i.e., v1.24. @@ -97,10 +98,10 @@ type SelectorRequirement struct { // Valid operators are Contains, NotIn, DoesNotContain, MatchRegex, and DoesNoteMatchRegex. // // Possible enum values: - // `"Contains"` line contains string (symbol: "|="） - // `"DoesNotContain"` line does not contain string (symbol: "!=") - // `"MatchRegex"` line contains a match to the regular expression (symbol: "|~"） - // `"DoesNotMatchRegex"` line does not contain a match to the regular expression (symbol: "!~") + // `"Contains"` line contains string + // `"DoesNotContain"` line does not contain string + // `"MatchRegex"` line contains a match to the regular expression + // `"DoesNotMatchRegex"` line does not contain a match to the regular expression // +kubebuilder:validation:Required Operator LineSelectorOperator `json:"operator"` @@ -109,9 +110,7 @@ type SelectorRequirement struct { Values []string `json:"values,omitempty" protobuf:"bytes,3,rep,name=values"` } -// HelmTypeInstallSpec defines a Helm release installation spec. type HelmTypeInstallSpec struct { - // A Helm Chart location URL. // +kubebuilder:validation:Required ChartLocationURL string `json:"chartLocationURL"` @@ -145,7 +144,7 @@ type HelmInstallValues struct { // a JSON or YAML string content, use key name with ".json" or ".yaml" or ".yml" // extension name to specify content type. // +optional - SecretRefs []DataObjectKeySelector `json:"secretsRefs,omitempty"` + SecretRefs []DataObjectKeySelector `json:"secretRefs,omitempty"` // Helm install set values, can specify multiple or separate values with commas(key1=val1,key2=val2). // +optional @@ -176,7 +175,25 @@ type HelmValuesMappingExtraItem struct { Name string `json:"name"` } -type HelmValueMapType map[KeyHelmValueKey]string +type HelmValueMapType struct { + // replicaCount sets replicaCount value mapping key. + // +optional + ReplicaCount string `json:"replicaCount,omitempty"` + + // persistentVolumeEnabled persistent volume enabled mapping key. + // +optional + PVEnabled string `json:"persistentVolumeEnabled,omitempty"` + + // storageClass sets storageClass mapping key. + // +optional + StorageClass string `json:"storageClass,omitempty"` +} + +type HelmJSONValueMapType struct { + // tolerations sets toleration mapping key. + // +optional + Tolerations string `json:"tolerations,omitempty"` +} type HelmValuesMappingItem struct { // valueMap define the "key" mapping values, valid keys are replicaCount, @@ -191,7 +208,7 @@ type HelmValuesMappingItem struct { // Enum values explained: // `"tolerations"` sets toleration mapping key // +optional - HelmJSONMap HelmValueMapType `json:"jsonMap,omitempty"` + HelmJSONMap HelmJSONValueMapType `json:"jsonMap,omitempty"` // resources sets resources related mapping keys. // +optional @@ -470,14 +487,14 @@ func (r *HelmTypeInstallSpec) BuildMergedValues(installSpec *AddonInstallSpec) H installValues := r.InstallValues processor := func(installSpecItem AddonInstallSpecItem, valueMapping HelmValuesMappingItem) { if installSpecItem.Replicas != nil && *installSpecItem.Replicas >= 0 { - if v, ok := valueMapping.HelmValueMap[ReplicaCount]; ok { + if v := valueMapping.HelmValueMap.ReplicaCount; v != "" { installValues.SetValues = append(installValues.SetValues, fmt.Sprintf("%s=%v", v, *installSpecItem.Replicas)) } } if installSpecItem.StorageClass != "" { - if v, ok := valueMapping.HelmValueMap[StorageClass]; ok { + if v := valueMapping.HelmValueMap.StorageClass; v != "" { if installSpecItem.StorageClass == "-" { installValues.SetValues = append(installValues.SetValues, fmt.Sprintf("%s=null", v)) @@ -489,14 +506,14 @@ func (r *HelmTypeInstallSpec) BuildMergedValues(installSpec *AddonInstallSpec) H } if installSpecItem.PVEnabled != nil { - if v, ok := valueMapping.HelmValueMap[PVEnabled]; ok { + if v := valueMapping.HelmValueMap.PVEnabled; v != "" { installValues.SetValues = append(installValues.SetValues, fmt.Sprintf("%s=%v", v, *installSpecItem.PVEnabled)) } } if installSpecItem.Tolerations != "" { - if v, ok := valueMapping.HelmJSONMap[Tolerations]; ok { + if v := valueMapping.HelmJSONMap.Tolerations; v != "" { installValues.SetJSONValues = append(installValues.SetJSONValues, fmt.Sprintf("%s=%s", v, installSpecItem.Tolerations)) } diff --git a/apis/extensions/v1alpha1/addon_types_test.go b/apis/extensions/v1alpha1/addon_types_test.go index b76eb3fe6..c988610d4 100644 --- a/apis/extensions/v1alpha1/addon_types_test.go +++ b/apis/extensions/v1alpha1/addon_types_test.go @@ -235,7 +235,7 @@ func TestHelmInstallSpecBuildMergedValues(t *testing.T) { StorageClass: mappingName(name, sc), PVEnabled: mappingName(name, pvEnabled), }, - HelmJSONMap: HelmValueMapType{ + HelmJSONMap: HelmJSONValueMapType{ Tolerations: mappingName(name, tolerations), }, ResourcesMapping: &ResourceMappingItem{ diff --git a/apis/extensions/v1alpha1/type.go b/apis/extensions/v1alpha1/type.go index b632025f2..a52106c16 100644 --- a/apis/extensions/v1alpha1/type.go +++ b/apis/extensions/v1alpha1/type.go @@ -49,17 +49,6 @@ const ( AddonDisabling AddonPhase = "Disabling" ) -// KeyHelmValueKey defines "key" Helm value's key types. -// +enum -type KeyHelmValueKey string - -const ( - ReplicaCount KeyHelmValueKey = "replicaCount" - PVEnabled KeyHelmValueKey = "persistentVolumeEnabled" - StorageClass KeyHelmValueKey = "storageClass" - Tolerations KeyHelmValueKey = "tolerations" -) - // AddonSelectorKey are selector requirement key types. // +enum // +kubebuilder:validation:Enum={KubeGitVersion,KubeVersion} diff --git a/config/crd/bases/extensions.kubeblocks.io_addons.yaml b/config/crd/bases/extensions.kubeblocks.io_addons.yaml index 696a43ccf..a17aa6fcd 100644 --- a/config/crd/bases/extensions.kubeblocks.io_addons.yaml +++ b/config/crd/bases/extensions.kubeblocks.io_addons.yaml @@ -150,8 +150,6 @@ spec: multiple selectors are provided that all selectors must evaluate to true. items: - description: SelectorRequirement is the installation selector - requirement. properties: key: description: The selector key, valid values are KubeVersion, @@ -166,12 +164,10 @@ spec: description: "Represents a key's relationship to a set of values. Valid operators are Contains, NotIn, DoesNotContain, MatchRegex, and DoesNoteMatchRegex. \n Possible enum - values: `\"Contains\"` line contains string (symbol: - \"|=\"） `\"DoesNotContain\"` line does not contain string - (symbol: \"!=\") `\"MatchRegex\"` line contains a match - to the regular expression (symbol: \"|~\"） `\"DoesNotMatchRegex\"` - line does not contain a match to the regular expression - (symbol: \"!~\")" + values: `\"Contains\"` line contains string `\"DoesNotContain\"` + line does not contain string `\"MatchRegex\"` line contains + a match to the regular expression `\"DoesNotMatchRegex\"` + line does not contain a match to the regular expression" enum: - Contains - DoesNotContain @@ -235,7 +231,7 @@ spec: - name type: object type: array - secretsRefs: + secretRefs: description: Selects a key of a Secrets item list, the value of Secrets can be a JSON or YAML string content, use key name with ".json" or ".yaml" or ".yml" extension name to @@ -279,11 +275,14 @@ spec: items: properties: jsonMap: - additionalProperties: - type: string description: 'jsonMap define the "key" mapping values, valid keys are tolerations. Enum values explained: `"tolerations"` sets toleration mapping key' + properties: + tolerations: + description: tolerations sets toleration mapping + key. + type: string type: object name: description: Name of the item. @@ -320,14 +319,25 @@ spec: type: string type: object valueMap: - additionalProperties: - type: string description: 'valueMap define the "key" mapping values, valid keys are replicaCount, persistentVolumeEnabled, and storageClass. Enum values explained: `"replicaCount"` sets replicaCount value mapping key `"persistentVolumeEnabled"` sets persistent volume enabled mapping key `"storageClass"` sets storageClass mapping key' + properties: + persistentVolumeEnabled: + description: persistentVolumeEnabled persistent + volume enabled mapping key. + type: string + replicaCount: + description: replicaCount sets replicaCount value + mapping key. + type: string + storageClass: + description: storageClass sets storageClass mapping + key. + type: string type: object required: - name @@ -337,11 +347,13 @@ spec: - name x-kubernetes-list-type: map jsonMap: - additionalProperties: - type: string description: 'jsonMap define the "key" mapping values, valid keys are tolerations. Enum values explained: `"tolerations"` sets toleration mapping key' + properties: + tolerations: + description: tolerations sets toleration mapping key. + type: string type: object resources: description: resources sets resources related mapping keys. @@ -373,14 +385,24 @@ spec: type: string type: object valueMap: - additionalProperties: - type: string description: 'valueMap define the "key" mapping values, valid keys are replicaCount, persistentVolumeEnabled, and storageClass. Enum values explained: `"replicaCount"` sets replicaCount value mapping key `"persistentVolumeEnabled"` sets persistent volume enabled mapping key `"storageClass"` sets storageClass mapping key' + properties: + persistentVolumeEnabled: + description: persistentVolumeEnabled persistent volume + enabled mapping key. + type: string + replicaCount: + description: replicaCount sets replicaCount value mapping + key. + type: string + storageClass: + description: storageClass sets storageClass mapping key. + type: string type: object type: object required: @@ -499,8 +521,6 @@ spec: description: Addon installable selectors. If multiple selectors are provided that all selectors must evaluate to true. items: - description: SelectorRequirement is the installation selector - requirement. properties: key: description: The selector key, valid values are KubeVersion, @@ -515,11 +535,10 @@ spec: description: "Represents a key's relationship to a set of values. Valid operators are Contains, NotIn, DoesNotContain, MatchRegex, and DoesNoteMatchRegex. \n Possible enum values: - `\"Contains\"` line contains string (symbol: \"|=\"） `\"DoesNotContain\"` - line does not contain string (symbol: \"!=\") `\"MatchRegex\"` - line contains a match to the regular expression (symbol: - \"|~\"） `\"DoesNotMatchRegex\"` line does not contain - a match to the regular expression (symbol: \"!~\")" + `\"Contains\"` line contains string `\"DoesNotContain\"` + line does not contain string `\"MatchRegex\"` line contains + a match to the regular expression `\"DoesNotMatchRegex\"` + line does not contain a match to the regular expression" enum: - Contains - DoesNotContain @@ -549,6 +568,10 @@ spec: - defaultInstallValues - type type: object + x-kubernetes-validations: + - message: spec.helm is required when spec.type is Helm, and forbidden + otherwise + rule: 'has(self.type) && self.type == ''Helm'' ? has(self.helm) : !has(self.helm)' status: description: AddonStatus defines the observed state of Addon properties: diff --git a/config/samples/extensions_v1alpha1_addon.yaml b/config/samples/extensions_v1alpha1_addon.yaml index b77f95481..aae0e0f36 100644 --- a/config/samples/extensions_v1alpha1_addon.yaml +++ b/config/samples/extensions_v1alpha1_addon.yaml @@ -7,7 +7,7 @@ metadata: app.kubernetes.io/part-of: kubeblocks app.kubernetes.io/managed-by: kustomize app.kubernetes.io/created-by: kubeblocks - name: prometheus + name: prometheus-sample spec: # Addon spec. description. description: Prometheus is a monitoring system and time series database. @@ -37,7 +37,7 @@ spec: # namepsace: # key: # via YAML contents reside in secret.data. - secretRef: + secretRefs: # - name: # namepsace: # key: @@ -60,12 +60,12 @@ spec: # values mapping specific to resources related context, i.e., replicaCount, storage, CPU & Memory. valuesMapping: - valueMap: # map[KeyHelmValueKey]string + valueMap: replicaCount: server.replicaCount storageClass: server.persistentVolume.storageClass persistentVolumeEnabled: server.persistentVolume.enabled - jsonMap: # map[KeyHelmValueKey]string + jsonMap: tolerations: server.tolerations resources: diff --git a/deploy/helm/crds/extensions.kubeblocks.io_addons.yaml b/deploy/helm/crds/extensions.kubeblocks.io_addons.yaml index 696a43ccf..a17aa6fcd 100644 --- a/deploy/helm/crds/extensions.kubeblocks.io_addons.yaml +++ b/deploy/helm/crds/extensions.kubeblocks.io_addons.yaml @@ -150,8 +150,6 @@ spec: multiple selectors are provided that all selectors must evaluate to true. items: - description: SelectorRequirement is the installation selector - requirement. properties: key: description: The selector key, valid values are KubeVersion, @@ -166,12 +164,10 @@ spec: description: "Represents a key's relationship to a set of values. Valid operators are Contains, NotIn, DoesNotContain, MatchRegex, and DoesNoteMatchRegex. \n Possible enum - values: `\"Contains\"` line contains string (symbol: - \"|=\"） `\"DoesNotContain\"` line does not contain string - (symbol: \"!=\") `\"MatchRegex\"` line contains a match - to the regular expression (symbol: \"|~\"） `\"DoesNotMatchRegex\"` - line does not contain a match to the regular expression - (symbol: \"!~\")" + values: `\"Contains\"` line contains string `\"DoesNotContain\"` + line does not contain string `\"MatchRegex\"` line contains + a match to the regular expression `\"DoesNotMatchRegex\"` + line does not contain a match to the regular expression" enum: - Contains - DoesNotContain @@ -235,7 +231,7 @@ spec: - name type: object type: array - secretsRefs: + secretRefs: description: Selects a key of a Secrets item list, the value of Secrets can be a JSON or YAML string content, use key name with ".json" or ".yaml" or ".yml" extension name to @@ -279,11 +275,14 @@ spec: items: properties: jsonMap: - additionalProperties: - type: string description: 'jsonMap define the "key" mapping values, valid keys are tolerations. Enum values explained: `"tolerations"` sets toleration mapping key' + properties: + tolerations: + description: tolerations sets toleration mapping + key. + type: string type: object name: description: Name of the item. @@ -320,14 +319,25 @@ spec: type: string type: object valueMap: - additionalProperties: - type: string description: 'valueMap define the "key" mapping values, valid keys are replicaCount, persistentVolumeEnabled, and storageClass. Enum values explained: `"replicaCount"` sets replicaCount value mapping key `"persistentVolumeEnabled"` sets persistent volume enabled mapping key `"storageClass"` sets storageClass mapping key' + properties: + persistentVolumeEnabled: + description: persistentVolumeEnabled persistent + volume enabled mapping key. + type: string + replicaCount: + description: replicaCount sets replicaCount value + mapping key. + type: string + storageClass: + description: storageClass sets storageClass mapping + key. + type: string type: object required: - name @@ -337,11 +347,13 @@ spec: - name x-kubernetes-list-type: map jsonMap: - additionalProperties: - type: string description: 'jsonMap define the "key" mapping values, valid keys are tolerations. Enum values explained: `"tolerations"` sets toleration mapping key' + properties: + tolerations: + description: tolerations sets toleration mapping key. + type: string type: object resources: description: resources sets resources related mapping keys. @@ -373,14 +385,24 @@ spec: type: string type: object valueMap: - additionalProperties: - type: string description: 'valueMap define the "key" mapping values, valid keys are replicaCount, persistentVolumeEnabled, and storageClass. Enum values explained: `"replicaCount"` sets replicaCount value mapping key `"persistentVolumeEnabled"` sets persistent volume enabled mapping key `"storageClass"` sets storageClass mapping key' + properties: + persistentVolumeEnabled: + description: persistentVolumeEnabled persistent volume + enabled mapping key. + type: string + replicaCount: + description: replicaCount sets replicaCount value mapping + key. + type: string + storageClass: + description: storageClass sets storageClass mapping key. + type: string type: object type: object required: @@ -499,8 +521,6 @@ spec: description: Addon installable selectors. If multiple selectors are provided that all selectors must evaluate to true. items: - description: SelectorRequirement is the installation selector - requirement. properties: key: description: The selector key, valid values are KubeVersion, @@ -515,11 +535,10 @@ spec: description: "Represents a key's relationship to a set of values. Valid operators are Contains, NotIn, DoesNotContain, MatchRegex, and DoesNoteMatchRegex. \n Possible enum values: - `\"Contains\"` line contains string (symbol: \"|=\"） `\"DoesNotContain\"` - line does not contain string (symbol: \"!=\") `\"MatchRegex\"` - line contains a match to the regular expression (symbol: - \"|~\"） `\"DoesNotMatchRegex\"` line does not contain - a match to the regular expression (symbol: \"!~\")" + `\"Contains\"` line contains string `\"DoesNotContain\"` + line does not contain string `\"MatchRegex\"` line contains + a match to the regular expression `\"DoesNotMatchRegex\"` + line does not contain a match to the regular expression" enum: - Contains - DoesNotContain @@ -549,6 +568,10 @@ spec: - defaultInstallValues - type type: object + x-kubernetes-validations: + - message: spec.helm is required when spec.type is Helm, and forbidden + otherwise + rule: 'has(self.type) && self.type == ''Helm'' ? has(self.helm) : !has(self.helm)' status: description: AddonStatus defines the observed state of Addon properties: diff --git a/deploy/helm/templates/addons/nyancat-addon.yaml b/deploy/helm/templates/addons/nyancat-addon.yaml index 150b9dba0..4fde96f1e 100644 --- a/deploy/helm/templates/addons/nyancat-addon.yaml +++ b/deploy/helm/templates/addons/nyancat-addon.yaml @@ -20,7 +20,6 @@ spec: valuesMapping: valueMap: replicaCount: replicaCount - namespace: namespace jsonMap: tolerations: tolerations diff --git a/go.mod b/go.mod index 401480ae3..d66c5b558 100644 --- a/go.mod +++ b/go.mod @@ -10,13 +10,11 @@ require ( github.com/StudioSol/set v1.0.0 github.com/ahmetalpbalkan/go-cursor v0.0.0-20131010032410-8136607ea412 github.com/authzed/controller-idioms v0.7.0 - github.com/aws/aws-sdk-go v1.44.122 github.com/bhmj/jsonslice v1.1.2 github.com/briandowns/spinner v1.23.0 github.com/clbanning/mxj/v2 v2.5.7 github.com/containerd/stargz-snapshotter/estargz v0.13.0 github.com/containers/common v0.49.1 - github.com/coreos/go-iptables v0.5.0 github.com/dapr/components-contrib v1.9.6 github.com/dapr/dapr v1.9.5 github.com/dapr/go-sdk v1.7.0 @@ -64,7 +62,6 @@ require ( github.com/stretchr/testify v1.8.1 github.com/sykesm/zap-logfmt v0.0.4 github.com/valyala/fasthttp v1.41.0 - github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5 github.com/vmware-tanzu/velero v1.10.1 go.etcd.io/etcd/client/v3 v3.5.6 go.etcd.io/etcd/server/v3 v3.5.6 @@ -75,7 +72,6 @@ require ( golang.org/x/exp v0.0.0-20221028150844-83b7d23a625f golang.org/x/net v0.7.0 golang.org/x/sync v0.1.0 - golang.org/x/sys v0.5.0 google.golang.org/grpc v1.52.0 google.golang.org/protobuf v1.28.1 gopkg.in/inf.v0 v0.9.1 @@ -121,6 +117,7 @@ require ( github.com/andybalholm/brotli v1.0.4 // indirect github.com/antlr/antlr4/runtime/Go/antlr v1.4.10 // indirect github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect + github.com/aws/aws-sdk-go v1.44.122 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d // indirect github.com/bhmj/xpression v0.9.1 // indirect @@ -312,7 +309,6 @@ require ( github.com/ulikunitz/xz v0.5.11 // indirect github.com/valyala/bytebufferpool v1.0.0 // indirect github.com/vbatts/tar-split v0.11.2 // indirect - github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f // indirect github.com/xanzy/ssh-agent v0.3.3 // indirect github.com/xdg-go/pbkdf2 v1.0.0 // indirect github.com/xdg-go/scram v1.1.1 // indirect @@ -349,6 +345,7 @@ require ( go4.org/unsafe/assume-no-moving-gc v0.0.0-20220617031537-928513b29760 // indirect golang.org/x/mod v0.7.0 // indirect golang.org/x/oauth2 v0.4.0 // indirect + golang.org/x/sys v0.5.0 // indirect golang.org/x/term v0.5.0 // indirect golang.org/x/text v0.7.0 // indirect golang.org/x/time v0.3.0 // indirect diff --git a/go.sum b/go.sum index 4aa950629..1e33980cc 100644 --- a/go.sum +++ b/go.sum @@ -416,7 +416,6 @@ github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkE github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= github.com/coreos/go-iptables v0.4.5/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU= -github.com/coreos/go-iptables v0.5.0 h1:mw6SAibtHKZcNzAsOxjoHIG0gy5YFHhypWSSNc6EjbQ= github.com/coreos/go-iptables v0.5.0/go.mod h1:/mVI274lEDI2ns62jHCDnCyBF9Iwsmekav8Dbxlm1MU= github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc= github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= @@ -1580,13 +1579,9 @@ github.com/vbatts/tar-split v0.11.2/go.mod h1:vV3ZuO2yWSVsz+pfFzDG/upWH1JhjOiEaW github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk= github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYppBueQtXaqoE= github.com/vishvananda/netlink v1.1.1-0.20201029203352-d40f9887b852/go.mod h1:twkDnbuQxJYemMlGd4JFIcuhgX83tXhKS2B/PRMpOho= -github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5 h1:+UB2BJA852UkGH42H+Oee69djmxS3ANzl2b/JtT1YiA= -github.com/vishvananda/netlink v1.1.1-0.20210330154013-f5de75959ad5/go.mod h1:twkDnbuQxJYemMlGd4JFIcuhgX83tXhKS2B/PRMpOho= github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI= github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU= github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= -github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f h1:p4VB7kIXpOQvVn1ZaTIVp+3vuYAXFe3OJEvjbUYJLaA= -github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0= github.com/vmihailenco/msgpack v3.3.3+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk= github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4= github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI= diff --git a/test/testdata/addon/addon.yaml b/test/testdata/addon/addon.yaml index 553f966ea..29baf41e7 100644 --- a/test/testdata/addon/addon.yaml +++ b/test/testdata/addon/addon.yaml @@ -31,7 +31,7 @@ spec: # namespace: default # key: values-kubeblocks-override.yaml # via YAML contents reside in secret.data. - secretRef: + secretRefs: # - name: # namepsace: # key: From 54ec867ba105594f568bf50dcd751ea1c7092e1d Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Mon, 3 Apr 2023 22:19:55 +0800 Subject: [PATCH 36/80] feat: addon support for chatgpt-retrieval-plugin milvus qdrant weaviate (#2385) --- Makefile | 2 + .../addons/chatgpt-retrieval-plugin.yaml | 39 +++++++++++++++++++ .../helm/templates/addons/milvus-addon.yaml | 24 ++++++++++++ .../helm/templates/addons/qdrant-addon.yaml | 24 ++++++++++++ .../helm/templates/addons/weaviate-addon.yaml | 24 ++++++++++++ .../weaviate/templates/clusterdefinition.yaml | 2 - .../weaviate/templates/headlessService.yaml | 18 --------- 7 files changed, 113 insertions(+), 20 deletions(-) create mode 100644 deploy/helm/templates/addons/chatgpt-retrieval-plugin.yaml create mode 100644 deploy/helm/templates/addons/milvus-addon.yaml create mode 100644 deploy/helm/templates/addons/qdrant-addon.yaml create mode 100644 deploy/helm/templates/addons/weaviate-addon.yaml delete mode 100644 deploy/weaviate/templates/headlessService.yaml diff --git a/Makefile b/Makefile index 345cebf01..46ac19038 100644 --- a/Makefile +++ b/Makefile @@ -393,6 +393,8 @@ bump-chart-ver: \ bump-single-chart-ver.milvus \ bump-single-chart-ver.qdrant \ bump-single-chart-ver.qdrant-cluster \ + bump-single-chart-ver.weaviate \ + bump-single-chart-ver.weaviate-cluster \ bump-single-chart-ver.chatgpt-retrieval-plugin bump-chart-ver: ## Bump helm chart version. diff --git a/deploy/helm/templates/addons/chatgpt-retrieval-plugin.yaml b/deploy/helm/templates/addons/chatgpt-retrieval-plugin.yaml new file mode 100644 index 000000000..de7826305 --- /dev/null +++ b/deploy/helm/templates/addons/chatgpt-retrieval-plugin.yaml @@ -0,0 +1,39 @@ +apiVersion: extensions.kubeblocks.io/v1alpha1 +kind: Addon +metadata: + name: chatgpt-retrieval-plugin + labels: + {{- include "kubeblocks.labels" . | nindent 4 }} + "kubeblocks.io/provider": apecloud + {{- if .Values.keepAddons }} + annotations: + helm.sh/resource-policy: keep + {{- end }} +spec: + description: 'Deploys a ChatGPT Retrieval Plugin application in a cluster. + ChatGPT Retrieval Plugin is an application for personalizing your ChatGPT dialogue through your private data.' + type: Helm + + helm: + chartLocationURL: https://jihulab.com/api/v4/projects/85949/packages/helm/stable/charts/chatgpt-retrieval-plugin-{{ default .Chart.Version .Values.versionOverride }}.tgz + valuesMapping: + valueMap: + replicaCount: replicaCount + + jsonMap: + tolerations: tolerations + + resources: + cpu: + requests: resources.requests.cpu + limits: resources.limits.cpu + memory: + requests: resources.requests.memory + limits: resources.limits.memory + + defaultInstallValues: + - replicas: 1 + + installable: + autoInstall: false + diff --git a/deploy/helm/templates/addons/milvus-addon.yaml b/deploy/helm/templates/addons/milvus-addon.yaml new file mode 100644 index 000000000..40d9f022f --- /dev/null +++ b/deploy/helm/templates/addons/milvus-addon.yaml @@ -0,0 +1,24 @@ +apiVersion: extensions.kubeblocks.io/v1alpha1 +kind: Addon +metadata: + name: milvus + labels: + {{- include "kubeblocks.labels" . | nindent 4 }} + "kubeblocks.io/provider": community + {{- if .Values.keepAddons }} + annotations: + helm.sh/resource-policy: keep + {{- end }} +spec: + description: 'Milvus is an open source (Apache-2.0 licensed) vector database built to power embedding similarity search and AI applications.' + + type: Helm + + helm: + chartLocationURL: https://jihulab.com/api/v4/projects/85949/packages/helm/stable/charts/milvus-{{ default .Chart.Version .Values.versionOverride }}.tgz + + installable: + autoInstall: false + + defaultInstallValues: + - enabled: true diff --git a/deploy/helm/templates/addons/qdrant-addon.yaml b/deploy/helm/templates/addons/qdrant-addon.yaml new file mode 100644 index 000000000..9f3b8ab10 --- /dev/null +++ b/deploy/helm/templates/addons/qdrant-addon.yaml @@ -0,0 +1,24 @@ +apiVersion: extensions.kubeblocks.io/v1alpha1 +kind: Addon +metadata: + name: qdrant + labels: + {{- include "kubeblocks.labels" . | nindent 4 }} + "kubeblocks.io/provider": community + {{- if .Values.keepAddons }} + annotations: + helm.sh/resource-policy: keep + {{- end }} +spec: + description: 'Qdrant is an open source (Apache-2.0 licensed), vector similarity search engine and vector database.' + + type: Helm + + helm: + chartLocationURL: https://jihulab.com/api/v4/projects/85949/packages/helm/stable/charts/qdrant-{{ default .Chart.Version .Values.versionOverride }}.tgz + + installable: + autoInstall: false + + defaultInstallValues: + - enabled: true diff --git a/deploy/helm/templates/addons/weaviate-addon.yaml b/deploy/helm/templates/addons/weaviate-addon.yaml new file mode 100644 index 000000000..3d7891897 --- /dev/null +++ b/deploy/helm/templates/addons/weaviate-addon.yaml @@ -0,0 +1,24 @@ +apiVersion: extensions.kubeblocks.io/v1alpha1 +kind: Addon +metadata: + name: weaviate + labels: + {{- include "kubeblocks.labels" . | nindent 4 }} + "kubeblocks.io/provider": community + {{- if .Values.keepAddons }} + annotations: + helm.sh/resource-policy: keep + {{- end }} +spec: + description: 'Weaviate is an open-source (BSD-3.0 licensed) vector database. It allows you to store data objects and vector embeddings from your favorite ML-models, and scale seamlessly into billions of data objects.' + + type: Helm + + helm: + chartLocationURL: https://jihulab.com/api/v4/projects/85949/packages/helm/stable/charts/weaviate-{{ default .Chart.Version .Values.versionOverride }}.tgz + + installable: + autoInstall: false + + defaultInstallValues: + - enabled: true diff --git a/deploy/weaviate/templates/clusterdefinition.yaml b/deploy/weaviate/templates/clusterdefinition.yaml index e30b95f96..34abdb070 100644 --- a/deploy/weaviate/templates/clusterdefinition.yaml +++ b/deploy/weaviate/templates/clusterdefinition.yaml @@ -126,5 +126,3 @@ spec: value: "7000" - name: CLUSTER_DATA_BIND_PORT value: "7001" - - name: CLUSTER_JOIN - value: weaviate-headless.default.svc.cluster.local diff --git a/deploy/weaviate/templates/headlessService.yaml b/deploy/weaviate/templates/headlessService.yaml deleted file mode 100644 index 5c138d5d3..000000000 --- a/deploy/weaviate/templates/headlessService.yaml +++ /dev/null @@ -1,18 +0,0 @@ ---- -apiVersion: v1 -kind: Service -metadata: - name: weaviate-headless - labels: - app.kubernetes.io/name: weaviate - app.kubernetes.io/managed-by: helm -spec: - type: ClusterIP - clusterIP: None - selector: - app: weaviate - ports: - - protocol: TCP - port: 80 - targetPort: 7000 - publishNotReadyAddresses: true From fc35ad5d17acf886116931d528f7659cf1178029 Mon Sep 17 00:00:00 2001 From: Ziang Guo Date: Tue, 4 Apr 2023 09:54:50 +0800 Subject: [PATCH 37/80] fix: install aws-loadbalancer-controller failed (#2388) --- .../addons/aws-loadbalancer-controller-addon.yaml | 5 +---- deploy/helm/values.yaml | 14 +++++++++----- internal/cli/cmd/playground/init.go | 14 ++++++++++++-- internal/cli/cmd/playground/init_test.go | 2 +- 4 files changed, 23 insertions(+), 12 deletions(-) diff --git a/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml b/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml index 47fb4f6fe..90244fccc 100644 --- a/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml +++ b/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml @@ -22,10 +22,7 @@ spec: key: values-kubeblocks-override.yaml setValues: - - serviceAccount.create=true - - serviceAccount.name=kubeblocks-service-account-aws-load-balancer-controller - clusterName={{ index .Values "aws-loadbalancer-controller" "cluterName " }} - - image.repository={{ index .Values "aws-loadbalancer-controller" "image" "registry"}} valuesMapping: valueMap: @@ -46,7 +43,7 @@ spec: - replicas: 1 installable: - autoInstall: false + autoInstall: {{ index .Values "aws-loadbalancer-controller" "enabled" }} selectors: - key: KubeGitVersion operator: Contains diff --git a/deploy/helm/values.yaml b/deploy/helm/values.yaml index 954fce3e9..a129d1e92 100644 --- a/deploy/helm/values.yaml +++ b/deploy/helm/values.yaml @@ -1651,9 +1651,13 @@ csi-hostpath-driver: default: true aws-loadbalancer-controller: + enabled: false replicaCount: 1 - clusterName: "" - image: - # please specify image registry by region - # reference: https://docs.amazonaws.cn/eks/latest/userguide/add-ons-images.html - registry: 961992271922.dkr.ecr.cn-northwest-1.amazonaws.com.cn/amazon/aws-load-balancer-controller + tolerations: + - key: kb-controller + operator: Equal + value: "true" + effect: NoSchedule + serviceAccount: + create: true + name: kubeblocks-service-account-aws-load-balancer-controller diff --git a/internal/cli/cmd/playground/init.go b/internal/cli/cmd/playground/init.go index 919fca5f4..dd62f9e57 100644 --- a/internal/cli/cmd/playground/init.go +++ b/internal/cli/cmd/playground/init.go @@ -182,7 +182,7 @@ func (o *initOptions) installKBAndCluster(k8sClusterName string) error { o.helmCfg = helm.NewConfig("", configPath, "", klog.V(1).Enabled()) // Install KubeBlocks - if err = o.installKubeBlocks(); err != nil { + if err = o.installKubeBlocks(k8sClusterName); err != nil { return errors.Wrap(err, "failed to install KubeBlocks") } @@ -351,7 +351,7 @@ func (o *initOptions) setKubeConfig(provider cp.Interface) error { return nil } -func (o *initOptions) installKubeBlocks() error { +func (o *initOptions) installKubeBlocks(k8sClusterName string) error { f := util.NewFactory() client, err := f.KubernetesClientSet() if err != nil { @@ -381,6 +381,10 @@ func (o *initOptions) installKubeBlocks() error { "snapshot-controller.enabled=true", "csi-hostpath-driver.enabled=true", + // enable aws loadbalancer controller addon automatically on playground + "aws-loadbalancer-controller.enabled=true", + fmt.Sprintf("aws-loadbalancer-controller.clusterName=%s", k8sClusterName), + // disable the persistent volume of prometheus, if not, the prometheus // will dependent the hostpath csi driver ready to create persistent // volume, but the order of addon installation is not guaranteed that @@ -389,6 +393,12 @@ func (o *initOptions) installKubeBlocks() error { "prometheus.server.statefulSet.enabled=false", "prometheus.alertmanager.persistentVolume.enabled=false", "prometheus.alertmanager.statefulSet.enabled=false") + } else if o.cloudProvider == cp.AWS { + insOpts.ValueOpts.Values = append(insOpts.ValueOpts.Values, + // enable aws loadbalancer controller addon automatically on playground + "aws-loadbalancer-controller.enabled=true", + fmt.Sprintf("aws-loadbalancer-controller.clusterName=%s", k8sClusterName), + ) } return insOpts.Install() diff --git a/internal/cli/cmd/playground/init_test.go b/internal/cli/cmd/playground/init_test.go index 5ca974890..3bfd4031b 100644 --- a/internal/cli/cmd/playground/init_test.go +++ b/internal/cli/cmd/playground/init_test.go @@ -48,7 +48,7 @@ var _ = Describe("playground", func() { } Expect(o.validate()).Should(Succeed()) Expect(o.run()).Should(HaveOccurred()) - Expect(o.installKubeBlocks()).Should(HaveOccurred()) + Expect(o.installKubeBlocks("test")).Should(HaveOccurred()) Expect(o.createCluster()).Should(HaveOccurred()) }) From 4d5462923b5b5d99aca985a5c259a0059793c39f Mon Sep 17 00:00:00 2001 From: xingran Date: Tue, 4 Apr 2023 10:30:29 +0800 Subject: [PATCH 38/80] feat: redis support high-availablity with redis-sentinel (#2321) --- deploy/redis-cluster/Chart.yaml | 2 +- deploy/redis-cluster/templates/cluster.yaml | 25 ++++- deploy/redis-cluster/values.yaml | 2 + deploy/redis/Chart.yaml | 2 +- .../redis/config/redis7-config-constraint.cue | 4 + deploy/redis/config/redis7-config.tpl | 3 + .../redis/scripts/redis-sentinel-setup.sh.tpl | 35 ++++++ .../redis/scripts/redis-sentinel-start.sh.tpl | 19 ++++ deploy/redis/templates/clusterdefinition.yaml | 100 +++++++++++++++++- deploy/redis/templates/clusterversion.yaml | 10 ++ deploy/redis/templates/scripts.yaml | 33 ++++++ deploy/redis/values.yaml | 8 +- 12 files changed, 232 insertions(+), 11 deletions(-) create mode 100644 deploy/redis/scripts/redis-sentinel-setup.sh.tpl create mode 100644 deploy/redis/scripts/redis-sentinel-start.sh.tpl diff --git a/deploy/redis-cluster/Chart.yaml b/deploy/redis-cluster/Chart.yaml index ea3c6ed3f..55783b46c 100644 --- a/deploy/redis-cluster/Chart.yaml +++ b/deploy/redis-cluster/Chart.yaml @@ -6,7 +6,7 @@ type: application version: 0.5.0-alpha.3 -appVersion: "7.0.5" +appVersion: "7.0.6" home: https://redis.io/ icon: https://bitnami.com/assets/stacks/redis/img/redis-stack-220x234.png diff --git a/deploy/redis-cluster/templates/cluster.yaml b/deploy/redis-cluster/templates/cluster.yaml index cf1f6551d..d24ee37d9 100644 --- a/deploy/redis-cluster/templates/cluster.yaml +++ b/deploy/redis-cluster/templates/cluster.yaml @@ -15,7 +15,7 @@ spec: tolerations: {{ . | toYaml | nindent 4 }} {{- end }} componentSpecs: - - name: redis-repl # user-defined + - name: redis # user-defined componentDefRef: redis # ref clusterDefinition componentDefs.name monitor: {{ .Values.monitor.enabled | default false }} enabledLogs: {{ .Values.enabledLogs | toJson | indent 4 }} @@ -42,4 +42,25 @@ spec: requests: storage: {{ .Values.persistence.data.size }} {{- end }} - + - name: redis-sentinel # user-defined + componentDefRef: redis-sentinel # ref clusterDefinition componentDefs.name + replicas: {{ .Values.sentinelReplicaCount | default 3 }} + {{- with .Values.resources }} + resources: + limits: + cpu: {{ .limits.cpu | quote }} + memory: {{ .limits.memory | quote }} + requests: + cpu: {{ .requests.cpu | quote }} + memory: {{ .requests.memory | quote }} + {{- end }} + {{- if .Values.persistence.enabled }} + volumeClaimTemplates: + - name: data # ref clusterdefinition components.containers.volumeMounts.name + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: {{ .Values.persistence.data.size }} + {{- end }} diff --git a/deploy/redis-cluster/values.yaml b/deploy/redis-cluster/values.yaml index 08cbbc236..6cf767984 100644 --- a/deploy/redis-cluster/values.yaml +++ b/deploy/redis-cluster/values.yaml @@ -4,6 +4,8 @@ replicaCount: 2 +sentinelReplicaCount: 3 + terminationPolicy: Delete clusterVersionOverride: "" diff --git a/deploy/redis/Chart.yaml b/deploy/redis/Chart.yaml index 07629aa7c..5950941a2 100644 --- a/deploy/redis/Chart.yaml +++ b/deploy/redis/Chart.yaml @@ -6,7 +6,7 @@ type: application version: 0.5.0-alpha.3 -appVersion: "7.0.5" +appVersion: "7.0.6" home: https://redis.io/ icon: https://bitnami.com/assets/stacks/redis/img/redis-stack-220x234.png diff --git a/deploy/redis/config/redis7-config-constraint.cue b/deploy/redis/config/redis7-config-constraint.cue index 155d3ceba..46974f0f1 100644 --- a/deploy/redis/config/redis7-config-constraint.cue +++ b/deploy/redis/config/redis7-config-constraint.cue @@ -142,5 +142,9 @@ "zset-max-listpack-value": int | *64 + "protected-mode"?: string & "yes" | "no" + + "enable-debug-command"?: string & "yes" | "no" | "local" + ... } diff --git a/deploy/redis/config/redis7-config.tpl b/deploy/redis/config/redis7-config.tpl index d6e351f56..4f86d27fa 100644 --- a/deploy/redis/config/redis7-config.tpl +++ b/deploy/redis/config/redis7-config.tpl @@ -68,3 +68,6 @@ dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes jemalloc-bg-thread yes +enable-debug-command yes +protected-mode no + diff --git a/deploy/redis/scripts/redis-sentinel-setup.sh.tpl b/deploy/redis/scripts/redis-sentinel-setup.sh.tpl new file mode 100644 index 000000000..af020399f --- /dev/null +++ b/deploy/redis/scripts/redis-sentinel-setup.sh.tpl @@ -0,0 +1,35 @@ +#!/bin/sh +set -ex +{{- $clusterName := $.cluster.metadata.name }} +{{- $namespace := $.cluster.metadata.namespace }} +{{- /* find redis-sentinel component */}} +{{- $sentinel_component := fromJson "{}" }} +{{- $redis_component := fromJson "{}" }} +{{- $primary_index := 0 }} +{{- $primary_pod := "" }} +{{- range $i, $e := $.cluster.spec.componentSpecs }} + {{- if eq $e.componentDefRef "redis-sentinel" }} + {{- $sentinel_component = $e }} + {{- else if eq $e.componentDefRef "redis" }} + {{- $redis_component = $e }} + {{- if ne ($e.primaryIndex | int) 0 }} + {{- $primary_index = ($e.primaryIndex | int) }} + {{- end }} + {{- end }} +{{- end }} +{{- /* build primary pod message, because currently does not support cross-component acquisition of environment variables, the service of the redis master node is assembled here through specific rules */}} +{{- $primary_pod = printf "%s-%s-0.%s-%s-headless.%s.svc" $clusterName $redis_component.name $clusterName $redis_component.name $namespace }} +{{- if ne $primary_index 0 }} + {{- $primary_pod = printf "%s-%s-%d-0.%s-%s-headless.%s.svc" $clusterName $redis_component.name $primary_index $clusterName $redis_component.name $namespace }} +{{- end }} +{{- $sentinel_monitor := printf "%s-%s %s" $clusterName $sentinel_component.name $primary_pod }} +cat>/etc/sentinel/redis-sentinel.conf<> /etc/sentinel/redis-sentinel.conf +exec redis-server /etc/sentinel/redis-sentinel.conf --sentinel +echo "Start sentinel succeeded!" \ No newline at end of file diff --git a/deploy/redis/templates/clusterdefinition.yaml b/deploy/redis/templates/clusterdefinition.yaml index c47e4bc20..5bd4f80d1 100644 --- a/deploy/redis/templates/clusterdefinition.yaml +++ b/deploy/redis/templates/clusterdefinition.yaml @@ -56,8 +56,14 @@ spec: - redis-cli -h $(KB_SWITCH_ROLE_ENDPOINT) -p 6379 $(KB_SWITCH_DEMOTE_STATEMENT) service: ports: - - protocol: TCP + - name: redis + protocol: TCP port: 6379 + targetPort: redis + - name: metrics + targetPort: metrics + port: 9121 + nodePort: null configSpecs: - name: redis-replication-config templateRef: redis7-config-template @@ -86,7 +92,6 @@ spec: podSpec: containers: - name: redis - image: redis:7.0.5 ports: - name: redis containerPort: 6379 @@ -97,7 +102,9 @@ spec: mountPath: /etc/conf - name: scripts mountPath: /scripts - args: [ "/etc/conf/redis.conf" ] + - name: redis-conf + mountPath: /etc/redis + command: ["/scripts/redis-start.sh"] lifecycle: postStart: exec: @@ -116,11 +123,11 @@ spec: livenessProbe: httpGet: path: / - port: 9121 + port: metrics readinessProbe: httpGet: path: / - port: 9121 + port: metrics systemAccounts: # Seems redis-cli has its own mechanism to parse input tokens and there is no elegent way # to pass $(KB_ACCOUNT_STATEMENT) to redis-cli without causing parsing error. @@ -168,3 +175,86 @@ spec: scope: AnyPods statements: creation: ACL SETUSER $(USERNAME) ON >$(PASSWD) +psync +replconf +ping + - name: redis-sentinel + workloadType: Stateful + characterType: redis + service: + ports: + - name: redis-sentinel + protocol: TCP + targetPort: redis-sentinel + port: 26379 + configSpecs: + - name: redis-replication-config + templateRef: redis7-config-template + constraintRef: redis7-config-constraints + namespace: {{ .Release.Namespace }} + volumeName: redis-config + scriptSpecs: + - name: redis-scripts + templateRef: redis-scripts + namespace: {{ .Release.Namespace }} + volumeName: scripts + defaultMode: 493 + volumeTypes: + - name: data + type: data + podSpec: + initContainers: + - name: init-redis-sentinel + imagePullPolicy: IfNotPresent + volumeMounts: + - name: data + mountPath: /data + - name: redis-config + mountPath: /etc/conf + - name: sentinel-conf + mountPath: /etc/sentinel + - name: scripts + mountPath: /scripts + command: [ "/scripts/redis-sentinel-setup.sh" ] + containers: + - name: redis-sentinel + imagePullPolicy: IfNotPresent + ports: + - containerPort: 26379 + name: redis-sentinel + protocol: TCP + volumeMounts: + - name: data + mountPath: /data + - name: redis-config + mountPath: /etc/conf + - name: sentinel-conf + mountPath: /etc/sentinel + - name: scripts + mountPath: /scripts + command: + - /bin/bash + args: + - -c + - | + set -ex + /scripts/redis-sentinel-start.sh + livenessProbe: + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 5 + successThreshold: 1 + failureThreshold: 5 + exec: + command: + - sh + - -c + - /scripts/redis-sentinel-ping.sh 5 + readinessProbe: + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 1 + successThreshold: 1 + failureThreshold: 5 + exec: + command: + - sh + - -c + - /scripts/redis-sentinel-ping.sh 1 \ No newline at end of file diff --git a/deploy/redis/templates/clusterversion.yaml b/deploy/redis/templates/clusterversion.yaml index 4ab03e2dc..4ee5aba1d 100644 --- a/deploy/redis/templates/clusterversion.yaml +++ b/deploy/redis/templates/clusterversion.yaml @@ -13,3 +13,13 @@ spec: - name: redis image: {{ .Values.image.repository }}:{{ .Values.image.tag }} imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} + - componentDefRef: redis-sentinel + versionsContext: + initContainers: + - name: init-redis-sentinel + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} + imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} + containers: + - name: redis-sentinel + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} + imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} diff --git a/deploy/redis/templates/scripts.yaml b/deploy/redis/templates/scripts.yaml index d682d863e..3b3e8c794 100644 --- a/deploy/redis/templates/scripts.yaml +++ b/deploy/redis/templates/scripts.yaml @@ -14,3 +14,36 @@ data: until redis-cli -h $KB_PRIMARY_POD_NAME -p 6379 ping; do sleep 1; done redis-cli -h 127.0.0.1 -p 6379 replicaof $KB_PRIMARY_POD_NAME 6379 || exit 1 fi + redis-start.sh: | + #!/bin/sh + set -ex + echo "include /etc/conf/redis.conf" >> /etc/redis/redis.conf + echo "replica-announce-ip $KB_POD_FQDN" >> /etc/redis/redis.conf + exec redis-server /etc/redis/redis.conf \ + --loadmodule /opt/redis-stack/lib/redisearch.so ${REDISEARCH_ARGS} \ + --loadmodule /opt/redis-stack/lib/redisgraph.so ${REDISGRAPH_ARGS} \ + --loadmodule /opt/redis-stack/lib/redistimeseries.so ${REDISTIMESERIES_ARGS} \ + --loadmodule /opt/redis-stack/lib/rejson.so ${REDISJSON_ARGS} \ + --loadmodule /opt/redis-stack/lib/redisbloom.so ${REDISBLOOM_ARGS} + redis-sentinel-setup.sh: |- + {{- .Files.Get "scripts/redis-sentinel-setup.sh.tpl" | nindent 4 }} + redis-sentinel-start.sh: |- + {{- .Files.Get "scripts/redis-sentinel-start.sh.tpl" | nindent 4 }} + redis-sentinel-ping.sh: |- + #!/bin/sh + set -ex + response=$( + timeout -s 3 $1 \ + redis-cli \ + -h localhost \ + -p 26379 \ + ping + ) + if [ "$?" -eq "124" ]; then + echo "Timed out" + exit 1 + fi + if [ "$response" != "PONG" ]; then + echo "$response" + exit 1 + fi \ No newline at end of file diff --git a/deploy/redis/values.yaml b/deploy/redis/values.yaml index e41951632..2d5f589a1 100644 --- a/deploy/redis/values.yaml +++ b/deploy/redis/values.yaml @@ -3,10 +3,14 @@ # Declare variables to be passed into your templates. image: - repository: redis + # refer: https://redis.io/docs/stack/ + # Redis Stack Server, which combines open source Redis with RediSearch, RedisJSON, RedisGraph, RedisTimeSeries, and RedisBloom, + # is dual-licensed under the Redis Source Available License (RSALv2), as described below, and the Server Side Public License (SSPL) + # For information about licensing per version, see https://redis.io/docs/stack/license/ + repository: redis/redis-stack-server pullPolicy: IfNotPresent # Overrides the image tag whose default is the chart appVersion. - tag: 7.0.5 + tag: 7.0.6-RC8 imagePullSecrets: [] nameOverride: "" From 49159c3cd7062bc0a9669d40ca784d9e55bf0338 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Tue, 4 Apr 2023 10:38:01 +0800 Subject: [PATCH 39/80] chore: update clusters.kubeblocks.io.spec.componentsSpec.volumeClaimTemplates.spec type, removed use of corev1.PersistentVolumeClaimSpec struct, new struct with required properties only (#2059) --- apis/apps/v1alpha1/cluster_types.go | 115 ++++++++----- apis/apps/v1alpha1/cluster_types_test.go | 12 +- apis/apps/v1alpha1/cluster_webhook.go | 5 +- .../bases/apps.kubeblocks.io_clusters.yaml | 151 ------------------ controllers/apps/cluster_controller_test.go | 43 +++-- controllers/apps/cluster_status_utils.go | 2 +- controllers/apps/components/component.go | 2 +- .../apps/components/component_status_test.go | 32 ++-- .../components/consensusset/consensus_set.go | 2 +- .../components/consensusset/consensus_test.go | 4 +- .../replicationset/replication_set_test.go | 4 +- .../replicationset/replication_set_utils.go | 2 +- .../apps/components/stateful/stateful_test.go | 2 +- .../components/stateless/stateless_test.go | 2 +- .../apps/components/util/component_utils.go | 2 +- .../components/util/component_utils_test.go | 2 +- .../reconfigurerequest_controller.go | 2 +- controllers/apps/operations/ops_util.go | 2 +- .../apps/operations/volume_expansion.go | 7 +- .../apps/opsrequest_controller_test.go | 8 +- controllers/apps/systemaccount_util.go | 2 +- .../crds/apps.kubeblocks.io_clusters.yaml | 151 ------------------ deploy/kafka-cluster/values.yaml | 31 +++- deploy/kafka/templates/clusterdefinition.yaml | 18 +-- deploy/kafka/templates/clusterversion.yaml | 2 +- deploy/kafka/templates/configconstraint.yaml | 2 +- deploy/kafka/templates/configmap.yaml | 4 +- deploy/kafka/templates/scripts.yaml | 2 +- deploy/kafka/templates/storageclass.yaml | 130 +++++++++++++++ deploy/kafka/values.yaml | 31 +++- githooks/scripts/run-lint.sh | 2 +- internal/cli/cmd/accounts/util.go | 2 +- internal/cli/cmd/cluster/create.go | 2 +- internal/cli/testing/fake.go | 4 +- internal/controller/builder/builder_test.go | 4 +- .../controller/component/component_test.go | 4 +- internal/controller/component/type.go | 48 +++--- .../lifecycle/transformer_cluster.go | 2 +- .../lifecycle/transformer_fill_class.go | 2 +- .../lifecycle/validator_enable_logs.go | 2 +- internal/controller/plan/prepare_test.go | 20 +-- .../apps/cluster_consensus_test_util.go | 4 +- internal/testutil/apps/cluster_factory.go | 2 +- internal/testutil/apps/cluster_util.go | 14 ++ internal/testutil/apps/native_object_util.go | 2 - test/integration/backup_mysql_test.go | 4 +- test/integration/controller_suite_test.go | 2 +- test/integration/mysql_ha_test.go | 10 +- test/integration/mysql_scale_test.go | 13 +- test/integration/redis_hscale_test.go | 4 +- 50 files changed, 406 insertions(+), 515 deletions(-) create mode 100644 deploy/kafka/templates/storageclass.yaml diff --git a/apis/apps/v1alpha1/cluster_types.go b/apis/apps/v1alpha1/cluster_types.go index 49aae1b3b..01f19cf7f 100644 --- a/apis/apps/v1alpha1/cluster_types.go +++ b/apis/apps/v1alpha1/cluster_types.go @@ -279,7 +279,56 @@ type ClusterComponentVolumeClaimTemplate struct { // spec defines the desired characteristics of a volume requested by a pod author. // +kubebuilder:pruning:PreserveUnknownFields // +optional - Spec *corev1.PersistentVolumeClaimSpec `json:"spec,omitempty"` + Spec PersistentVolumeClaimSpec `json:"spec,omitempty"` +} + +func (r *ClusterComponentVolumeClaimTemplate) toVolumeClaimTemplate() corev1.PersistentVolumeClaimTemplate { + t := corev1.PersistentVolumeClaimTemplate{} + t.ObjectMeta.Name = r.Name + t.Spec = r.Spec.ToV1PersistentVolumeClaimSpec() + return t +} + +type PersistentVolumeClaimSpec struct { + // accessModes contains the desired access modes the volume should have. + // More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1 + // +optional + AccessModes []corev1.PersistentVolumeAccessMode `json:"accessModes,omitempty" protobuf:"bytes,1,rep,name=accessModes,casttype=PersistentVolumeAccessMode"` + // resources represents the minimum resources the volume should have. + // If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements + // that are lower than previous value but must still be higher than capacity recorded in the + // status field of the claim. + // More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources + // +optional + Resources corev1.ResourceRequirements `json:"resources,omitempty" protobuf:"bytes,2,opt,name=resources"` + // storageClassName is the name of the StorageClass required by the claim. + // More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 + // +optional + StorageClassName *string `json:"storageClassName,omitempty" protobuf:"bytes,5,opt,name=storageClassName"` + // TODO: + // // preferStorageClassNames added support specifying storageclasses.storage.k8s.io names, in order + // // to adapt multi-cloud deployment, where storageclasses are all distinctly different among clouds. + // // +listType=set + // // +optional + // PreferSCNames []string `json:"preferStorageClassNames,omitempty"` +} + +// ToV1PersistentVolumeClaimSpec converts to corev1.PersistentVolumeClaimSpec. +func (r PersistentVolumeClaimSpec) ToV1PersistentVolumeClaimSpec() corev1.PersistentVolumeClaimSpec { + return corev1.PersistentVolumeClaimSpec{ + AccessModes: r.AccessModes, + Resources: r.Resources, + StorageClassName: r.StorageClassName, + } +} + +// GetStorageClassName return PersistentVolumeClaimSpec.StorageClassName if value is assigned, otherwise +// return preferSC argument. +func (r PersistentVolumeClaimSpec) GetStorageClassName(preferSC string) *string { + if r.StorageClassName != nil && *r.StorageClassName != "" { + return r.StorageClassName + } + return &preferSC } type Affinity struct { @@ -409,10 +458,30 @@ func init() { SchemeBuilder.Register(&Cluster{}, &ClusterList{}) } +// GetComponentByName gets component by name. +func (r ClusterSpec) GetComponentByName(componentName string) *ClusterComponentSpec { + for _, v := range r.ComponentSpecs { + if v.Name == componentName { + return &v + } + } + return nil +} + +// GetComponentDefRefName gets the name of referenced component definition. +func (r ClusterSpec) GetComponentDefRefName(componentName string) string { + for _, component := range r.ComponentSpecs { + if componentName == component.Name { + return component.ComponentDefRef + } + } + return "" +} + // ValidateEnabledLogs validates enabledLogs config in cluster.yaml, and returns metav1.Condition when detect invalid values. -func (r *Cluster) ValidateEnabledLogs(cd *ClusterDefinition) error { +func (r ClusterSpec) ValidateEnabledLogs(cd *ClusterDefinition) error { message := make([]string, 0) - for _, comp := range r.Spec.ComponentSpecs { + for _, comp := range r.ComponentSpecs { invalidLogNames := cd.ValidateEnabledLogConfigs(comp.ComponentDefRef, comp.EnabledLogs) if len(invalidLogNames) == 0 { continue @@ -426,9 +495,9 @@ func (r *Cluster) ValidateEnabledLogs(cd *ClusterDefinition) error { } // GetDefNameMappingComponents returns ComponentDefRef name mapping ClusterComponentSpec. -func (r *Cluster) GetDefNameMappingComponents() map[string][]ClusterComponentSpec { +func (r ClusterSpec) GetDefNameMappingComponents() map[string][]ClusterComponentSpec { m := map[string][]ClusterComponentSpec{} - for _, c := range r.Spec.ComponentSpecs { + for _, c := range r.ComponentSpecs { v := m[c.ComponentDefRef] v = append(v, c) m[c.ComponentDefRef] = v @@ -483,31 +552,8 @@ func (m ComponentMessageMap) SetObjectMessage(objectKind, objectName, message st m[messageKey] = message } -// GetComponentByName gets component by name. -func (r *Cluster) GetComponentByName(componentName string) *ClusterComponentSpec { - for _, v := range r.Spec.ComponentSpecs { - if v.Name == componentName { - return &v - } - } - return nil -} - -// GetComponentDefRefName gets the name of referenced component definition. -func (r *Cluster) GetComponentDefRefName(componentName string) string { - for _, component := range r.Spec.ComponentSpecs { - if componentName == component.Name { - return component.ComponentDefRef - } - } - return "" -} - // SetComponentStatus does safe operation on ClusterStatus.Components map object update. func (r *ClusterStatus) SetComponentStatus(name string, status ClusterComponentStatus) { - if r == nil { - return - } r.checkedInitComponentsMap() r.Components[name] = status } @@ -524,8 +570,8 @@ func (r *ClusterComponentSpec) ToVolumeClaimTemplates() []corev1.PersistentVolum return nil } var ts []corev1.PersistentVolumeClaimTemplate - for _, template := range r.VolumeClaimTemplates { - ts = append(ts, toVolumeClaimTemplate(template)) + for _, t := range r.VolumeClaimTemplates { + ts = append(ts, t.toVolumeClaimTemplate()) } return ts } @@ -574,12 +620,3 @@ func GetComponentTerminalPhases() []ClusterComponentPhase { AbnormalClusterCompPhase, } } - -func toVolumeClaimTemplate(template ClusterComponentVolumeClaimTemplate) corev1.PersistentVolumeClaimTemplate { - t := corev1.PersistentVolumeClaimTemplate{} - t.ObjectMeta.Name = template.Name - if template.Spec != nil { - t.Spec = *template.Spec - } - return t -} diff --git a/apis/apps/v1alpha1/cluster_types_test.go b/apis/apps/v1alpha1/cluster_types_test.go index 9aa0d7ca6..6245259f6 100644 --- a/apis/apps/v1alpha1/cluster_types_test.go +++ b/apis/apps/v1alpha1/cluster_types_test.go @@ -59,12 +59,12 @@ spec: _ = yaml.Unmarshal([]byte(clusterByte), cluster) _ = yaml.Unmarshal([]byte(clusterDefByte), clusterDef) // normal case - if err := cluster.ValidateEnabledLogs(clusterDef); err != nil { + if err := cluster.Spec.ValidateEnabledLogs(clusterDef); err != nil { t.Error("Expected empty conditionList") } // corner case cluster.Spec.ComponentSpecs[0].EnabledLogs = []string{"error-test", "slow"} - if err := cluster.ValidateEnabledLogs(clusterDef); err == nil { + if err := cluster.Spec.ValidateEnabledLogs(clusterDef); err == nil { t.Error("Expected one element conditionList") } } @@ -130,20 +130,20 @@ func TestGetComponentOrName(t *testing.T) { }, }, } - compDefName := cluster.GetComponentDefRefName(componentName) + compDefName := cluster.Spec.GetComponentDefRefName(componentName) if compDefName != componentDefName { t.Errorf(`function GetComponentDefRefName should return %s`, componentDefName) } - component := cluster.GetComponentByName(componentName) + component := cluster.Spec.GetComponentByName(componentName) if component == nil { t.Errorf("function GetComponentByName should not return nil") } componentName = "mysql1" - compDefName = cluster.GetComponentDefRefName(componentName) + compDefName = cluster.Spec.GetComponentDefRefName(componentName) if compDefName != "" { t.Errorf(`function GetComponentDefRefName should return ""`) } - component = cluster.GetComponentByName(componentName) + component = cluster.Spec.GetComponentByName(componentName) if component != nil { t.Error("function GetComponentByName should return nil") } diff --git a/apis/apps/v1alpha1/cluster_webhook.go b/apis/apps/v1alpha1/cluster_webhook.go index 1630dd415..d7ac7cf69 100644 --- a/apis/apps/v1alpha1/cluster_webhook.go +++ b/apis/apps/v1alpha1/cluster_webhook.go @@ -41,7 +41,7 @@ func (r *Cluster) SetupWebhookWithManager(mgr ctrl.Manager) error { Complete() } -//+kubebuilder:webhook:path=/mutate-apps-kubeblocks-io-v1alpha1-cluster,mutating=true,failurePolicy=fail,sideEffects=None,groups=apps.kubeblocks.io,resources=clusters,verbs=create;update,versions=v1alpha1,name=mcluster.kb.io,admissionReviewVersions=v1 +// +kubebuilder:webhook:path=/mutate-apps-kubeblocks-io-v1alpha1-cluster,mutating=true,failurePolicy=fail,sideEffects=None,groups=apps.kubeblocks.io,resources=clusters,verbs=create;update,versions=v1alpha1,name=mcluster.kb.io,admissionReviewVersions=v1 var _ webhook.Defaulter = &Cluster{} @@ -151,9 +151,6 @@ func getLastComponentByName(lastCluster *Cluster, componentName string) *Cluster // setVolumeClaimStorageSizeZero set the volumeClaimTemplates storage size to zero. then we can diff last/current volumeClaimTemplates. func setVolumeClaimStorageSizeZero(volumeClaimTemplates []ClusterComponentVolumeClaimTemplate) { for i := range volumeClaimTemplates { - if volumeClaimTemplates[i].Spec == nil { - continue - } volumeClaimTemplates[i].Spec.Resources = corev1.ResourceRequirements{} } } diff --git a/config/crd/bases/apps.kubeblocks.io_clusters.yaml b/config/crd/bases/apps.kubeblocks.io_clusters.yaml index a6b1eafec..68e463d91 100644 --- a/config/crd/bases/apps.kubeblocks.io_clusters.yaml +++ b/config/crd/bases/apps.kubeblocks.io_clusters.yaml @@ -402,100 +402,6 @@ spec: items: type: string type: array - dataSource: - description: 'dataSource field can be used to specify - either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) - * An existing PVC (PersistentVolumeClaim) If the - provisioner or an external controller can support - the specified data source, it will create a new - volume based on the contents of the specified data - source. When the AnyVolumeDataSource feature gate - is enabled, dataSource contents will be copied to - dataSourceRef, and dataSourceRef contents will be - copied to dataSource when dataSourceRef.namespace - is not specified. If the namespace is specified, - then dataSourceRef will not be copied to dataSource.' - properties: - apiGroup: - description: APIGroup is the group for the resource - being referenced. If APIGroup is not specified, - the specified Kind must be in the core API group. - For any other third-party types, APIGroup is - required. - type: string - kind: - description: Kind is the type of resource being - referenced - type: string - name: - description: Name is the name of resource being - referenced - type: string - required: - - kind - - name - type: object - dataSourceRef: - description: 'dataSourceRef specifies the object from - which to populate the volume with data, if a non-empty - volume is desired. This may be any object from a - non-empty API group (non core object) or a PersistentVolumeClaim - object. When this field is specified, volume binding - will only succeed if the type of the specified object - matches some installed volume populator or dynamic - provisioner. This field will replace the functionality - of the dataSource field and as such if both fields - are non-empty, they must have the same value. For - backwards compatibility, when namespace isn''t specified - in dataSourceRef, both fields (dataSource and dataSourceRef) - will be set to the same value automatically if one - of them is empty and the other is non-empty. When - namespace is specified in dataSourceRef, dataSource - isn''t set to the same value and must be empty. - There are three important differences between dataSource - and dataSourceRef: * While dataSource only allows - two specific types of objects, dataSourceRef allows - any non-core object, as well as PersistentVolumeClaim - objects. * While dataSource ignores disallowed values - (dropping them), dataSourceRef preserves all values, - and generates an error if a disallowed value is - specified. * While dataSource only allows local - objects, dataSourceRef allows objects in any namespaces. - (Beta) Using this field requires the AnyVolumeDataSource - feature gate to be enabled. (Alpha) Using the namespace - field of dataSourceRef requires the CrossNamespaceVolumeDataSource - feature gate to be enabled.' - properties: - apiGroup: - description: APIGroup is the group for the resource - being referenced. If APIGroup is not specified, - the specified Kind must be in the core API group. - For any other third-party types, APIGroup is - required. - type: string - kind: - description: Kind is the type of resource being - referenced - type: string - name: - description: Name is the name of resource being - referenced - type: string - namespace: - description: Namespace is the namespace of resource - being referenced Note that when a namespace - is specified, a gateway.networking.k8s.io/ReferenceGrant - object is required in the referent namespace - to allow that namespace's owner to accept the - reference. See the ReferenceGrant documentation - for details. (Alpha) This field requires the - CrossNamespaceVolumeDataSource feature gate - to be enabled. - type: string - required: - - kind - - name - type: object resources: description: 'resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure @@ -552,67 +458,10 @@ spec: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object - selector: - description: selector is a label query over volumes - to consider for binding. - properties: - matchExpressions: - description: matchExpressions is a list of label - selector requirements. The requirements are - ANDed. - items: - description: A label selector requirement is - a selector that contains values, a key, and - an operator that relates the key and values. - properties: - key: - description: key is the label key that the - selector applies to. - type: string - operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and DoesNotExist. - type: string - values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If - the operator is Exists or DoesNotExist, - the values array must be empty. This array - is replaced during a strategic merge patch. - items: - type: string - type: array - required: - - key - - operator - type: object - type: array - matchLabels: - additionalProperties: - type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". - The requirements are ANDed. - type: object - type: object storageClassName: description: 'storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' type: string - volumeMode: - description: volumeMode defines what type of volume - is required by the claim. Value of Filesystem is - implied when not included in claim spec. - type: string - volumeName: - description: volumeName is the binding reference to - the PersistentVolume backing this claim. - type: string type: object x-kubernetes-preserve-unknown-fields: true required: diff --git a/controllers/apps/cluster_controller_test.go b/controllers/apps/cluster_controller_test.go index 4efad311d..7a00b4c6c 100644 --- a/controllers/apps/cluster_controller_test.go +++ b/controllers/apps/cluster_controller_test.go @@ -646,11 +646,11 @@ var _ = Describe("Cluster Controller", func() { updatedReplicas := int32(3) By("Creating a single component cluster with VolumeClaimTemplate") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). SetReplicas(initialReplicas). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -669,14 +669,14 @@ var _ = Describe("Cluster Controller", func() { secondMysqlCompName := mysqlCompName + "1" By("Creating a multi components cluster with VolumeClaimTemplate") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). SetReplicas(initialReplicas). AddComponent(secondMysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). SetReplicas(initialReplicas). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -704,14 +704,14 @@ var _ = Describe("Cluster Controller", func() { Expect(testCtx.CreateObj(testCtx.Ctx, storageClass)).Should(Succeed()) By("Creating a cluster with VolumeClaimTemplate") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") pvcSpec.StorageClassName = &storageClass.Name By("Create cluster and waiting for the cluster initialized") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). SetReplicas(replicas). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -733,7 +733,7 @@ var _ = Describe("Cluster Controller", func() { Labels: map[string]string{ constant.AppInstanceLabelKey: clusterKey.Name, }}, - Spec: pvcSpec, + Spec: pvcSpec.ToV1PersistentVolumeClaimSpec(), } Expect(testCtx.CreateObj(testCtx.Ctx, pvc)).Should(Succeed()) pvc.Status.Phase = corev1.ClaimBound // only bound pvc allows resize @@ -959,14 +959,7 @@ var _ = Describe("Cluster Controller", func() { const replicas = 3 By("Mock a cluster obj") - pvcSpec := &corev1.PersistentVolumeClaimSpec{ - AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, - Resources: corev1.ResourceRequirements{ - Requests: corev1.ResourceList{ - corev1.ResourceStorage: resource.MustParse("1Gi"), - }, - }, - } + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). @@ -1106,11 +1099,11 @@ var _ = Describe("Cluster Controller", func() { updatedReplicas := int32(3) By("Creating a cluster with VolumeClaimTemplate") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). SetReplicas(initialReplicas). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -1438,13 +1431,13 @@ var _ = Describe("Cluster Controller", func() { // with cluster.phase.phase=creating It("Should success with primary sts and secondary sts", func() { By("Mock a cluster obj with replication componentDefRef.") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(testapps.DefaultRedisCompName, testapps.DefaultRedisCompType). SetPrimaryIndex(testapps.DefaultReplicationPrimaryIndex). SetReplicas(testapps.DefaultReplicationReplicas). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -1486,9 +1479,9 @@ var _ = Describe("Cluster Controller", func() { It("Should successfully doing volume expansion", func() { storageClassName := "test-storage" - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") pvcSpec.StorageClassName = &storageClassName - updatedPVCSpec := testapps.NewPVC("2Gi") + updatedPVCSpec := testapps.NewPVCSpec("2Gi") updatedPVCSpec.StorageClassName = &storageClassName By("Mock a cluster obj with replication componentDefRef.") @@ -1497,7 +1490,7 @@ var _ = Describe("Cluster Controller", func() { AddComponent(testapps.DefaultRedisCompName, testapps.DefaultRedisCompType). SetPrimaryIndex(testapps.DefaultReplicationPrimaryIndex). SetReplicas(testapps.DefaultReplicationReplicas). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -1573,8 +1566,8 @@ var _ = Describe("Cluster Controller", func() { By("Updating PVC volume size") patch := client.MergeFrom(clusterObj.DeepCopy()) - componentSpec := clusterObj.GetComponentByName(testapps.DefaultRedisCompName) - componentSpec.VolumeClaimTemplates[0].Spec = &updatedPVCSpec + componentSpec := clusterObj.Spec.GetComponentByName(testapps.DefaultRedisCompName) + componentSpec.VolumeClaimTemplates[0].Spec = updatedPVCSpec Expect(testCtx.Cli.Patch(ctx, clusterObj, patch)).Should(Succeed()) By("Waiting cluster update reconcile succeed") diff --git a/controllers/apps/cluster_status_utils.go b/controllers/apps/cluster_status_utils.go index 3b11d1268..7d16d3a3f 100644 --- a/controllers/apps/cluster_status_utils.go +++ b/controllers/apps/cluster_status_utils.go @@ -240,7 +240,7 @@ func handleClusterStatusByEvent(ctx context.Context, cli client.Client, recorder // get the component phase by component name and sync to Cluster.status.components patch := client.MergeFrom(cluster.DeepCopy()) componentMap, clusterAvailabilityEffectMap, componentDef := getComponentRelatedInfo(cluster, clusterDef, componentName) - clusterComponent := cluster.GetComponentByName(componentName) + clusterComponent := cluster.Spec.GetComponentByName(componentName) if clusterComponent == nil { return nil } diff --git a/controllers/apps/components/component.go b/controllers/apps/components/component.go index b25ba3ee7..213c9e67a 100644 --- a/controllers/apps/components/component.go +++ b/controllers/apps/components/component.go @@ -138,7 +138,7 @@ func workloadCompClusterReconcile( // create a component object componentName := operand.GetLabels()[constant.KBAppComponentLabelKey] - componentSpec := cluster.GetComponentByName(componentName) + componentSpec := cluster.Spec.GetComponentByName(componentName) if componentSpec == nil { return intctrlutil.Reconciled() } diff --git a/controllers/apps/components/component_status_test.go b/controllers/apps/components/component_status_test.go index e98d9a779..38bfa79ef 100644 --- a/controllers/apps/components/component_status_test.go +++ b/controllers/apps/components/component_status_test.go @@ -94,14 +94,14 @@ var _ = Describe("ComponentStatusSynchronizer", func() { GetObject() component, err = NewComponentByType(testCtx.Cli, cluster, - cluster.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) + cluster.Spec.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) Expect(err).Should(Succeed()) Expect(component).ShouldNot(BeNil()) }) It("should not change component if no deployment or pod exists", func() { synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -142,7 +142,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Expect(mockContainerError(pod)).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -167,7 +167,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { })).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -207,14 +207,14 @@ var _ = Describe("ComponentStatusSynchronizer", func() { GetObject() component, err = NewComponentByType(testCtx.Cli, cluster, - cluster.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) + cluster.Spec.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) Expect(err).Should(Succeed()) Expect(component).ShouldNot(BeNil()) }) It("should not change component if no statefulset or pod exists", func() { synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -266,7 +266,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Expect(mockContainerError(pods[1])).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -293,7 +293,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { })).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -333,14 +333,14 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Create(&testCtx).GetObject() component, err = NewComponentByType(testCtx.Cli, cluster, - cluster.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) + cluster.Spec.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) Expect(err).Should(Succeed()) Expect(component).ShouldNot(BeNil()) }) It("should not change component if no statefulset or pod exists", func() { synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -390,7 +390,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Expect(mockContainerError(pods[0])).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -419,7 +419,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Expect(setPodRole(pods[2], "follower")).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -459,14 +459,14 @@ var _ = Describe("ComponentStatusSynchronizer", func() { GetObject() component, err = NewComponentByType(testCtx.Cli, cluster, - cluster.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) + cluster.Spec.GetComponentByName(compName), *clusterDef.GetComponentDefByName(compName)) Expect(err).Should(Succeed()) Expect(component).ShouldNot(BeNil()) }) It("should not change component if no deployment or pod exists", func() { synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -519,7 +519,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { Expect(mockContainerError(pods[0])).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) @@ -544,7 +544,7 @@ var _ = Describe("ComponentStatusSynchronizer", func() { })).Should(Succeed()) synchronizer, err := newClusterStatusSynchronizer(testCtx.Ctx, testCtx.Cli, cluster, - cluster.GetComponentByName(compName), component) + cluster.Spec.GetComponentByName(compName), component) Expect(err).Should(Succeed()) Expect(synchronizer).ShouldNot(BeNil()) diff --git a/controllers/apps/components/consensusset/consensus_set.go b/controllers/apps/components/consensusset/consensus_set.go index 707f47da0..eb1022a82 100644 --- a/controllers/apps/components/consensusset/consensus_set.go +++ b/controllers/apps/components/consensusset/consensus_set.go @@ -185,7 +185,7 @@ func (r *ConsensusSet) HandleUpdate(ctx context.Context, obj client.Object) erro stsObj := util.ConvertToStatefulSet(obj) // get compDefName from stsObj.name - compDefName := r.Cluster.GetComponentDefRefName(stsObj.Labels[constant.KBAppComponentLabelKey]) + compDefName := r.Cluster.Spec.GetComponentDefRefName(stsObj.Labels[constant.KBAppComponentLabelKey]) // get component from ClusterDefinition by compDefName component, err := util.GetComponentDefByCluster(ctx, r.Cli, *r.Cluster, compDefName) diff --git a/controllers/apps/components/consensusset/consensus_test.go b/controllers/apps/components/consensusset/consensus_test.go index 097648859..59e7afcee 100644 --- a/controllers/apps/components/consensusset/consensus_test.go +++ b/controllers/apps/components/consensusset/consensus_test.go @@ -99,9 +99,9 @@ var _ = Describe("Consensus Component", func() { sts := testapps.MockConsensusComponentStatefulSet(testCtx, clusterName, consensusCompName) componentName := consensusCompName - compDefName := cluster.GetComponentDefRefName(componentName) + compDefName := cluster.Spec.GetComponentDefRefName(componentName) componentDef := clusterDef.GetComponentDefByName(compDefName) - component := cluster.GetComponentByName(componentName) + component := cluster.Spec.GetComponentByName(componentName) By("test pods are not ready") consensusComponent, err := NewConsensusSet(k8sClient, cluster, component, *componentDef) diff --git a/controllers/apps/components/replicationset/replication_set_test.go b/controllers/apps/components/replicationset/replication_set_test.go index a2a27113f..a758cfd16 100644 --- a/controllers/apps/components/replicationset/replication_set_test.go +++ b/controllers/apps/components/replicationset/replication_set_test.go @@ -136,9 +136,9 @@ var _ = Describe("Replication Component", func() { Expect(sts.Spec.VolumeClaimTemplates).Should(BeEmpty()) } - compDefName := clusterObj.GetComponentDefRefName(testapps.DefaultRedisCompName) + compDefName := clusterObj.Spec.GetComponentDefRefName(testapps.DefaultRedisCompName) componentDef := clusterDefObj.GetComponentDefByName(compDefName) - component := clusterObj.GetComponentByName(testapps.DefaultRedisCompName) + component := clusterObj.Spec.GetComponentByName(testapps.DefaultRedisCompName) replicationComponent, err := NewReplicationSet(k8sClient, clusterObj, component, *componentDef) Expect(err).Should(Succeed()) var podList []*corev1.Pod diff --git a/controllers/apps/components/replicationset/replication_set_utils.go b/controllers/apps/components/replicationset/replication_set_utils.go index 1b58b4d87..6a61d09a6 100644 --- a/controllers/apps/components/replicationset/replication_set_utils.go +++ b/controllers/apps/components/replicationset/replication_set_utils.go @@ -384,7 +384,7 @@ func filterReplicationWorkload(ctx context.Context, if compSpecName == "" { return nil, fmt.Errorf("cluster's compSpecName is nil, pls check") } - compDefName := cluster.GetComponentDefRefName(compSpecName) + compDefName := cluster.Spec.GetComponentDefRefName(compSpecName) compDef, err := util.GetComponentDefByCluster(ctx, cli, *cluster, compDefName) if err != nil { return compDef, err diff --git a/controllers/apps/components/stateful/stateful_test.go b/controllers/apps/components/stateful/stateful_test.go index caf867c2e..5abd6114a 100644 --- a/controllers/apps/components/stateful/stateful_test.go +++ b/controllers/apps/components/stateful/stateful_test.go @@ -84,7 +84,7 @@ var _ = Describe("Stateful Component", func() { By("test pods number of sts is 0") sts := &stsList.Items[0] - clusterComponent := cluster.GetComponentByName(statefulCompName) + clusterComponent := cluster.Spec.GetComponentByName(statefulCompName) componentDef := clusterDef.GetComponentDefByName(clusterComponent.ComponentDefRef) stateful, err := NewStateful(k8sClient, cluster, clusterComponent, *componentDef) Expect(err).Should(Succeed()) diff --git a/controllers/apps/components/stateless/stateless_test.go b/controllers/apps/components/stateless/stateless_test.go index 3f0eea621..b7b5c2428 100644 --- a/controllers/apps/components/stateless/stateless_test.go +++ b/controllers/apps/components/stateless/stateless_test.go @@ -76,7 +76,7 @@ var _ = Describe("Stateful Component", func() { cluster := testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDefName, clusterVersionName). AddComponent(statelessCompName, statelessCompDefRef).SetReplicas(2).Create(&testCtx).GetObject() deploy := testapps.MockStatelessComponentDeploy(testCtx, clusterName, statelessCompName) - clusterComponent := cluster.GetComponentByName(statelessCompName) + clusterComponent := cluster.Spec.GetComponentByName(statelessCompName) componentDef := clusterDef.GetComponentDefByName(clusterComponent.ComponentDefRef) statelessComponent, err := NewStateless(k8sClient, cluster, clusterComponent, *componentDef) Expect(err).Should(Succeed()) diff --git a/controllers/apps/components/util/component_utils.go b/controllers/apps/components/util/component_utils.go index 952e376fb..237d490ca 100644 --- a/controllers/apps/components/util/component_utils.go +++ b/controllers/apps/components/util/component_utils.go @@ -272,7 +272,7 @@ func GetComponentInfoByPod(ctx context.Context, if !ok { return "", nil, errors.New("pod component name label is nil") } - compDefName := cluster.GetComponentDefRefName(componentName) + compDefName := cluster.Spec.GetComponentDefRefName(componentName) componentDef, err = GetComponentDefByCluster(ctx, cli, cluster, compDefName) if err != nil { return componentName, componentDef, err diff --git a/controllers/apps/components/util/component_utils_test.go b/controllers/apps/components/util/component_utils_test.go index 75139da23..3b49bd6d7 100644 --- a/controllers/apps/components/util/component_utils_test.go +++ b/controllers/apps/components/util/component_utils_test.go @@ -293,7 +293,7 @@ var _ = Describe("Consensus Component", func() { Expect(err).ShouldNot(Succeed()) By("test GetComponentPhaseWhenPodsNotReady function") - consensusComp := cluster.GetComponentByName(consensusCompName) + consensusComp := cluster.Spec.GetComponentByName(consensusCompName) checkExistFailedPodOfLatestRevision := func(pod *corev1.Pod, workload metav1.Object) bool { sts := workload.(*appsv1.StatefulSet) return !intctrlutil.PodIsReady(pod) && intctrlutil.PodIsControlledByLatestRevision(pod, sts) diff --git a/controllers/apps/configuration/reconfigurerequest_controller.go b/controllers/apps/configuration/reconfigurerequest_controller.go index 833a3cfe8..8ac52d49f 100644 --- a/controllers/apps/configuration/reconfigurerequest_controller.go +++ b/controllers/apps/configuration/reconfigurerequest_controller.go @@ -177,7 +177,7 @@ func (r *ReconfigureRequestReconciler) sync(reqCtx intctrlutil.RequestCtx, confi } // Find ClusterComponentSpec from cluster cr - clusterComponent := cluster.GetComponentByName(componentName) + clusterComponent := cluster.Spec.GetComponentByName(componentName) // Assumption: It is required that the cluster must have a component. if clusterComponent == nil { reqCtx.Log.Info("not found component.") diff --git a/controllers/apps/operations/ops_util.go b/controllers/apps/operations/ops_util.go index a5879b1e0..0d7f8343d 100644 --- a/controllers/apps/operations/ops_util.go +++ b/controllers/apps/operations/ops_util.go @@ -93,7 +93,7 @@ func ReconcileActionWithComponentOps(reqCtx intctrlutil.RequestCtx, if compStatus.Phase != v.Phase { compStatus.Phase = v.Phase } - clusterComponent := opsRes.Cluster.GetComponentByName(k) + clusterComponent := opsRes.Cluster.Spec.GetComponentByName(k) expectCount, completedCount, err := handleStatusProgress(reqCtx, cli, opsRes, progressResource{ opsMessageKey: opsMessageKey, clusterComponent: clusterComponent, diff --git a/controllers/apps/operations/volume_expansion.go b/controllers/apps/operations/volume_expansion.go index 42ca55517..79bd1a00d 100644 --- a/controllers/apps/operations/volume_expansion.go +++ b/controllers/apps/operations/volume_expansion.go @@ -72,19 +72,16 @@ func (ve volumeExpansionOpsHandler) Action(reqCtx intctrlutil.RequestCtx, cli cl if volumeExpansionOps, ok = volumeExpansionMap[component.Name]; !ok { continue } + compSpec := &opsRes.Cluster.Spec.ComponentSpecs[index] for _, v := range volumeExpansionOps.VolumeClaimTemplates { for i, vct := range component.VolumeClaimTemplates { if vct.Name != v.Name { continue } - if vct.Spec == nil { - continue - } - opsRes.Cluster.Spec.ComponentSpecs[index].VolumeClaimTemplates[i]. + compSpec.VolumeClaimTemplates[i]. Spec.Resources.Requests[corev1.ResourceStorage] = v.Storage } } - } return cli.Update(reqCtx.Ctx, opsRes.Cluster) } diff --git a/controllers/apps/opsrequest_controller_test.go b/controllers/apps/opsrequest_controller_test.go index e4751d322..af7999a45 100644 --- a/controllers/apps/opsrequest_controller_test.go +++ b/controllers/apps/opsrequest_controller_test.go @@ -279,12 +279,12 @@ var _ = Describe("OpsRequest Controller", func() { clusterDef.Spec.ComponentDefs[0].HorizontalScalePolicy = &appsv1alpha1.HorizontalScalePolicy{Type: appsv1alpha1.HScaleDataClonePolicyFromSnapshot} })()).ShouldNot(HaveOccurred()) - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). SetReplicas(replicas). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) @@ -379,12 +379,12 @@ var _ = Describe("OpsRequest Controller", func() { Create(&testCtx).GetObject() By("Creating a cluster with replication workloadType.") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") pvcSpec.StorageClassName = &storageClassName clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(testapps.DefaultRedisCompName, testapps.DefaultRedisCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec).SetPrimaryIndex(0). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec).SetPrimaryIndex(0). SetReplicas(testapps.DefaultReplicationReplicas). Create(&testCtx).GetObject() // mock sts ready and create pod diff --git a/controllers/apps/systemaccount_util.go b/controllers/apps/systemaccount_util.go index 9a6910b99..0f6d5c8fc 100644 --- a/controllers/apps/systemaccount_util.go +++ b/controllers/apps/systemaccount_util.go @@ -372,7 +372,7 @@ func calibrateJobMetaAndSpec(job *batchv1.Job, cluster *appsv1alpha1.Cluster, co // add toleration tolerations := cluster.Spec.Tolerations - clusterComp := cluster.GetComponentByName(compKey.componentName) + clusterComp := cluster.Spec.GetComponentByName(compKey.componentName) if clusterComp != nil { if len(clusterComp.Tolerations) != 0 { tolerations = clusterComp.Tolerations diff --git a/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml b/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml index a6b1eafec..68e463d91 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_clusters.yaml @@ -402,100 +402,6 @@ spec: items: type: string type: array - dataSource: - description: 'dataSource field can be used to specify - either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) - * An existing PVC (PersistentVolumeClaim) If the - provisioner or an external controller can support - the specified data source, it will create a new - volume based on the contents of the specified data - source. When the AnyVolumeDataSource feature gate - is enabled, dataSource contents will be copied to - dataSourceRef, and dataSourceRef contents will be - copied to dataSource when dataSourceRef.namespace - is not specified. If the namespace is specified, - then dataSourceRef will not be copied to dataSource.' - properties: - apiGroup: - description: APIGroup is the group for the resource - being referenced. If APIGroup is not specified, - the specified Kind must be in the core API group. - For any other third-party types, APIGroup is - required. - type: string - kind: - description: Kind is the type of resource being - referenced - type: string - name: - description: Name is the name of resource being - referenced - type: string - required: - - kind - - name - type: object - dataSourceRef: - description: 'dataSourceRef specifies the object from - which to populate the volume with data, if a non-empty - volume is desired. This may be any object from a - non-empty API group (non core object) or a PersistentVolumeClaim - object. When this field is specified, volume binding - will only succeed if the type of the specified object - matches some installed volume populator or dynamic - provisioner. This field will replace the functionality - of the dataSource field and as such if both fields - are non-empty, they must have the same value. For - backwards compatibility, when namespace isn''t specified - in dataSourceRef, both fields (dataSource and dataSourceRef) - will be set to the same value automatically if one - of them is empty and the other is non-empty. When - namespace is specified in dataSourceRef, dataSource - isn''t set to the same value and must be empty. - There are three important differences between dataSource - and dataSourceRef: * While dataSource only allows - two specific types of objects, dataSourceRef allows - any non-core object, as well as PersistentVolumeClaim - objects. * While dataSource ignores disallowed values - (dropping them), dataSourceRef preserves all values, - and generates an error if a disallowed value is - specified. * While dataSource only allows local - objects, dataSourceRef allows objects in any namespaces. - (Beta) Using this field requires the AnyVolumeDataSource - feature gate to be enabled. (Alpha) Using the namespace - field of dataSourceRef requires the CrossNamespaceVolumeDataSource - feature gate to be enabled.' - properties: - apiGroup: - description: APIGroup is the group for the resource - being referenced. If APIGroup is not specified, - the specified Kind must be in the core API group. - For any other third-party types, APIGroup is - required. - type: string - kind: - description: Kind is the type of resource being - referenced - type: string - name: - description: Name is the name of resource being - referenced - type: string - namespace: - description: Namespace is the namespace of resource - being referenced Note that when a namespace - is specified, a gateway.networking.k8s.io/ReferenceGrant - object is required in the referent namespace - to allow that namespace's owner to accept the - reference. See the ReferenceGrant documentation - for details. (Alpha) This field requires the - CrossNamespaceVolumeDataSource feature gate - to be enabled. - type: string - required: - - kind - - name - type: object resources: description: 'resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure @@ -552,67 +458,10 @@ spec: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' type: object type: object - selector: - description: selector is a label query over volumes - to consider for binding. - properties: - matchExpressions: - description: matchExpressions is a list of label - selector requirements. The requirements are - ANDed. - items: - description: A label selector requirement is - a selector that contains values, a key, and - an operator that relates the key and values. - properties: - key: - description: key is the label key that the - selector applies to. - type: string - operator: - description: operator represents a key's - relationship to a set of values. Valid - operators are In, NotIn, Exists and DoesNotExist. - type: string - values: - description: values is an array of string - values. If the operator is In or NotIn, - the values array must be non-empty. If - the operator is Exists or DoesNotExist, - the values array must be empty. This array - is replaced during a strategic merge patch. - items: - type: string - type: array - required: - - key - - operator - type: object - type: array - matchLabels: - additionalProperties: - type: string - description: matchLabels is a map of {key,value} - pairs. A single {key,value} in the matchLabels - map is equivalent to an element of matchExpressions, - whose key field is "key", the operator is "In", - and the values array contains only "value". - The requirements are ANDed. - type: object - type: object storageClassName: description: 'storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' type: string - volumeMode: - description: volumeMode defines what type of volume - is required by the claim. Value of Filesystem is - implied when not included in claim spec. - type: string - volumeName: - description: volumeName is the binding reference to - the PersistentVolume backing this claim. - type: string type: object x-kubernetes-preserve-unknown-fields: true required: diff --git a/deploy/kafka-cluster/values.yaml b/deploy/kafka-cluster/values.yaml index 058e024e1..17c1cc836 100644 --- a/deploy/kafka-cluster/values.yaml +++ b/deploy/kafka-cluster/values.yaml @@ -1,4 +1,3 @@ - ## @param terminationPolicy define Cluster termination policy. One of DoNotTerminate, Halt, Delete, WipeOut. ## terminationPolicy: Halt @@ -23,6 +22,15 @@ persistence: ## `metadata` volume settings ## metadata: + ## @param persistence.metadata.preferStorageClassNames + preferStorageClassNames: + - kafka-meta-eks + - kafka-meta-aks + - kafka-meta-gke + - kafka-meta-ack + - kafka-meta-tke + + ## @param persistence.data.storageClassName Storage class of backing PVC ## @param persistence.data.storageClassName Storage class of backing PVC ## If defined, storageClassName: ## If set to "-", storageClassName: "", which disables dynamic provisioning @@ -37,6 +45,14 @@ persistence: ## `data` volume settings ## data: + ## @param persistence.data.preferStorageClassNames + preferStorageClassNames: + - kafka-data-eks + - kafka-data-aks + - kafka-data-gke + - kafka-data-ack + - kafka-data-tke + ## @param persistence.data.storageClassName Storage class of backing PVC ## If defined, storageClassName: ## If set to "-", storageClassName: "", which disables dynamic provisioning @@ -44,7 +60,8 @@ persistence: ## set, choosing the default provisioner. (gp2 on AWS, standard on ## GKE, AWS & OpenStack) ## - storageClassName: + storageClassName: + ## @param persistence.data.size Size of data volume ## size: 10Gi @@ -54,6 +71,16 @@ persistence: ## @param persistence.log.enabled Enable persistence using Persistent Volume Claims ## enabled: false + + ## @param persistence.log.preferStorageClassNames + preferStorageClassNames: + - kafka-eks-standard + - kafka-aks-standard + - kafka-gke-standard + - kafka-ack-standard + - kafka-tke-standard + + ## @param persistence.data.storageClassName Storage class of backing PVC ## @param persistence.log.storageClassName Storage class of backing PVC ## If defined, storageClassName: ## If set to "-", storageClassName: "", which disables dynamic provisioning diff --git a/deploy/kafka/templates/clusterdefinition.yaml b/deploy/kafka/templates/clusterdefinition.yaml index 87abce565..674e059dc 100644 --- a/deploy/kafka/templates/clusterdefinition.yaml +++ b/deploy/kafka/templates/clusterdefinition.yaml @@ -1,7 +1,7 @@ apiVersion: apps.kubeblocks.io/v1alpha1 kind: ClusterDefinition metadata: - name: kafka + name: {{ include "kafka.name" . }} labels: {{- include "kafka.labels" . | nindent 4 }} {{- if .Values.commonLabels }} @@ -13,7 +13,7 @@ metadata: spec: connectionCredential: superusers: "User:admin" - endpoint: "$(SVC_FQDN):$(SVC_PORT_kafka-ctrl)" + endpoint: "$(SVC_FQDN):$(SVC_PORT_kafka-ctrlr)" kraftClusterID: "$(UUID_STR_B64)" sslCertPassword: "$(RANDOM_PASSWD)" @@ -37,30 +37,30 @@ spec: scrapePort: 5556 configSpecs: - name: kafka-configuration-tpl - constraintRef: kafka-cc + constraintRef: {{ include "kafka.name" . }}-cc templateRef: kafka-configuration-tpl volumeName: kafka-config namespace: {{ .Release.Namespace }} - name: kafka-jmx-configuration-tpl - templateRef: kafka-jmx-configuration-tpl + templateRef: {{ include "kafka.name" . }}-jmx-configuration-tpl volumeName: jmx-config namespace: {{ .Release.Namespace }} scriptSpecs: - name: kafka-scripts-tpl - templateRef: kafka-scripts-tpl + templateRef: {{ include "kafka.name" . }}-scripts-tpl volumeName: scripts namespace: {{ .Release.Namespace }} defaultMode: 0755 service: ports: - - name: kafka-client - targetPort: kafka-client - port: 9092 - nodePort: null - name: kafka-ctrlr targetPort: kafka-ctrlr port: 9093 nodePort: null + - name: kafka-client + targetPort: kafka-client + port: 9092 + nodePort: null - name: metrics targetPort: metrics port: 5556 diff --git a/deploy/kafka/templates/clusterversion.yaml b/deploy/kafka/templates/clusterversion.yaml index 774eed7a1..9bd1c2b48 100644 --- a/deploy/kafka/templates/clusterversion.yaml +++ b/deploy/kafka/templates/clusterversion.yaml @@ -1,7 +1,7 @@ apiVersion: apps.kubeblocks.io/v1alpha1 kind: ClusterVersion metadata: - name: kafka-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} + name: {{ include "kafka.name" . }}-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} labels: {{- include "kafka.labels" . | nindent 4 }} {{- if .Values.commonLabels }} diff --git a/deploy/kafka/templates/configconstraint.yaml b/deploy/kafka/templates/configconstraint.yaml index 4d1644e32..0c4eeacb1 100644 --- a/deploy/kafka/templates/configconstraint.yaml +++ b/deploy/kafka/templates/configconstraint.yaml @@ -1,7 +1,7 @@ apiVersion: apps.kubeblocks.io/v1alpha1 kind: ConfigConstraint metadata: - name: kafka-cc + name: {{ include "kafka.name" . }}-cc labels: {{- include "kafka.labels" . | nindent 4 }} {{- if .Values.commonLabels }} diff --git a/deploy/kafka/templates/configmap.yaml b/deploy/kafka/templates/configmap.yaml index f079dbaab..2043d0271 100644 --- a/deploy/kafka/templates/configmap.yaml +++ b/deploy/kafka/templates/configmap.yaml @@ -1,7 +1,7 @@ apiVersion: v1 kind: ConfigMap metadata: - name: kafka-configuration-tpl + name: {{ include "kafka.name" . }}-configuration-tpl namespace: {{ .Release.Namespace | quote }} labels: {{- include "common.labels.standard" . | nindent 4 }} {{- if .Values.commonLabels }} @@ -18,7 +18,7 @@ data: apiVersion: v1 kind: ConfigMap metadata: - name: kafka-jmx-configuration-tpl + name: {{ include "kafka.name" . }}-jmx-configuration-tpl data: jmx-kafka-prometheus.yml: |- jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi diff --git a/deploy/kafka/templates/scripts.yaml b/deploy/kafka/templates/scripts.yaml index d6a1dd123..4d7e4343a 100644 --- a/deploy/kafka/templates/scripts.yaml +++ b/deploy/kafka/templates/scripts.yaml @@ -1,7 +1,7 @@ apiVersion: v1 kind: ConfigMap metadata: - name: kafka-scripts-tpl + name: {{ include "kafka.name" . }}-scripts-tpl namespace: {{ .Release.Namespace | quote }} labels: {{- include "common.labels.standard" . | nindent 4 }} {{- if .Values.commonLabels }} diff --git a/deploy/kafka/templates/storageclass.yaml b/deploy/kafka/templates/storageclass.yaml new file mode 100644 index 000000000..7a29a9048 --- /dev/null +++ b/deploy/kafka/templates/storageclass.yaml @@ -0,0 +1,130 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-data-eks + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/parameters.md + type: st1 + "csi.storage.k8s.io/fstype": xfs +provisioner: kubernetes.io/aws-ebs +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-data-aks + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/driver-parameters.md + fsType: xfs + kind: managed + skuName: Standard_LRS +provisioner: kubernetes.io/azure-disk +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-data-gke + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#createvolume-parameters + type: pd-standard + # TODO: how-to specify FS type? +provisioner: kubernetes.io/gce-pd +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} + +--- +## storage classes for meta-data +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-meta-eks + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/parameters.md + type: {{ .Values.storageClassParameters.metadata.awsEBSVolumeType }} # io2, io1, gp3, gp2 are all SSD variant + blockExpress: {{ .Values.storageClassParameters.metadata.awsEBSEnableBlockExpress | default "false" | quote }} # for io2 only + "csi.storage.k8s.io/fstype": xfs +provisioner: kubernetes.io/aws-ebs +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-meta-aks + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/driver-parameters.md + fsType: xfs + kind: managed + skuName: StandardSSD_LRS # StandardSSD_LRS, UltraSSD_LRS, StandardSSD_ZRS (ZoneRedundantStorage) +provisioner: kubernetes.io/azure-disk +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: {{ include "kafka.name" . }}-meta-gke + labels: + {{- include "kafka.labels" . | nindent 4 }} + {{- if .Values.commonLabels }} + {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }} + {{- end }} + {{- if .Values.commonAnnotations }} + annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} + {{- end }} +parameters: + # parameters references: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#createvolume-parameters + type: pd-ssd # or pd-extreme + # TODO: how-to specify FS type? +provisioner: kubernetes.io/gce-pd +reclaimPolicy: Delete +volumeBindingMode: Immediate +mountOptions: {{ .Values.mountOptions | toYaml }} \ No newline at end of file diff --git a/deploy/kafka/values.yaml b/deploy/kafka/values.yaml index 5afa574dd..af3d97c1b 100644 --- a/deploy/kafka/values.yaml +++ b/deploy/kafka/values.yaml @@ -1,18 +1,16 @@ -# Default values for kafka. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - - +## Common override values: +## +## @param clusterVersionOverride +## @param nameOverride +## @param fullnameOverride clusterVersionOverride: "" nameOverride: "" fullnameOverride: "" - ## @param commonLabels Labels to add to all deployed objects ## commonLabels: {} - ## @param application images ## images: @@ -30,4 +28,21 @@ images: ## @param debugEnabled enables containers' debug logging ## -debugEnabled: true \ No newline at end of file +debugEnabled: true + +## @param mountOptions set the storageclass mountOptions attributes +## +mountOptions: + - noatime + - nobarrier + +## storageclass parameters settings +## +## @param storageClassParameters.metadata.awsEBSVolumeType +## @param storageClassParameters.metadata.awsEBSEnableBlockExpress +storageClassParameters: + ## storageclass for KRaft metadata + ## + metadata: + awsEBSVolumeType: io2 + awsEBSEnableBlockExpress: false diff --git a/githooks/scripts/run-lint.sh b/githooks/scripts/run-lint.sh index 79f2bcfbf..03d59e99a 100755 --- a/githooks/scripts/run-lint.sh +++ b/githooks/scripts/run-lint.sh @@ -1,2 +1,2 @@ #!/bin/sh -make vet fast-lint \ No newline at end of file +make fast-lint diff --git a/internal/cli/cmd/accounts/util.go b/internal/cli/cmd/accounts/util.go index 47d20d243..2ccf2c356 100644 --- a/internal/cli/cmd/accounts/util.go +++ b/internal/cli/cmd/accounts/util.go @@ -82,7 +82,7 @@ func fillCompInfoByName(ctx context.Context, dynamic dynamic.Interface, namespac if len(componentName) == 0 { compInfo.comp = &cluster.Spec.ComponentSpecs[0] } else { - compInfo.comp = cluster.GetComponentByName(componentName) + compInfo.comp = cluster.Spec.GetComponentByName(componentName) } if compInfo.comp == nil { return nil, fmt.Errorf("component %s not found in cluster %s", componentName, clusterName) diff --git a/internal/cli/cmd/cluster/create.go b/internal/cli/cmd/cluster/create.go index 25b404ec6..03ae63a02 100644 --- a/internal/cli/cmd/cluster/create.go +++ b/internal/cli/cmd/cluster/create.go @@ -497,7 +497,7 @@ func buildClusterComp(cd *appsv1alpha1.ClusterDefinition, setsMap map[string]map }, VolumeClaimTemplates: []appsv1alpha1.ClusterComponentVolumeClaimTemplate{{ Name: "data", - Spec: &corev1.PersistentVolumeClaimSpec{ + Spec: appsv1alpha1.PersistentVolumeClaimSpec{ AccessModes: []corev1.PersistentVolumeAccessMode{ corev1.ReadWriteOnce, }, diff --git a/internal/cli/testing/fake.go b/internal/cli/testing/fake.go index bf375a56a..1461d4f7f 100644 --- a/internal/cli/testing/fake.go +++ b/internal/cli/testing/fake.go @@ -109,7 +109,7 @@ func FakeCluster(name, namespace string, conditions ...metav1.Condition) *appsv1 VolumeClaimTemplates: []appsv1alpha1.ClusterComponentVolumeClaimTemplate{ { Name: "data", - Spec: &corev1.PersistentVolumeClaimSpec{ + Spec: appsv1alpha1.PersistentVolumeClaimSpec{ AccessModes: []corev1.PersistentVolumeAccessMode{ corev1.ReadWriteOnce, }, @@ -135,7 +135,7 @@ func FakeCluster(name, namespace string, conditions ...metav1.Condition) *appsv1 VolumeClaimTemplates: []appsv1alpha1.ClusterComponentVolumeClaimTemplate{ { Name: "data", - Spec: &corev1.PersistentVolumeClaimSpec{ + Spec: appsv1alpha1.PersistentVolumeClaimSpec{ AccessModes: []corev1.PersistentVolumeAccessMode{ corev1.ReadWriteOnce, }, diff --git a/internal/controller/builder/builder_test.go b/internal/controller/builder/builder_test.go index 16c0ab384..065dc55fb 100644 --- a/internal/controller/builder/builder_test.go +++ b/internal/controller/builder/builder_test.go @@ -106,11 +106,11 @@ var _ = Describe("builder", func() { clusterVersionObj = allFieldsClusterVersionObj(needCreate) } - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj := testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDefObj.Name, clusterVersionObj.Name). AddComponent(mysqlCompName, mysqlCompType).SetReplicas(1). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). AddService(testapps.ServiceVPCName, corev1.ServiceTypeLoadBalancer). AddService(testapps.ServiceInternetName, corev1.ServiceTypeLoadBalancer). GetObject() diff --git a/internal/controller/component/component_test.go b/internal/controller/component/component_test.go index 5a66f75df..ffe190b5e 100644 --- a/internal/controller/component/component_test.go +++ b/internal/controller/component/component_test.go @@ -65,11 +65,11 @@ var _ = Describe("component module", func() { AddInitContainerShort("nginx-init", testapps.NginxImage). AddContainerShort("nginx", testapps.NginxImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) diff --git a/internal/controller/component/type.go b/internal/controller/component/type.go index 7c7ec56c8..fbaaa91d6 100644 --- a/internal/controller/component/type.go +++ b/internal/controller/component/type.go @@ -17,7 +17,7 @@ limitations under the License. package component import ( - v12 "k8s.io/api/core/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/util/intstr" "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" @@ -30,29 +30,29 @@ type MonitorConfig struct { } type SynthesizedComponent struct { - ClusterDefName string `json:"clusterDefName,omitempty"` - Name string `json:"name,omitempty"` - Type string `json:"type,omitempty"` - CharacterType string `json:"characterType,omitempty"` - MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"` - Replicas int32 `json:"replicas"` - WorkloadType v1alpha1.WorkloadType `json:"workloadType,omitempty"` - ConsensusSpec *v1alpha1.ConsensusSetSpec `json:"consensusSpec,omitempty"` - PrimaryIndex *int32 `json:"primaryIndex,omitempty"` - PodSpec *v12.PodSpec `json:"podSpec,omitempty"` - Services []v12.Service `json:"services,omitempty"` - Probes *v1alpha1.ClusterDefinitionProbes `json:"probes,omitempty"` - VolumeClaimTemplates []v12.PersistentVolumeClaimTemplate `json:"volumeClaimTemplates,omitempty"` - Monitor *MonitorConfig `json:"monitor,omitempty"` - EnabledLogs []string `json:"enabledLogs,omitempty"` - LogConfigs []v1alpha1.LogConfig `json:"logConfigs,omitempty"` - ConfigTemplates []v1alpha1.ComponentConfigSpec `json:"configTemplates,omitempty"` - ScriptTemplates []v1alpha1.ComponentTemplateSpec `json:"scriptTemplates,omitempty"` - HorizontalScalePolicy *v1alpha1.HorizontalScalePolicy `json:"horizontalScalePolicy,omitempty"` - TLS bool `json:"tls"` - Issuer *v1alpha1.Issuer `json:"issuer,omitempty"` - VolumeTypes []v1alpha1.VolumeTypeSpec `json:"VolumeTypes,omitempty"` - CustomLabelSpecs []v1alpha1.CustomLabelSpec `json:"customLabelSpecs,omitempty"` + ClusterDefName string `json:"clusterDefName,omitempty"` + Name string `json:"name,omitempty"` + Type string `json:"type,omitempty"` + CharacterType string `json:"characterType,omitempty"` + MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty"` + Replicas int32 `json:"replicas"` + WorkloadType v1alpha1.WorkloadType `json:"workloadType,omitempty"` + ConsensusSpec *v1alpha1.ConsensusSetSpec `json:"consensusSpec,omitempty"` + PrimaryIndex *int32 `json:"primaryIndex,omitempty"` + PodSpec *corev1.PodSpec `json:"podSpec,omitempty"` + Services []corev1.Service `json:"services,omitempty"` + Probes *v1alpha1.ClusterDefinitionProbes `json:"probes,omitempty"` + VolumeClaimTemplates []corev1.PersistentVolumeClaimTemplate `json:"volumeClaimTemplates,omitempty"` + Monitor *MonitorConfig `json:"monitor,omitempty"` + EnabledLogs []string `json:"enabledLogs,omitempty"` + LogConfigs []v1alpha1.LogConfig `json:"logConfigs,omitempty"` + ConfigTemplates []v1alpha1.ComponentConfigSpec `json:"configTemplates,omitempty"` + ScriptTemplates []v1alpha1.ComponentTemplateSpec `json:"scriptTemplates,omitempty"` + HorizontalScalePolicy *v1alpha1.HorizontalScalePolicy `json:"horizontalScalePolicy,omitempty"` + TLS bool `json:"tls"` + Issuer *v1alpha1.Issuer `json:"issuer,omitempty"` + VolumeTypes []v1alpha1.VolumeTypeSpec `json:"VolumeTypes,omitempty"` + CustomLabelSpecs []v1alpha1.CustomLabelSpec `json:"customLabelSpecs,omitempty"` } // GetPrimaryIndex provides PrimaryIndex value getter, if PrimaryIndex is diff --git a/internal/controller/lifecycle/transformer_cluster.go b/internal/controller/lifecycle/transformer_cluster.go index d3e75b39e..856216447 100644 --- a/internal/controller/lifecycle/transformer_cluster.go +++ b/internal/controller/lifecycle/transformer_cluster.go @@ -69,7 +69,7 @@ func (c *clusterTransformer) Transform(dag *graph.DAG) error { return err } - clusterCompSpecMap := cluster.GetDefNameMappingComponents() + clusterCompSpecMap := cluster.Spec.GetDefNameMappingComponents() clusterCompVerMap := c.cc.cv.GetDefNameMappingComponents() process1stComp := true diff --git a/internal/controller/lifecycle/transformer_fill_class.go b/internal/controller/lifecycle/transformer_fill_class.go index 73651e9d4..60616cf42 100644 --- a/internal/controller/lifecycle/transformer_fill_class.go +++ b/internal/controller/lifecycle/transformer_fill_class.go @@ -141,7 +141,7 @@ func buildVolumeClaimByClass(cls *class.ComponentClass) []appsv1alpha1.ClusterCo for _, disk := range cls.Storage { volume := appsv1alpha1.ClusterComponentVolumeClaimTemplate{ Name: disk.Name, - Spec: &corev1.PersistentVolumeClaimSpec{ + Spec: appsv1alpha1.PersistentVolumeClaimSpec{ // TODO define access mode in class AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, Resources: corev1.ResourceRequirements{ diff --git a/internal/controller/lifecycle/validator_enable_logs.go b/internal/controller/lifecycle/validator_enable_logs.go index b18055fba..52c59f50f 100644 --- a/internal/controller/lifecycle/validator_enable_logs.go +++ b/internal/controller/lifecycle/validator_enable_logs.go @@ -27,5 +27,5 @@ type enableLogsValidator struct { func (e *enableLogsValidator) Validate() error { // validate config and send warning event log necessarily - return e.cluster.ValidateEnabledLogs(e.clusterDef) + return e.cluster.Spec.ValidateEnabledLogs(e.clusterDef) } diff --git a/internal/controller/plan/prepare_test.go b/internal/controller/plan/prepare_test.go index e2b3c968d..ae6861e16 100644 --- a/internal/controller/plan/prepare_test.go +++ b/internal/controller/plan/prepare_test.go @@ -128,11 +128,11 @@ var _ = Describe("Cluster Controller", func() { AddComponent(mysqlCompType). AddContainerShort("mysql", testapps.ApeCloudMySQLImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) @@ -184,11 +184,11 @@ var _ = Describe("Cluster Controller", func() { AddComponent(mysqlCompType). AddContainerShort("mysql", testapps.ApeCloudMySQLImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) @@ -237,11 +237,11 @@ var _ = Describe("Cluster Controller", func() { AddComponent(mysqlCompType). AddContainerShort("mysql", testapps.ApeCloudMySQLImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) @@ -301,11 +301,11 @@ var _ = Describe("Cluster Controller", func() { AddComponent(nginxCompType). AddContainerShort("nginx", testapps.NginxImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) @@ -418,13 +418,13 @@ var _ = Describe("Cluster Controller", func() { AddComponent(nginxCompType). AddContainerShort("nginx", testapps.NginxImage). GetObject() - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") cluster = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDef.Name, clusterVersion.Name). AddComponent(redisCompName, redisCompType). SetReplicas(2). SetPrimaryIndex(0). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). GetObject() }) diff --git a/internal/testutil/apps/cluster_consensus_test_util.go b/internal/testutil/apps/cluster_consensus_test_util.go index 2f3442f48..1f5c4a3a1 100644 --- a/internal/testutil/apps/cluster_consensus_test_util.go +++ b/internal/testutil/apps/cluster_consensus_test_util.go @@ -59,10 +59,10 @@ func CreateConsensusMysqlCluster( clusterName, workloadType, consensusCompName string) *appsv1alpha1.Cluster { - pvcSpec := NewPVC("2Gi") + pvcSpec := NewPVCSpec("2Gi") return NewClusterFactory(testCtx.DefaultNamespace, clusterName, clusterDefName, clusterVersionName). AddComponent(consensusCompName, workloadType).SetReplicas(3).SetEnabledLogs(errorLogName). - AddVolumeClaimTemplate("data", &pvcSpec).Create(&testCtx).GetObject() + AddVolumeClaimTemplate("data", pvcSpec).Create(&testCtx).GetObject() } // CreateConsensusMysqlClusterDef creates a mysql clusterDefinition with a component of ConsensusSet type. diff --git a/internal/testutil/apps/cluster_factory.go b/internal/testutil/apps/cluster_factory.go index 314fe3dc6..d6b517243 100644 --- a/internal/testutil/apps/cluster_factory.go +++ b/internal/testutil/apps/cluster_factory.go @@ -117,7 +117,7 @@ func (factory *MockClusterFactory) AddComponentToleration(toleration corev1.Tole } func (factory *MockClusterFactory) AddVolumeClaimTemplate(volumeName string, - pvcSpec *corev1.PersistentVolumeClaimSpec) *MockClusterFactory { + pvcSpec appsv1alpha1.PersistentVolumeClaimSpec) *MockClusterFactory { comps := factory.get().Spec.ComponentSpecs if len(comps) > 0 { comp := comps[len(comps)-1] diff --git a/internal/testutil/apps/cluster_util.go b/internal/testutil/apps/cluster_util.go index 534ca66a9..94dc37d16 100644 --- a/internal/testutil/apps/cluster_util.go +++ b/internal/testutil/apps/cluster_util.go @@ -20,6 +20,8 @@ import ( "context" "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" @@ -97,3 +99,15 @@ func GetClusterObservedGeneration(testCtx *testutil.TestContext, clusterKey type return cluster.Status.ObservedGeneration } } + +// NewPVCSpec create appsv1alpha1.PersistentVolumeClaimSpec. +func NewPVCSpec(size string) appsv1alpha1.PersistentVolumeClaimSpec { + return appsv1alpha1.PersistentVolumeClaimSpec{ + AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, + Resources: corev1.ResourceRequirements{ + Requests: corev1.ResourceList{ + corev1.ResourceStorage: resource.MustParse(size), + }, + }, + } +} diff --git a/internal/testutil/apps/native_object_util.go b/internal/testutil/apps/native_object_util.go index ad287f963..e1c07ff1b 100644 --- a/internal/testutil/apps/native_object_util.go +++ b/internal/testutil/apps/native_object_util.go @@ -54,8 +54,6 @@ func SetConfigMapData(key string, value string) func(*corev1.ConfigMap) { } } -// PVC - func NewPVC(size string) corev1.PersistentVolumeClaimSpec { return corev1.PersistentVolumeClaimSpec{ AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, diff --git a/test/integration/backup_mysql_test.go b/test/integration/backup_mysql_test.go index 17c25a17a..6e18d32f3 100644 --- a/test/integration/backup_mysql_test.go +++ b/test/integration/backup_mysql_test.go @@ -106,12 +106,12 @@ var _ = Describe("MySQL data protection function", func() { By("Create a cluster obj") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). SetReplicas(1). - AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) Eventually(testapps.GetClusterObservedGeneration(&testCtx, clusterKey)).Should(BeEquivalentTo(1)) diff --git a/test/integration/controller_suite_test.go b/test/integration/controller_suite_test.go index ca5d13fca..1a922b817 100644 --- a/test/integration/controller_suite_test.go +++ b/test/integration/controller_suite_test.go @@ -210,7 +210,7 @@ func CreateSimpleConsensusMySQLClusterWithConfig( Create(&testCtx).GetObject() By("Creating a cluster") - pvcSpec := &corev1.PersistentVolumeClaimSpec{ + pvcSpec := appsv1alpha1.PersistentVolumeClaimSpec{ AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, Resources: corev1.ResourceRequirements{ Requests: corev1.ResourceList{ diff --git a/test/integration/mysql_ha_test.go b/test/integration/mysql_ha_test.go index 37e5590be..94930d643 100644 --- a/test/integration/mysql_ha_test.go +++ b/test/integration/mysql_ha_test.go @@ -25,7 +25,6 @@ import ( appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" @@ -110,14 +109,7 @@ var _ = Describe("MySQL High-Availability function", func() { testThreeReplicasAndFailover := func() { By("Create a cluster obj") - pvcSpec := &corev1.PersistentVolumeClaimSpec{ - AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, - Resources: corev1.ResourceRequirements{ - Requests: corev1.ResourceList{ - corev1.ResourceStorage: resource.MustParse("1Gi"), - }, - }, - } + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). diff --git a/test/integration/mysql_scale_test.go b/test/integration/mysql_scale_test.go index 7d4e0e0f8..2cfe6bf2b 100644 --- a/test/integration/mysql_scale_test.go +++ b/test/integration/mysql_scale_test.go @@ -158,21 +158,14 @@ var _ = Describe("MySQL Scaling function", func() { } By("Create a cluster obj with both log and data volume of 1GB size") - dataPvcSpec := corev1.PersistentVolumeClaimSpec{ - AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce}, - Resources: corev1.ResourceRequirements{ - Requests: corev1.ResourceList{ - corev1.ResourceStorage: oldStorageValue, - }, - }, - } + dataPvcSpec := testapps.NewPVCSpec(oldStorageValue.String()) logPvcSpec := dataPvcSpec logPvcSpec.StorageClassName = &defaultStorageClass.Name clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(mysqlCompName, mysqlCompType). - AddVolumeClaimTemplate(testapps.DataVolumeName, &dataPvcSpec). - AddVolumeClaimTemplate(testapps.LogVolumeName, &logPvcSpec). + AddVolumeClaimTemplate(testapps.DataVolumeName, dataPvcSpec). + AddVolumeClaimTemplate(testapps.LogVolumeName, logPvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) diff --git a/test/integration/redis_hscale_test.go b/test/integration/redis_hscale_test.go index c5155cf79..5fec39ab2 100644 --- a/test/integration/redis_hscale_test.go +++ b/test/integration/redis_hscale_test.go @@ -88,12 +88,12 @@ var _ = Describe("Redis Horizontal Scale function", func() { testReplicationRedisHorizontalScale := func() { By("Mock a cluster obj with replication workloadType.") - pvcSpec := testapps.NewPVC("1Gi") + pvcSpec := testapps.NewPVCSpec("1Gi") clusterObj = testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterNamePrefix, clusterDefObj.Name, clusterVersionObj.Name).WithRandomName(). AddComponent(testapps.DefaultRedisCompName, testapps.DefaultRedisCompType). SetPrimaryIndex(testapps.DefaultReplicationPrimaryIndex). - SetReplicas(replicas).AddVolumeClaimTemplate(testapps.DataVolumeName, &pvcSpec). + SetReplicas(replicas).AddVolumeClaimTemplate(testapps.DataVolumeName, pvcSpec). Create(&testCtx).GetObject() clusterKey = client.ObjectKeyFromObject(clusterObj) From 9466d38eae49437ec93e0d31158f5969b1b2e4d7 Mon Sep 17 00:00:00 2001 From: Nayuta <111858489+nayutah@users.noreply.github.com> Date: Tue, 4 Apr 2023 10:57:10 +0800 Subject: [PATCH 40/80] chore: gptplugin support openapi.yaml & ai-plugin.json config (#2404) --- .../templates/configmap.yaml | 306 ++++++++++++++++++ .../templates/deployment.yaml | 17 + deploy/chatgpt-retrieval-plugin/values.yaml | 9 + 3 files changed, 332 insertions(+) create mode 100644 deploy/chatgpt-retrieval-plugin/templates/configmap.yaml diff --git a/deploy/chatgpt-retrieval-plugin/templates/configmap.yaml b/deploy/chatgpt-retrieval-plugin/templates/configmap.yaml new file mode 100644 index 000000000..aa8bb62a0 --- /dev/null +++ b/deploy/chatgpt-retrieval-plugin/templates/configmap.yaml @@ -0,0 +1,306 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: gptplugin-config + namespace: {{ .Release.Namespace | quote }} + labels: + {{- include "gptplugin.labels" . | nindent 4 }} + +data: + ai-plugin.json: |- + { + "schema_version": "v1", + "name_for_model": "retrieval", + "name_for_human": "Retrieval Plugin", + "description_for_model": "Plugin for searching through the user's documents (such as files, emails, and more) to find answers to questions and retrieve relevant information. Use it whenever a user asks something that might be found in their personal information, or asks you to save information for later.", + "description_for_human": "Search through your documents.", + "auth": { + "type": "user_http", + "authorization_type": "bearer" + }, + "api": { + "type": "openapi", + "url": "{{- .Values.website.url | default "https://your-app-url.com/.well-known/openapi.yaml"}}", + "has_user_authentication": false + }, + "logo_url": "{{- .Values.website.logo_url | default "https://your-app-url.com/.well-known/logo.png"}}", + "contact_email": "{{- .Values.website.contact_email | default "hello@contact.com"}}", + "legal_info_url": "{{- .Values.website.legal_info_url | default "hello@legal.com"}}" + } + openapi.yaml: |- + openapi: 3.0.2 + info: + title: Retrieval Plugin API + description: A retrieval API for querying and filtering documents based on natural language queries and metadata + version: 1.0.0 + servers: + - url: "{{- .Values.servers.url | default "https://your-app-url.com"}}" + paths: + /upsert: + post: + summary: Upsert + description: Save chat information. Accepts an array of documents with text (potential questions + conversation text), metadata (source 'chat' and timestamp, no ID as this will be generated). Confirm with the user before saving, ask for more details/context. + operationId: upsert_upsert_post + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/UpsertRequest" + required: true + responses: + "200": + description: Successful Response + content: + application/json: + schema: + $ref: "#/components/schemas/UpsertResponse" + "422": + description: Validation Error + content: + application/json: + schema: + $ref: "#/components/schemas/HTTPValidationError" + security: + - HTTPBearer: [ ] + /query: + post: + summary: Query + description: Accepts search query objects array each with query and optional filter. Break down complex questions into sub-questions. Refine results by criteria, e.g. time / source, don't do this often. Split queries if ResponseTooLargeError occurs. + operationId: query_query_post + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/QueryRequest" + required: true + responses: + "200": + description: Successful Response + content: + application/json: + schema: + $ref: "#/components/schemas/QueryResponse" + "422": + description: Validation Error + content: + application/json: + schema: + $ref: "#/components/schemas/HTTPValidationError" + security: + - HTTPBearer: [ ] + components: + schemas: + Document: + title: Document + required: + - text + type: object + properties: + id: + title: Id + type: string + text: + title: Text + type: string + metadata: + $ref: "#/components/schemas/DocumentMetadata" + DocumentChunkMetadata: + title: DocumentChunkMetadata + type: object + properties: + source: + $ref: "#/components/schemas/Source" + source_id: + title: Source Id + type: string + url: + title: Url + type: string + created_at: + title: Created At + type: string + author: + title: Author + type: string + document_id: + title: Document Id + type: string + DocumentChunkWithScore: + title: DocumentChunkWithScore + required: + - text + - metadata + - score + type: object + properties: + id: + title: Id + type: string + text: + title: Text + type: string + metadata: + $ref: "#/components/schemas/DocumentChunkMetadata" + embedding: + title: Embedding + type: array + items: + type: number + score: + title: Score + type: number + DocumentMetadata: + title: DocumentMetadata + type: object + properties: + source: + $ref: "#/components/schemas/Source" + source_id: + title: Source Id + type: string + url: + title: Url + type: string + created_at: + title: Created At + type: string + author: + title: Author + type: string + DocumentMetadataFilter: + title: DocumentMetadataFilter + type: object + properties: + document_id: + title: Document Id + type: string + source: + $ref: "#/components/schemas/Source" + source_id: + title: Source Id + type: string + author: + title: Author + type: string + start_date: + title: Start Date + type: string + end_date: + title: End Date + type: string + HTTPValidationError: + title: HTTPValidationError + type: object + properties: + detail: + title: Detail + type: array + items: + $ref: "#/components/schemas/ValidationError" + Query: + title: Query + required: + - query + type: object + properties: + query: + title: Query + type: string + filter: + $ref: "#/components/schemas/DocumentMetadataFilter" + top_k: + title: Top K + type: integer + default: 3 + QueryRequest: + title: QueryRequest + required: + - queries + type: object + properties: + queries: + title: Queries + type: array + items: + $ref: "#/components/schemas/Query" + QueryResponse: + title: QueryResponse + required: + - results + type: object + properties: + results: + title: Results + type: array + items: + $ref: "#/components/schemas/QueryResult" + QueryResult: + title: QueryResult + required: + - query + - results + type: object + properties: + query: + title: Query + type: string + results: + title: Results + type: array + items: + $ref: "#/components/schemas/DocumentChunkWithScore" + Source: + title: Source + enum: + - email + - file + - chat + type: string + description: An enumeration. + UpsertRequest: + title: UpsertRequest + required: + - documents + type: object + properties: + documents: + title: Documents + type: array + items: + $ref: "#/components/schemas/Document" + UpsertResponse: + title: UpsertResponse + required: + - ids + type: object + properties: + ids: + title: Ids + type: array + items: + type: string + ValidationError: + title: ValidationError + required: + - loc + - msg + - type + type: object + properties: + loc: + title: Location + type: array + items: + anyOf: + - type: string + - type: integer + msg: + title: Message + type: string + type: + title: Error Type + type: string + securitySchemes: + HTTPBearer: + type: http + scheme: bearer \ No newline at end of file diff --git a/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml index e605215ce..f3e250980 100644 --- a/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml +++ b/deploy/chatgpt-retrieval-plugin/templates/deployment.yaml @@ -33,6 +33,13 @@ spec: {{- toYaml .Values.securityContext | nindent 12 }} image: "{{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} + volumeMounts: + - mountPath: /code/.well-known/ai-plugin.json + name: config + subPath: ai-plugin.json + - mountPath: /code/.well-known/openapi.yaml + name: config + subPath: openapi.yaml ports: - name: http containerPort: 8080 @@ -134,3 +141,13 @@ spec: tolerations: {{- toYaml . | nindent 8 }} {{- end }} + + volumes: + - name: config + configMap: + name: gptplugin-config + items: + - key: ai-plugin.json + path: ai-plugin.json + - key: openapi.yaml + path: openapi.yaml diff --git a/deploy/chatgpt-retrieval-plugin/values.yaml b/deploy/chatgpt-retrieval-plugin/values.yaml index 610591ec7..282dbf940 100644 --- a/deploy/chatgpt-retrieval-plugin/values.yaml +++ b/deploy/chatgpt-retrieval-plugin/values.yaml @@ -41,6 +41,15 @@ service: type: ClusterIP port: 8080 +servers: + url: https://your-app-url.com + +website: + url: https://your-app-url.com/.well-known/openapi.yaml + logo_url: https://your-app-url.com/.well-known/logo.png + contact_email: hello@contact.com + legal_info_url: hello@legal.com + datastore: # in list of (pinecone, weaviate, zilliz, milvus, qdrant, redis) DATASTORE: From 912c31a5b3ea1966de86ac7e32bfefbf7deedddc Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Tue, 4 Apr 2023 11:40:58 +0800 Subject: [PATCH 41/80] fix: playground does not output error message when kubernetes cluster is not ready (#2391) --- Makefile | 1 - internal/cli/cloudprovider/interface.go | 2 +- internal/cli/cloudprovider/k3d.go | 2 +- internal/cli/cloudprovider/provider.go | 6 +- internal/cli/cloudprovider/terraform.go | 15 +-- internal/cli/cmd/playground/destroy.go | 5 +- internal/cli/cmd/playground/init.go | 117 +++++++++++++---------- internal/cli/cmd/playground/init_test.go | 2 +- internal/cli/cmd/playground/types.go | 4 + 9 files changed, 85 insertions(+), 69 deletions(-) diff --git a/Makefile b/Makefile index 46ac19038..2dc5e3f82 100644 --- a/Makefile +++ b/Makefile @@ -269,7 +269,6 @@ kbcli-doc: generate ## generate CLI command reference manual. $(GO) run ./hack/docgen/cli/main.go ./docs/user_docs/cli - ##@ Operator Controller Manager .PHONY: manager diff --git a/internal/cli/cloudprovider/interface.go b/internal/cli/cloudprovider/interface.go index 996df73f9..8ba91bdbc 100644 --- a/internal/cli/cloudprovider/interface.go +++ b/internal/cli/cloudprovider/interface.go @@ -28,7 +28,7 @@ type Interface interface { Name() string // CreateK8sCluster creates a kubernetes cluster - CreateK8sCluster(clusterInfo *K8sClusterInfo, init bool) error + CreateK8sCluster(clusterInfo *K8sClusterInfo) error // DeleteK8sCluster deletes the created kubernetes cluster DeleteK8sCluster(clusterInfo *K8sClusterInfo) error diff --git a/internal/cli/cloudprovider/k3d.go b/internal/cli/cloudprovider/k3d.go index 18e007ab9..3561782fa 100644 --- a/internal/cli/cloudprovider/k3d.go +++ b/internal/cli/cloudprovider/k3d.go @@ -89,7 +89,7 @@ func (p *localCloudProvider) Name() string { } // CreateK8sCluster create a local kubernetes cluster using k3d -func (p *localCloudProvider) CreateK8sCluster(clusterInfo *K8sClusterInfo, init bool) error { +func (p *localCloudProvider) CreateK8sCluster(clusterInfo *K8sClusterInfo) error { var err error if p.cfg, err = buildClusterRunConfig(clusterInfo.ClusterName); err != nil { diff --git a/internal/cli/cloudprovider/provider.go b/internal/cli/cloudprovider/provider.go index 2e6d91dc2..abc89b6d1 100644 --- a/internal/cli/cloudprovider/provider.go +++ b/internal/cli/cloudprovider/provider.go @@ -57,7 +57,7 @@ func (p *cloudProvider) Name() string { } // CreateK8sCluster create a kubernetes cluster -func (p *cloudProvider) CreateK8sCluster(clusterInfo *K8sClusterInfo, init bool) error { +func (p *cloudProvider) CreateK8sCluster(clusterInfo *K8sClusterInfo) error { // init terraform fmt.Fprintf(p.stdout, "Check and install terraform... \n") if err := initTerraform(); err != nil { @@ -66,7 +66,7 @@ func (p *cloudProvider) CreateK8sCluster(clusterInfo *K8sClusterInfo, init bool) // create cluster fmt.Fprintf(p.stdout, "\nInit and apply %s in %s\n", K8sService(p.name), p.tfPath) - return tfInitAndApply(p.tfPath, init, p.stdout, p.stderr, clusterInfo.buildApplyOpts()...) + return tfInitAndApply(p.tfPath, p.stdout, p.stderr, clusterInfo.buildApplyOpts()...) } func (p *cloudProvider) DeleteK8sCluster(clusterInfo *K8sClusterInfo) error { @@ -85,7 +85,7 @@ func (p *cloudProvider) DeleteK8sCluster(clusterInfo *K8sClusterInfo) error { // destroy cluster fmt.Fprintf(p.stdout, "\nDestroy %s cluster in %s\n", K8sService(p.name), p.tfPath) - return tfDestroy(p.tfPath, p.stdout, p.stderr, clusterInfo.buildDestroyOpts()...) + return tfInitAndDestroy(p.tfPath, p.stdout, p.stderr, clusterInfo.buildDestroyOpts()...) } func (p *cloudProvider) GetClusterInfo() (*K8sClusterInfo, error) { diff --git a/internal/cli/cloudprovider/terraform.go b/internal/cli/cloudprovider/terraform.go index 68f8bdad8..d52919e0f 100644 --- a/internal/cli/cloudprovider/terraform.go +++ b/internal/cli/cloudprovider/terraform.go @@ -77,17 +77,15 @@ func initTerraform() error { return nil } -func tfInitAndApply(workingDir string, init bool, stdout, stderr io.Writer, opts ...tfexec.ApplyOption) error { +func tfInitAndApply(workingDir string, stdout, stderr io.Writer, opts ...tfexec.ApplyOption) error { ctx := context.Background() tf, err := newTerraform(workingDir, stdout, stderr) if err != nil { return err } - if init { - if err = tf.Init(ctx, tfexec.Upgrade(true)); err != nil { - return err - } + if err = tf.Init(ctx, tfexec.Upgrade(false)); err != nil { + return err } if err = tf.Apply(ctx, opts...); err != nil { @@ -96,12 +94,17 @@ func tfInitAndApply(workingDir string, init bool, stdout, stderr io.Writer, opts return nil } -func tfDestroy(workingDir string, stdout, stderr io.Writer, opts ...tfexec.DestroyOption) error { +func tfInitAndDestroy(workingDir string, stdout, stderr io.Writer, opts ...tfexec.DestroyOption) error { ctx := context.Background() tf, err := newTerraform(workingDir, stdout, stderr) if err != nil { return err } + + if err = tf.Init(ctx, tfexec.Upgrade(false)); err != nil { + return err + } + return tf.Destroy(ctx, opts...) } diff --git a/internal/cli/cmd/playground/destroy.go b/internal/cli/cmd/playground/destroy.go index ccc53379d..249953ab8 100644 --- a/internal/cli/cmd/playground/destroy.go +++ b/internal/cli/cmd/playground/destroy.go @@ -164,10 +164,9 @@ func (o *destroyOptions) destroyCloud() error { } func (o *destroyOptions) removeKubeConfig() error { - configPath := util.ConfigPath("config") - spinner := printer.Spinner(o.Out, "Remove kubeconfig from %s", configPath) + spinner := printer.Spinner(o.Out, "Remove kubeconfig from %s", defaultKubeConfigPath) defer spinner(false) - if err := kubeConfigRemove(o.prevCluster.KubeConfig, configPath); err != nil { + if err := kubeConfigRemove(o.prevCluster.KubeConfig, defaultKubeConfigPath); err != nil { return err } spinner(true) diff --git a/internal/cli/cmd/playground/init.go b/internal/cli/cmd/playground/init.go index dd62f9e57..47867099b 100644 --- a/internal/cli/cmd/playground/init.go +++ b/internal/cli/cmd/playground/init.go @@ -26,6 +26,7 @@ import ( "github.com/spf13/cobra" "golang.org/x/exp/slices" "k8s.io/apimachinery/pkg/util/rand" + "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/klog/v2" cmdutil "k8s.io/kubectl/pkg/cmd/util" @@ -156,7 +157,7 @@ func (o *initOptions) local() error { // create a local kubernetes cluster (k3d cluster) to deploy KubeBlocks spinner := printer.Spinner(o.Out, "%-40s", "Create k3d cluster: "+clusterInfo.ClusterName) defer spinner(false) - if err = provider.CreateK8sCluster(clusterInfo, true); err != nil { + if err = provider.CreateK8sCluster(clusterInfo); err != nil { return errors.Wrap(err, "failed to set up k3d cluster") } spinner(true) @@ -169,52 +170,6 @@ func (o *initOptions) local() error { return o.installKBAndCluster(clusterInfo.ClusterName) } -func (o *initOptions) installKBAndCluster(k8sClusterName string) error { - var err error - - // playground always use the default kubeconfig at ~/.kube/config - configPath := util.ConfigPath("config") - if err = util.SetKubeConfig(configPath); err != nil { - return err - } - - // create helm config - o.helmCfg = helm.NewConfig("", configPath, "", klog.V(1).Enabled()) - - // Install KubeBlocks - if err = o.installKubeBlocks(k8sClusterName); err != nil { - return errors.Wrap(err, "failed to install KubeBlocks") - } - - // Install database cluster - clusterInfo := "ClusterDefinition: " + o.clusterDef - if o.clusterVersion != "" { - clusterInfo += ", ClusterVersion: " + o.clusterVersion - } - spinner := printer.Spinner(o.Out, "Create cluster %s (%s)", kbClusterName, clusterInfo) - defer spinner(false) - if err = o.createCluster(); err != nil { - return errors.Wrapf(err, "failed to create cluster %s", kbClusterName) - } - spinner(true) - - // Print guide information - fmt.Fprintf(os.Stdout, "\nKubeBlocks playground init SUCCESSFULLY!\n\n") - if k8sClusterName != "" { - fmt.Fprintf(os.Stdout, "Kubernetes cluster \"%s\" has been created.\n", k8sClusterName) - } - fmt.Fprintf(os.Stdout, "Cluster \"%s\" has been created.\n", kbClusterName) - - // output elapsed time - if !o.startTime.IsZero() { - fmt.Fprintf(o.Out, "Elapsed time: %s\n", time.Since(o.startTime).Truncate(time.Second)) - } - - printGuide() - - return nil -} - // bootstraps a playground in the remote cloud func (o *initOptions) cloud() error { cpPath, err := cloudProviderRepoDir() @@ -268,11 +223,13 @@ func (o *initOptions) cloud() error { return err } - if err = provider.CreateK8sCluster(clusterInfo, true); err != nil { + // create a kubernetes cluster in the cloud + if err = provider.CreateK8sCluster(clusterInfo); err != nil { return err } printer.PrintBlankLine(o.Out) + // write cluster kubeconfig to local kubeconfig file and switch current context to it if err = o.setKubeConfig(provider); err != nil { return err } @@ -289,7 +246,8 @@ func (o *initOptions) confirmToContinue() error { fmt.Fprintf(o.Out, "\nPlayground init cancelled, please destroy the old cluster first.\n") return cmdutil.ErrExit } - fmt.Fprintf(o.Out, "Continue to initialize %s %s cluster %s... \n", o.cloudProvider, cp.K8sService(o.cloudProvider), clusterName) + fmt.Fprintf(o.Out, "Continue to initialize %s %s cluster %s... \n", + o.cloudProvider, cp.K8sService(o.cloudProvider), clusterName) return nil } @@ -328,14 +286,15 @@ func (o *initOptions) setKubeConfig(provider cp.Interface) error { return errors.New("failed to get kubernetes cluster kubeconfig") } if err = writeClusterInfoToFile(o.stateFilePath, clusterInfo); err != nil { - return errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", o.stateFilePath, clusterInfo) + return errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", + o.stateFilePath, clusterInfo) } // merge created kubernetes cluster kubeconfig to ~/.kube/config and set it as default - configPath := util.ConfigPath("config") - spinner := printer.Spinner(o.Out, "Write kubeconfig to %s", configPath) + spinner := printer.Spinner(o.Out, "Write kubeconfig to %s", defaultKubeConfigPath) defer spinner(false) - if err = kubeConfigWrite(clusterInfo.KubeConfig, configPath, writeKubeConfigOptions{UpdateExisting: true, UpdateCurrentContext: true}); err != nil { + if err = kubeConfigWrite(clusterInfo.KubeConfig, defaultKubeConfigPath, + writeKubeConfigOptions{UpdateExisting: true, UpdateCurrentContext: true}); err != nil { return errors.Wrapf(err, "failed to write cluster %s kubeconfig", clusterInfo.ClusterName) } spinner(true) @@ -351,6 +310,58 @@ func (o *initOptions) setKubeConfig(provider cp.Interface) error { return nil } +func (o *initOptions) installKBAndCluster(k8sClusterName string) error { + var err error + + // when the kubernetes cluster is not ready, the runtime will output the error + // message like "couldn't get resource list for", we ignore it + runtime.ErrorHandlers[0] = func(err error) { + if klog.V(1).Enabled() { + klog.ErrorDepth(2, err) + } + } + + // playground always use the default kubeconfig at ~/.kube/config + if err = util.SetKubeConfig(defaultKubeConfigPath); err != nil { + return err + } + + // create helm config + o.helmCfg = helm.NewConfig("", defaultKubeConfigPath, "", klog.V(1).Enabled()) + + // Install KubeBlocks + if err = o.installKubeBlocks(k8sClusterName); err != nil { + return errors.Wrap(err, "failed to install KubeBlocks") + } + + // Install database cluster + clusterInfo := "ClusterDefinition: " + o.clusterDef + if o.clusterVersion != "" { + clusterInfo += ", ClusterVersion: " + o.clusterVersion + } + spinner := printer.Spinner(o.Out, "Create cluster %s (%s)", kbClusterName, clusterInfo) + defer spinner(false) + if err = o.createCluster(); err != nil { + return errors.Wrapf(err, "failed to create cluster %s", kbClusterName) + } + spinner(true) + + // Print guide information + fmt.Fprintf(os.Stdout, "\nKubeBlocks playground init SUCCESSFULLY!\n\n") + if k8sClusterName != "" { + fmt.Fprintf(os.Stdout, "Kubernetes cluster \"%s\" has been created.\n", k8sClusterName) + } + fmt.Fprintf(os.Stdout, "Cluster \"%s\" has been created.\n", kbClusterName) + + // output elapsed time + if !o.startTime.IsZero() { + fmt.Fprintf(o.Out, "Elapsed time: %s\n", time.Since(o.startTime).Truncate(time.Second)) + } + + printGuide() + return nil +} + func (o *initOptions) installKubeBlocks(k8sClusterName string) error { f := util.NewFactory() client, err := f.KubernetesClientSet() diff --git a/internal/cli/cmd/playground/init_test.go b/internal/cli/cmd/playground/init_test.go index 3bfd4031b..f6ae12944 100644 --- a/internal/cli/cmd/playground/init_test.go +++ b/internal/cli/cmd/playground/init_test.go @@ -44,7 +44,7 @@ var _ = Describe("playground", func() { clusterVersion: clitesting.ClusterVersionName, IOStreams: streams, cloudProvider: defaultCloudProvider, - helmCfg: helm.NewConfig("", util.ConfigPath("config"), "", false), + helmCfg: helm.NewConfig("", util.ConfigPath("config_kb_test"), "", false), } Expect(o.validate()).Should(Succeed()) Expect(o.run()).Should(HaveOccurred()) diff --git a/internal/cli/cmd/playground/types.go b/internal/cli/cmd/playground/types.go index 6e2ddfcd5..95e01ae0b 100644 --- a/internal/cli/cmd/playground/types.go +++ b/internal/cli/cmd/playground/types.go @@ -18,6 +18,7 @@ package playground import ( "github.com/apecloud/kubeblocks/internal/cli/cloudprovider" + "github.com/apecloud/kubeblocks/internal/cli/util" ) const ( @@ -41,6 +42,9 @@ const ( var ( // kbClusterName is the playground cluster name that created by KubeBlocks kbClusterName = "mycluster" + + // defaultKubeConfigPath is the default kubeconfig path, it is ~/.kube/config + defaultKubeConfigPath = util.ConfigPath("config") ) var guideStr = ` From c288cba63c0bcffe1d5761a183744ff73c148548 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Tue, 4 Apr 2023 14:37:54 +0800 Subject: [PATCH 42/80] chore: tidyup ClusterDefintion.spec.componentDefs.service go struct type, remove corev1.ServiceSpec usage (#2410) --- apis/apps/v1alpha1/cluster_types.go | 33 +- apis/apps/v1alpha1/clusterdefinition_types.go | 79 ++++- ...apps.kubeblocks.io_clusterdefinitions.yaml | 292 +----------------- .../templates/clusterdefinition.yaml | 65 ++-- .../templates/clusterdefinition.yaml | 2 +- .../templates/clusterdefinition.yaml | 17 - deploy/etcd/templates/clusterdefinition.yaml | 5 +- ...apps.kubeblocks.io_clusterdefinitions.yaml | 292 +----------------- deploy/kafka/templates/clusterdefinition.yaml | 11 - .../milvus/templates/clusterdefinition.yaml | 4 - .../mongodb/templates/clusterdefinition.yaml | 20 +- .../templates/clusterdefinition.yaml | 1 - .../templates/clusterdefinition.yaml | 1 - .../qdrant/templates/clusterdefinition.yaml | 2 - deploy/redis/templates/clusterdefinition.yaml | 6 - .../weaviate/templates/clusterdefinition.yaml | 1 - internal/controller/component/component.go | 4 +- internal/testutil/apps/clusterdef_factory.go | 8 +- internal/testutil/apps/constant.go | 16 +- 19 files changed, 146 insertions(+), 713 deletions(-) diff --git a/apis/apps/v1alpha1/cluster_types.go b/apis/apps/v1alpha1/cluster_types.go index 01f19cf7f..79dbd2aa4 100644 --- a/apis/apps/v1alpha1/cluster_types.go +++ b/apis/apps/v1alpha1/cluster_types.go @@ -506,50 +506,47 @@ func (r ClusterSpec) GetDefNameMappingComponents() map[string][]ClusterComponent } // GetMessage gets message map deep copy object -func (in *ClusterComponentStatus) GetMessage() ComponentMessageMap { +func (r ClusterComponentStatus) GetMessage() ComponentMessageMap { messageMap := map[string]string{} - for k, v := range in.Message { + for k, v := range r.Message { messageMap[k] = v } return messageMap } // SetMessage override message map object -func (in *ClusterComponentStatus) SetMessage(messageMap ComponentMessageMap) { - if in == nil { +func (r *ClusterComponentStatus) SetMessage(messageMap ComponentMessageMap) { + if r == nil { return } - in.Message = messageMap + r.Message = messageMap } // SetObjectMessage sets k8s workload message to component status message map -func (in *ClusterComponentStatus) SetObjectMessage(objectKind, objectName, message string) { - if in == nil { +func (r *ClusterComponentStatus) SetObjectMessage(objectKind, objectName, message string) { + if r == nil { return } - if in.Message == nil { - in.Message = map[string]string{} + if r.Message == nil { + r.Message = map[string]string{} } messageKey := fmt.Sprintf("%s/%s", objectKind, objectName) - in.Message[messageKey] = message + r.Message[messageKey] = message } // GetObjectMessage gets the k8s workload message in component status message map -func (in *ClusterComponentStatus) GetObjectMessage(objectKind, objectName string) string { - if in == nil { - return "" - } +func (r ClusterComponentStatus) GetObjectMessage(objectKind, objectName string) string { messageKey := fmt.Sprintf("%s/%s", objectKind, objectName) - return in.Message[messageKey] + return r.Message[messageKey] } // SetObjectMessage sets k8s workload message to component status message map -func (m ComponentMessageMap) SetObjectMessage(objectKind, objectName, message string) { - if m == nil { +func (r ComponentMessageMap) SetObjectMessage(objectKind, objectName, message string) { + if r == nil { return } messageKey := fmt.Sprintf("%s/%s", objectKind, objectName) - m[messageKey] = message + r[messageKey] = message } // SetComponentStatus does safe operation on ClusterStatus.Components map object update. diff --git a/apis/apps/v1alpha1/clusterdefinition_types.go b/apis/apps/v1alpha1/clusterdefinition_types.go index d4f5ada35..2e693349b 100644 --- a/apis/apps/v1alpha1/clusterdefinition_types.go +++ b/apis/apps/v1alpha1/clusterdefinition_types.go @@ -354,10 +354,8 @@ type ClusterComponentDefinition struct { // service defines the behavior of a service spec. // provide read-write service when WorkloadType is Consensus. - // https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status - // +kubebuilder:pruning:PreserveUnknownFields // +optional - Service *corev1.ServiceSpec `json:"service,omitempty"` + Service *ServiceSpec `json:"service,omitempty"` // consensusSpec defines consensus related spec if workloadType is Consensus, required if workloadType is Consensus. // +optional @@ -398,6 +396,80 @@ type ClusterComponentDefinition struct { CustomLabelSpecs []CustomLabelSpec `json:"customLabelSpecs,omitempty"` } +type ServiceSpec struct { + // The list of ports that are exposed by this service. + // More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies + // +patchMergeKey=port + // +patchStrategy=merge + // +listType=map + // +listMapKey=port + // +listMapKey=protocol + Ports []ServicePort `json:"ports,omitempty" patchStrategy:"merge" patchMergeKey:"port" protobuf:"bytes,1,rep,name=ports"` +} + +func (r *ServiceSpec) toSVCPorts() []corev1.ServicePort { + ports := make([]corev1.ServicePort, 0, len(r.Ports)) + for _, p := range r.Ports { + ports = append(ports, p.toSVCPort()) + } + return ports +} + +func (r ServiceSpec) ToSVCSpec() corev1.ServiceSpec { + return corev1.ServiceSpec{ + Ports: r.toSVCPorts(), + } +} + +type ServicePort struct { + // The name of this port within the service. This must be a DNS_LABEL. + // All ports within a ServiceSpec must have unique names. When considering + // the endpoints for a Service, this must match the 'name' field in the + // EndpointPort. + // +kubebuilder:validation:Required + Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"` + + // The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". + // Default is TCP. + // +kubebuilder:validation:Enum={TCP,UDP,SCTP} + // +default="TCP" + // +optional + Protocol corev1.Protocol `json:"protocol,omitempty" protobuf:"bytes,2,opt,name=protocol,casttype=Protocol"` + + // The application protocol for this port. + // This field follows standard Kubernetes label syntax. + // Un-prefixed names are reserved for IANA standard service names (as per + // RFC-6335 and https://www.iana.org/assignments/service-names). + // Non-standard protocols should use prefixed names such as + // mycompany.com/my-custom-protocol. + // +optional + AppProtocol *string `json:"appProtocol,omitempty" protobuf:"bytes,6,opt,name=appProtocol"` + + // The port that will be exposed by this service. + Port int32 `json:"port" protobuf:"varint,3,opt,name=port"` + + // Number or name of the port to access on the pods targeted by the service. + // Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. + // If this is a string, it will be looked up as a named port in the + // target Pod's container ports. If this is not specified, the value + // of the 'port' field is used (an identity map). + // This field is ignored for services with clusterIP=None, and should be + // omitted or set equal to the 'port' field. + // More info: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service + // +optional + TargetPort intstr.IntOrString `json:"targetPort,omitempty" protobuf:"bytes,4,opt,name=targetPort"` +} + +func (r *ServicePort) toSVCPort() corev1.ServicePort { + return corev1.ServicePort{ + Name: r.Name, + Protocol: r.Protocol, + AppProtocol: r.AppProtocol, + Port: r.Port, + TargetPort: r.TargetPort, + } +} + type HorizontalScalePolicy struct { // type controls what kind of data synchronization do when component scale out. // Policy is in enum of {None, Snapshot}. The default policy is `None`. @@ -454,7 +526,6 @@ type ClusterDefinitionProbe struct { } type ClusterDefinitionProbes struct { - // Probe for DB running check. // +optional RunningProbe *ClusterDefinitionProbe `json:"runningProbe,omitempty"` diff --git a/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml b/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml index b9595c56e..599bccf91 100644 --- a/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml +++ b/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml @@ -8344,214 +8344,11 @@ spec: service: description: service defines the behavior of a service spec. provide read-write service when WorkloadType is Consensus. - https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status properties: - allocateLoadBalancerNodePorts: - description: allocateLoadBalancerNodePorts defines if NodePorts - will be automatically allocated for services with type - LoadBalancer. Default is "true". It may be set to "false" - if the cluster load-balancer does not rely on NodePorts. If - the caller requests specific NodePorts (by specifying - a value), those requests will be respected, regardless - of this field. This field may only be set for services - with type LoadBalancer and will be cleared if the type - is changed to any other type. - type: boolean - clusterIP: - description: 'clusterIP is the IP address of the service - and is usually assigned randomly. If an address is specified - manually, is in-range (as per system configuration), and - is not in use, it will be allocated to the service; otherwise - creation of the service will fail. This field may not - be changed through updates unless the type field is also - being changed to ExternalName (which requires this field - to be blank) or the type field is being changed from ExternalName - (in which case this field may optionally be specified, - as describe above). Valid values are "None", empty string - (""), or a valid IP address. Setting this to "None" makes - a "headless service" (no virtual IP), which is useful - when direct endpoint connections are preferred and proxying - is not required. Only applies to types ClusterIP, NodePort, - and LoadBalancer. If this field is specified when creating - a Service of type ExternalName, creation will fail. This - field will be wiped when updating a Service to type ExternalName. - More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' - type: string - clusterIPs: - description: "ClusterIPs is a list of IP addresses assigned - to this service, and are usually assigned randomly. If - an address is specified manually, is in-range (as per - system configuration), and is not in use, it will be allocated - to the service; otherwise creation of the service will - fail. This field may not be changed through updates unless - the type field is also being changed to ExternalName (which - requires this field to be empty) or the type field is - being changed from ExternalName (in which case this field - may optionally be specified, as describe above). Valid - values are \"None\", empty string (\"\"), or a valid IP - address. Setting this to \"None\" makes a \"headless - service\" (no virtual IP), which is useful when direct - endpoint connections are preferred and proxying is not - required. Only applies to types ClusterIP, NodePort, - and LoadBalancer. If this field is specified when creating - a Service of type ExternalName, creation will fail. This - field will be wiped when updating a Service to type ExternalName. - \ If this field is not specified, it will be initialized - from the clusterIP field. If this field is specified, - clients must ensure that clusterIPs[0] and clusterIP have - the same value. \n This field may hold a maximum of two - entries (dual-stack IPs, in either order). These IPs must - correspond to the values of the ipFamilies field. Both - clusterIPs and ipFamilies are governed by the ipFamilyPolicy - field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" - items: - type: string - type: array - x-kubernetes-list-type: atomic - externalIPs: - description: externalIPs is a list of IP addresses for which - nodes in the cluster will also accept traffic for this - service. These IPs are not managed by Kubernetes. The - user is responsible for ensuring that traffic arrives - at a node with this IP. A common example is external - load-balancers that are not part of the Kubernetes system. - items: - type: string - type: array - externalName: - description: externalName is the external reference that - discovery mechanisms will return as an alias for this - service (e.g. a DNS CNAME record). No proxying will be - involved. Must be a lowercase RFC-1123 hostname (https://tools.ietf.org/html/rfc1123) - and requires `type` to be "ExternalName". - type: string - externalTrafficPolicy: - description: externalTrafficPolicy describes how nodes distribute - service traffic they receive on one of the Service's "externally-facing" - addresses (NodePorts, ExternalIPs, and LoadBalancer IPs). - If set to "Local", the proxy will configure the service - in a way that assumes that external load balancers will - take care of balancing the service traffic between nodes, - and so each node will deliver traffic only to the node-local - endpoints of the service, without masquerading the client - source IP. (Traffic mistakenly sent to a node with no - endpoints will be dropped.) The default value, "Cluster", - uses the standard behavior of routing to all endpoints - evenly (possibly modified by topology and other features). - Note that traffic sent to an External IP or LoadBalancer - IP from within the cluster will always get "Cluster" semantics, - but clients sending to a NodePort from within the cluster - may need to take traffic policy into account when picking - a node. - type: string - healthCheckNodePort: - description: healthCheckNodePort specifies the healthcheck - nodePort for the service. This only applies when type - is set to LoadBalancer and externalTrafficPolicy is set - to Local. If a value is specified, is in-range, and is - not in use, it will be used. If not specified, a value - will be automatically allocated. External systems (e.g. - load-balancers) can use this port to determine if a given - node holds endpoints for this service or not. If this - field is specified when creating a Service which does - not need it, creation will fail. This field will be wiped - when updating a Service to no longer need it (e.g. changing - type). This field cannot be updated once set. - format: int32 - type: integer - internalTrafficPolicy: - description: InternalTrafficPolicy describes how nodes distribute - service traffic they receive on the ClusterIP. If set - to "Local", the proxy will assume that pods only want - to talk to endpoints of the service on the same node as - the pod, dropping the traffic if there are no local endpoints. - The default value, "Cluster", uses the standard behavior - of routing to all endpoints evenly (possibly modified - by topology and other features). - type: string - ipFamilies: - description: "IPFamilies is a list of IP families (e.g. - IPv4, IPv6) assigned to this service. This field is usually - assigned automatically based on cluster configuration - and the ipFamilyPolicy field. If this field is specified - manually, the requested family is available in the cluster, - and ipFamilyPolicy allows it, it will be used; otherwise - creation of the service will fail. This field is conditionally - mutable: it allows for adding or removing a secondary - IP family, but it does not allow changing the primary - IP family of the Service. Valid values are \"IPv4\" and - \"IPv6\". This field only applies to Services of types - ClusterIP, NodePort, and LoadBalancer, and does apply - to \"headless\" services. This field will be wiped when - updating a Service to type ExternalName. \n This field - may hold a maximum of two entries (dual-stack families, - in either order). These families must correspond to the - values of the clusterIPs field, if specified. Both clusterIPs - and ipFamilies are governed by the ipFamilyPolicy field." - items: - description: IPFamily represents the IP Family (IPv4 or - IPv6). This type is used to express the family of an - IP expressed by a type (e.g. service.spec.ipFamilies). - type: string - type: array - x-kubernetes-list-type: atomic - ipFamilyPolicy: - description: IPFamilyPolicy represents the dual-stack-ness - requested or required by this Service. If there is no - value provided, then this field will be set to SingleStack. - Services can be "SingleStack" (a single IP family), "PreferDualStack" - (two IP families on dual-stack configured clusters or - a single IP family on single-stack clusters), or "RequireDualStack" - (two IP families on dual-stack configured clusters, otherwise - fail). The ipFamilies and clusterIPs fields depend on - the value of this field. This field will be wiped when - updating a service to type ExternalName. - type: string - loadBalancerClass: - description: loadBalancerClass is the class of the load - balancer implementation this Service belongs to. If specified, - the value of this field must be a label-style identifier, - with an optional prefix, e.g. "internal-vip" or "example.com/internal-vip". - Unprefixed names are reserved for end-users. This field - can only be set when the Service type is 'LoadBalancer'. - If not set, the default load balancer implementation is - used, today this is typically done through the cloud provider - integration, but should apply for any default implementation. - If set, it is assumed that a load balancer implementation - is watching for Services with a matching class. Any default - load balancer implementation (e.g. cloud providers) should - ignore Services that set this field. This field can only - be set when creating or updating a Service to type 'LoadBalancer'. - Once set, it can not be changed. This field will be wiped - when a service is updated to a non 'LoadBalancer' type. - type: string - loadBalancerIP: - description: 'Only applies to Service Type: LoadBalancer. - This feature depends on whether the underlying cloud-provider - supports specifying the loadBalancerIP when a load balancer - is created. This field will be ignored if the cloud-provider - does not support the feature. Deprecated: This field was - under-specified and its meaning varies across implementations, - and it cannot support dual-stack. As of Kubernetes v1.24, - users are encouraged to use implementation-specific annotations - when available. This field may be removed in a future - API version.' - type: string - loadBalancerSourceRanges: - description: 'If specified and supported by the platform, - this will restrict traffic through the cloud-provider - load-balancer will be restricted to the specified client - IPs. This field will be ignored if the cloud-provider - does not support the feature." More info: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/' - items: - type: string - type: array ports: description: 'The list of ports that are exposed by this service. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' items: - description: ServicePort contains information on service's - port. properties: appProtocol: description: The application protocol for this port. @@ -8566,23 +8363,8 @@ spec: This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoints for a Service, this must match the 'name' field - in the EndpointPort. Optional if only one ServicePort - is defined on this service. + in the EndpointPort. type: string - nodePort: - description: 'The port on each node on which this - service is exposed when type is NodePort or LoadBalancer. Usually - assigned by the system. If a value is specified, - in-range, and not in use it will be used, otherwise - the operation will fail. If not specified, a port - will be allocated if this Service requires one. If - this field is specified when creating a Service - which does not need it, creation will fail. This - field will be wiped when updating a Service to no - longer need it (e.g. changing type from NodePort - to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport' - format: int32 - type: integer port: description: The port that will be exposed by this service. @@ -8592,6 +8374,10 @@ spec: default: TCP description: The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". Default is TCP. + enum: + - TCP + - UDP + - SCTP type: string targetPort: anyOf: @@ -8616,75 +8402,7 @@ spec: - port - protocol x-kubernetes-list-type: map - publishNotReadyAddresses: - description: publishNotReadyAddresses indicates that any - agent which deals with endpoints for this Service should - disregard any indications of ready/not-ready. The primary - use case for setting this field is for a StatefulSet's - Headless Service to propagate SRV DNS records for its - Pods for the purpose of peer discovery. The Kubernetes - controllers that generate Endpoints and EndpointSlice - resources for Services interpret this to mean that all - endpoints are considered "ready" even if the Pods themselves - are not. Agents which consume only Kubernetes generated - endpoints through the Endpoints or EndpointSlice resources - can safely assume this behavior. - type: boolean - selector: - additionalProperties: - type: string - description: 'Route service traffic to pods with label keys - and values matching this selector. If empty or not present, - the service is assumed to have an external process managing - its endpoints, which Kubernetes will not modify. Only - applies to types ClusterIP, NodePort, and LoadBalancer. - Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/' - type: object - x-kubernetes-map-type: atomic - sessionAffinity: - description: 'Supports "ClientIP" and "None". Used to maintain - session affinity. Enable client IP based session affinity. - Must be ClientIP or None. Defaults to None. More info: - https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' - type: string - sessionAffinityConfig: - description: sessionAffinityConfig contains the configurations - of session affinity. - properties: - clientIP: - description: clientIP contains the configurations of - Client IP based session affinity. - properties: - timeoutSeconds: - description: timeoutSeconds specifies the seconds - of ClientIP type session sticky time. The value - must be >0 && <=86400(for 1 day) if ServiceAffinity - == "ClientIP". Default value is 10800(for 3 hours). - format: int32 - type: integer - type: object - type: object - type: - description: 'type determines how the Service is exposed. - Defaults to ClusterIP. Valid options are ExternalName, - ClusterIP, NodePort, and LoadBalancer. "ClusterIP" allocates - a cluster-internal IP address for load-balancing to endpoints. - Endpoints are determined by the selector or if that is - not specified, by manual construction of an Endpoints - object or EndpointSlice objects. If clusterIP is "None", - no virtual IP is allocated and the endpoints are published - as a set of endpoints rather than a virtual IP. "NodePort" - builds on ClusterIP and allocates a port on every node - which routes to the same endpoints as the clusterIP. "LoadBalancer" - builds on NodePort and creates an external load-balancer - (if supported in the current cloud) which routes to the - same endpoints as the clusterIP. "ExternalName" aliases - this service to the specified externalName. Several other - fields do not apply to ExternalName services. More info: - https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types' - type: string type: object - x-kubernetes-preserve-unknown-fields: true systemAccounts: description: Statement to create system account. properties: diff --git a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml index 23ae4a208..7f97de009 100644 --- a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml @@ -55,22 +55,18 @@ spec: accessMode: Readonly service: ports: - - protocol: TCP + - name: mysql port: 3306 targetPort: mysql - name: mysql - - protocol: TCP - port: 15100 + - name: vttabletport + port: 15100 targetPort: vttabletport - name: vttabletport - - protocol: TCP + - name: vttabletgrpc port: 16100 targetPort: vttabletgrpc - name: vttabletgrpc - - protocol: TCP + - name: delvedebug port: 40000 targetPort: delvedebug - name: delvedebug horizontalScalePolicy: type: Snapshot backupTemplateSelector: @@ -173,14 +169,11 @@ spec: imagePullPolicy: {{ default .Values.wesqlscale.image.pullPolicy "IfNotPresent" }} ports: - containerPort: 15100 - protocol: TCP - name: vttabletport + name: vttabletport - containerPort: 16100 - protocol: TCP name: vttabletgrpc - containerPort: 40000 - protocol: TCP - name: delvedebug + name: delvedebug env: - name: CELL value: {{ .Values.wesqlscale.cell | default "zone1" | quote }} @@ -340,17 +333,15 @@ spec: workloadType: Stateful service: ports: - - protocol: TCP + - name: client port: 2379 targetPort: client - name: client podSpec: containers: - name: etcd imagePullPolicy: IfNotPresent ports: - containerPort: 2379 - protocol: TCP name: client env: - name: CELL @@ -411,31 +402,25 @@ spec: workloadType: Stateful service: ports: - - protocol: TCP + - name: webport port: 15000 targetPort: webport - name: webport - - protocol: TCP + - name: grpcport port: 15999 targetPort: grpcport - name: grpcport - - protocol: TCP + - name: delvedebug port: 40000 targetPort: delvedebug - name: delvedebug podSpec: containers: - name: vtctld imagePullPolicy: IfNotPresent ports: - containerPort: 15000 - protocol: TCP name: webport - containerPort: 15999 - protocol: TCP name: grpcport - containerPort: 40000 - protocol: TCP name: delvedebug env: - name: CELL @@ -480,24 +465,20 @@ spec: workloadType: Stateful service: ports: - - protocol: TCP + - name: port port: 16000 targetPort: port - name: port - - protocol: TCP + - name: delvedebug port: 40000 targetPort: delvedebug - name: delvedebug podSpec: containers: - name: vtconsensus imagePullPolicy: IfNotPresent ports: - containerPort: 16000 - protocol: TCP name: port - containerPort: 40000 - protocol: TCP name: delvedebug env: - name: CELL @@ -548,39 +529,31 @@ spec: workloadType: Stateful service: ports: - - protocol: TCP + - name: webport port: 15001 targetPort: webport - name: webport - - protocol: TCP + - name: grpcport port: 15991 targetPort: grpcport - name: grpcport - - protocol: TCP + - name: serverport port: 15306 targetPort: serverport - name: serverport - - protocol: TCP + - name: delvedebug port: 40000 targetPort: delvedebug - name: delvedebug podSpec: containers: - name: vtgate imagePullPolicy: IfNotPresent ports: - containerPort: 15001 - protocol: TCP name: webport - containerPort: 15991 - protocol: TCP name: grpcport - containerPort: 15306 - protocol: TCP - name: serverport + name: serverport - containerPort: 40000 - protocol: TCP - name: delvedebug + name: delvedebug env: - name: MYSQL_ROOT_USER valueFrom: diff --git a/deploy/apecloud-mysql/templates/clusterdefinition.yaml b/deploy/apecloud-mysql/templates/clusterdefinition.yaml index 133005d17..f345d4b1f 100644 --- a/deploy/apecloud-mysql/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql/templates/clusterdefinition.yaml @@ -51,7 +51,7 @@ spec: accessMode: Readonly service: ports: - - protocol: TCP + - name: mysql port: 3306 targetPort: mysql horizontalScalePolicy: diff --git a/deploy/clickhouse/templates/clusterdefinition.yaml b/deploy/clickhouse/templates/clusterdefinition.yaml index 7d8a7c977..1cae0bf96 100644 --- a/deploy/clickhouse/templates/clusterdefinition.yaml +++ b/deploy/clickhouse/templates/clusterdefinition.yaml @@ -111,22 +111,16 @@ spec: ports: - name: http containerPort: 8123 - # protocol: TCP - name: tcp containerPort: 9000 - # protocol: TCP - name: tcp-postgresql containerPort: 9005 - # protocol: TCP - name: tcp-mysql containerPort: 9004 - # protocol: TCP - name: http-intersrv containerPort: 9009 - # protocol: TCP - name: http-metrics containerPort: 8001 - # protocol: TCP livenessProbe: failureThreshold: 3 initialDelaySeconds: 10 @@ -175,13 +169,9 @@ spec: - name: tcp targetPort: tcp port: 2181 - protocol: TCP - nodePort: null - name: http-metrics targetPort: http-metrics port: 8001 - protocol: TCP - nodePort: null podSpec: securityContext: fsGroup: 1001 @@ -213,13 +203,10 @@ spec: ports: - name: tcp containerPort: 2181 - # protocol: TCP - name: raft containerPort: 9444 - # protocol: TCP - name: http-metrics containerPort: 8001 - # protocol: TCP # livenessProbe: # failureThreshold: 6 # initialDelaySeconds: 30 @@ -360,16 +347,12 @@ spec: ports: - name: client containerPort: 2181 - # protocol: TCP - name: follower containerPort: 2888 - # protocol: TCP - name: election containerPort: 3888 - # protocol: TCP - name: metrics containerPort: 9141 - # protocol: TCP livenessProbe: failureThreshold: 6 initialDelaySeconds: 30 diff --git a/deploy/etcd/templates/clusterdefinition.yaml b/deploy/etcd/templates/clusterdefinition.yaml index 8c4657927..ce92152d5 100644 --- a/deploy/etcd/templates/clusterdefinition.yaml +++ b/deploy/etcd/templates/clusterdefinition.yaml @@ -23,18 +23,17 @@ spec: failureThreshold: 3 service: ports: - - protocol: TCP + - name: client port: 2379 + targetPort: client podSpec: containers: - name: etcd imagePullPolicy: IfNotPresent ports: - containerPort: 2379 - protocol: TCP name: client - containerPort: 2380 - protocol: TCP name: peer volumeMounts: - name: data diff --git a/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml b/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml index b9595c56e..599bccf91 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml @@ -8344,214 +8344,11 @@ spec: service: description: service defines the behavior of a service spec. provide read-write service when WorkloadType is Consensus. - https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status properties: - allocateLoadBalancerNodePorts: - description: allocateLoadBalancerNodePorts defines if NodePorts - will be automatically allocated for services with type - LoadBalancer. Default is "true". It may be set to "false" - if the cluster load-balancer does not rely on NodePorts. If - the caller requests specific NodePorts (by specifying - a value), those requests will be respected, regardless - of this field. This field may only be set for services - with type LoadBalancer and will be cleared if the type - is changed to any other type. - type: boolean - clusterIP: - description: 'clusterIP is the IP address of the service - and is usually assigned randomly. If an address is specified - manually, is in-range (as per system configuration), and - is not in use, it will be allocated to the service; otherwise - creation of the service will fail. This field may not - be changed through updates unless the type field is also - being changed to ExternalName (which requires this field - to be blank) or the type field is being changed from ExternalName - (in which case this field may optionally be specified, - as describe above). Valid values are "None", empty string - (""), or a valid IP address. Setting this to "None" makes - a "headless service" (no virtual IP), which is useful - when direct endpoint connections are preferred and proxying - is not required. Only applies to types ClusterIP, NodePort, - and LoadBalancer. If this field is specified when creating - a Service of type ExternalName, creation will fail. This - field will be wiped when updating a Service to type ExternalName. - More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' - type: string - clusterIPs: - description: "ClusterIPs is a list of IP addresses assigned - to this service, and are usually assigned randomly. If - an address is specified manually, is in-range (as per - system configuration), and is not in use, it will be allocated - to the service; otherwise creation of the service will - fail. This field may not be changed through updates unless - the type field is also being changed to ExternalName (which - requires this field to be empty) or the type field is - being changed from ExternalName (in which case this field - may optionally be specified, as describe above). Valid - values are \"None\", empty string (\"\"), or a valid IP - address. Setting this to \"None\" makes a \"headless - service\" (no virtual IP), which is useful when direct - endpoint connections are preferred and proxying is not - required. Only applies to types ClusterIP, NodePort, - and LoadBalancer. If this field is specified when creating - a Service of type ExternalName, creation will fail. This - field will be wiped when updating a Service to type ExternalName. - \ If this field is not specified, it will be initialized - from the clusterIP field. If this field is specified, - clients must ensure that clusterIPs[0] and clusterIP have - the same value. \n This field may hold a maximum of two - entries (dual-stack IPs, in either order). These IPs must - correspond to the values of the ipFamilies field. Both - clusterIPs and ipFamilies are governed by the ipFamilyPolicy - field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies" - items: - type: string - type: array - x-kubernetes-list-type: atomic - externalIPs: - description: externalIPs is a list of IP addresses for which - nodes in the cluster will also accept traffic for this - service. These IPs are not managed by Kubernetes. The - user is responsible for ensuring that traffic arrives - at a node with this IP. A common example is external - load-balancers that are not part of the Kubernetes system. - items: - type: string - type: array - externalName: - description: externalName is the external reference that - discovery mechanisms will return as an alias for this - service (e.g. a DNS CNAME record). No proxying will be - involved. Must be a lowercase RFC-1123 hostname (https://tools.ietf.org/html/rfc1123) - and requires `type` to be "ExternalName". - type: string - externalTrafficPolicy: - description: externalTrafficPolicy describes how nodes distribute - service traffic they receive on one of the Service's "externally-facing" - addresses (NodePorts, ExternalIPs, and LoadBalancer IPs). - If set to "Local", the proxy will configure the service - in a way that assumes that external load balancers will - take care of balancing the service traffic between nodes, - and so each node will deliver traffic only to the node-local - endpoints of the service, without masquerading the client - source IP. (Traffic mistakenly sent to a node with no - endpoints will be dropped.) The default value, "Cluster", - uses the standard behavior of routing to all endpoints - evenly (possibly modified by topology and other features). - Note that traffic sent to an External IP or LoadBalancer - IP from within the cluster will always get "Cluster" semantics, - but clients sending to a NodePort from within the cluster - may need to take traffic policy into account when picking - a node. - type: string - healthCheckNodePort: - description: healthCheckNodePort specifies the healthcheck - nodePort for the service. This only applies when type - is set to LoadBalancer and externalTrafficPolicy is set - to Local. If a value is specified, is in-range, and is - not in use, it will be used. If not specified, a value - will be automatically allocated. External systems (e.g. - load-balancers) can use this port to determine if a given - node holds endpoints for this service or not. If this - field is specified when creating a Service which does - not need it, creation will fail. This field will be wiped - when updating a Service to no longer need it (e.g. changing - type). This field cannot be updated once set. - format: int32 - type: integer - internalTrafficPolicy: - description: InternalTrafficPolicy describes how nodes distribute - service traffic they receive on the ClusterIP. If set - to "Local", the proxy will assume that pods only want - to talk to endpoints of the service on the same node as - the pod, dropping the traffic if there are no local endpoints. - The default value, "Cluster", uses the standard behavior - of routing to all endpoints evenly (possibly modified - by topology and other features). - type: string - ipFamilies: - description: "IPFamilies is a list of IP families (e.g. - IPv4, IPv6) assigned to this service. This field is usually - assigned automatically based on cluster configuration - and the ipFamilyPolicy field. If this field is specified - manually, the requested family is available in the cluster, - and ipFamilyPolicy allows it, it will be used; otherwise - creation of the service will fail. This field is conditionally - mutable: it allows for adding or removing a secondary - IP family, but it does not allow changing the primary - IP family of the Service. Valid values are \"IPv4\" and - \"IPv6\". This field only applies to Services of types - ClusterIP, NodePort, and LoadBalancer, and does apply - to \"headless\" services. This field will be wiped when - updating a Service to type ExternalName. \n This field - may hold a maximum of two entries (dual-stack families, - in either order). These families must correspond to the - values of the clusterIPs field, if specified. Both clusterIPs - and ipFamilies are governed by the ipFamilyPolicy field." - items: - description: IPFamily represents the IP Family (IPv4 or - IPv6). This type is used to express the family of an - IP expressed by a type (e.g. service.spec.ipFamilies). - type: string - type: array - x-kubernetes-list-type: atomic - ipFamilyPolicy: - description: IPFamilyPolicy represents the dual-stack-ness - requested or required by this Service. If there is no - value provided, then this field will be set to SingleStack. - Services can be "SingleStack" (a single IP family), "PreferDualStack" - (two IP families on dual-stack configured clusters or - a single IP family on single-stack clusters), or "RequireDualStack" - (two IP families on dual-stack configured clusters, otherwise - fail). The ipFamilies and clusterIPs fields depend on - the value of this field. This field will be wiped when - updating a service to type ExternalName. - type: string - loadBalancerClass: - description: loadBalancerClass is the class of the load - balancer implementation this Service belongs to. If specified, - the value of this field must be a label-style identifier, - with an optional prefix, e.g. "internal-vip" or "example.com/internal-vip". - Unprefixed names are reserved for end-users. This field - can only be set when the Service type is 'LoadBalancer'. - If not set, the default load balancer implementation is - used, today this is typically done through the cloud provider - integration, but should apply for any default implementation. - If set, it is assumed that a load balancer implementation - is watching for Services with a matching class. Any default - load balancer implementation (e.g. cloud providers) should - ignore Services that set this field. This field can only - be set when creating or updating a Service to type 'LoadBalancer'. - Once set, it can not be changed. This field will be wiped - when a service is updated to a non 'LoadBalancer' type. - type: string - loadBalancerIP: - description: 'Only applies to Service Type: LoadBalancer. - This feature depends on whether the underlying cloud-provider - supports specifying the loadBalancerIP when a load balancer - is created. This field will be ignored if the cloud-provider - does not support the feature. Deprecated: This field was - under-specified and its meaning varies across implementations, - and it cannot support dual-stack. As of Kubernetes v1.24, - users are encouraged to use implementation-specific annotations - when available. This field may be removed in a future - API version.' - type: string - loadBalancerSourceRanges: - description: 'If specified and supported by the platform, - this will restrict traffic through the cloud-provider - load-balancer will be restricted to the specified client - IPs. This field will be ignored if the cloud-provider - does not support the feature." More info: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/' - items: - type: string - type: array ports: description: 'The list of ports that are exposed by this service. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' items: - description: ServicePort contains information on service's - port. properties: appProtocol: description: The application protocol for this port. @@ -8566,23 +8363,8 @@ spec: This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoints for a Service, this must match the 'name' field - in the EndpointPort. Optional if only one ServicePort - is defined on this service. + in the EndpointPort. type: string - nodePort: - description: 'The port on each node on which this - service is exposed when type is NodePort or LoadBalancer. Usually - assigned by the system. If a value is specified, - in-range, and not in use it will be used, otherwise - the operation will fail. If not specified, a port - will be allocated if this Service requires one. If - this field is specified when creating a Service - which does not need it, creation will fail. This - field will be wiped when updating a Service to no - longer need it (e.g. changing type from NodePort - to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport' - format: int32 - type: integer port: description: The port that will be exposed by this service. @@ -8592,6 +8374,10 @@ spec: default: TCP description: The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". Default is TCP. + enum: + - TCP + - UDP + - SCTP type: string targetPort: anyOf: @@ -8616,75 +8402,7 @@ spec: - port - protocol x-kubernetes-list-type: map - publishNotReadyAddresses: - description: publishNotReadyAddresses indicates that any - agent which deals with endpoints for this Service should - disregard any indications of ready/not-ready. The primary - use case for setting this field is for a StatefulSet's - Headless Service to propagate SRV DNS records for its - Pods for the purpose of peer discovery. The Kubernetes - controllers that generate Endpoints and EndpointSlice - resources for Services interpret this to mean that all - endpoints are considered "ready" even if the Pods themselves - are not. Agents which consume only Kubernetes generated - endpoints through the Endpoints or EndpointSlice resources - can safely assume this behavior. - type: boolean - selector: - additionalProperties: - type: string - description: 'Route service traffic to pods with label keys - and values matching this selector. If empty or not present, - the service is assumed to have an external process managing - its endpoints, which Kubernetes will not modify. Only - applies to types ClusterIP, NodePort, and LoadBalancer. - Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/' - type: object - x-kubernetes-map-type: atomic - sessionAffinity: - description: 'Supports "ClientIP" and "None". Used to maintain - session affinity. Enable client IP based session affinity. - Must be ClientIP or None. Defaults to None. More info: - https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies' - type: string - sessionAffinityConfig: - description: sessionAffinityConfig contains the configurations - of session affinity. - properties: - clientIP: - description: clientIP contains the configurations of - Client IP based session affinity. - properties: - timeoutSeconds: - description: timeoutSeconds specifies the seconds - of ClientIP type session sticky time. The value - must be >0 && <=86400(for 1 day) if ServiceAffinity - == "ClientIP". Default value is 10800(for 3 hours). - format: int32 - type: integer - type: object - type: object - type: - description: 'type determines how the Service is exposed. - Defaults to ClusterIP. Valid options are ExternalName, - ClusterIP, NodePort, and LoadBalancer. "ClusterIP" allocates - a cluster-internal IP address for load-balancing to endpoints. - Endpoints are determined by the selector or if that is - not specified, by manual construction of an Endpoints - object or EndpointSlice objects. If clusterIP is "None", - no virtual IP is allocated and the endpoints are published - as a set of endpoints rather than a virtual IP. "NodePort" - builds on ClusterIP and allocates a port on every node - which routes to the same endpoints as the clusterIP. "LoadBalancer" - builds on NodePort and creates an external load-balancer - (if supported in the current cloud) which routes to the - same endpoints as the clusterIP. "ExternalName" aliases - this service to the specified externalName. Several other - fields do not apply to ExternalName services. More info: - https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types' - type: string type: object - x-kubernetes-preserve-unknown-fields: true systemAccounts: description: Statement to create system account. properties: diff --git a/deploy/kafka/templates/clusterdefinition.yaml b/deploy/kafka/templates/clusterdefinition.yaml index 674e059dc..042795763 100644 --- a/deploy/kafka/templates/clusterdefinition.yaml +++ b/deploy/kafka/templates/clusterdefinition.yaml @@ -56,20 +56,15 @@ spec: - name: kafka-ctrlr targetPort: kafka-ctrlr port: 9093 - nodePort: null - name: kafka-client targetPort: kafka-client port: 9092 - nodePort: null - name: metrics targetPort: metrics port: 5556 - nodePort: null - podSpec: securityContext: fsGroup: 1001 - containers: - name: kafka image: docker.io/bitnami/kafka:3.4.0-debian-11-r8 @@ -245,12 +240,9 @@ spec: - name: kafka-ctrlr targetPort: kafka-ctrlr port: 9093 - nodePort: null - name: metrics targetPort: metrics port: 5556 - nodePort: null - podSpec: securityContext: fsGroup: 1001 @@ -413,12 +405,9 @@ spec: - name: kafka-client targetPort: kafka-client port: 9092 - nodePort: null - name: metrics targetPort: metrics port: 5556 - nodePort: null - podSpec: securityContext: fsGroup: 1001 diff --git a/deploy/milvus/templates/clusterdefinition.yaml b/deploy/milvus/templates/clusterdefinition.yaml index db5a18da3..0bc3e9753 100644 --- a/deploy/milvus/templates/clusterdefinition.yaml +++ b/deploy/milvus/templates/clusterdefinition.yaml @@ -33,7 +33,6 @@ spec: service: ports: - name: tcp-milvus - protocol: TCP port: 19530 targetPort: tcp-milvus volumeTypes: @@ -173,10 +172,8 @@ spec: ports: - name: client containerPort: 2379 - protocol: TCP - name: peer containerPort: 2380 - protocol: TCP env: - name: BITNAMI_DEBUG value: "false" @@ -235,7 +232,6 @@ spec: service: ports: - name: http - protocol: TCP port: 9000 targetPort: 9000 volumeTypes: diff --git a/deploy/mongodb/templates/clusterdefinition.yaml b/deploy/mongodb/templates/clusterdefinition.yaml index be750c359..8a84e3415 100644 --- a/deploy/mongodb/templates/clusterdefinition.yaml +++ b/deploy/mongodb/templates/clusterdefinition.yaml @@ -5,6 +5,9 @@ metadata: labels: {{- include "mongodb.labels" . | nindent 4 }} spec: + connectionCredential: + username: admin + password: "" componentDefs: - name: mongos scriptSpecs: @@ -16,14 +19,14 @@ spec: workloadType: Stateless service: ports: - - protocol: TCP + - name: mongos port: 27017 + targetPort: mongos podSpec: containers: - name: mongos ports: - - protocol: TCP - name: mongos + - name: mongos containerPort: 27017 command: - /scripts/mongos-setup.sh @@ -54,14 +57,14 @@ spec: failureThreshold: 3 service: ports: - - protocol: TCP + - name: configsvr port: 27018 + targetPort: configsvr podSpec: containers: - name: configsvr ports: - name: configsvr - protocol: TCP containerPort: 27018 command: - /scripts/replicaset-setup.sh @@ -103,14 +106,14 @@ spec: failureThreshold: 3 service: ports: - - protocol: TCP + - name: shard port: 27018 + targetPort: shard podSpec: containers: - name: shard ports: - name: shard - protocol: TCP containerPort: 27018 command: - /scripts/replicaset-setup.sh @@ -136,6 +139,3 @@ spec: - name: scripts mountPath: /scripts/shard-agent.sh subPath: shard-agent.sh - connectionCredential: - username: admin - password: "" \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml index 288cac1fa..62841e385 100644 --- a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml +++ b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml @@ -63,7 +63,6 @@ spec: service: ports: - name: tcp-postgresql - protocol: TCP port: 5432 targetPort: tcp-postgresql - name: http-metrics-postgresql diff --git a/deploy/postgresql/templates/clusterdefinition.yaml b/deploy/postgresql/templates/clusterdefinition.yaml index 2d37e3035..aa0cd001d 100644 --- a/deploy/postgresql/templates/clusterdefinition.yaml +++ b/deploy/postgresql/templates/clusterdefinition.yaml @@ -53,7 +53,6 @@ spec: service: ports: - name: tcp-postgresql - protocol: TCP port: 5432 targetPort: tcp-postgresql - name: http-metrics-postgresql diff --git a/deploy/qdrant/templates/clusterdefinition.yaml b/deploy/qdrant/templates/clusterdefinition.yaml index e55b569d9..169468f60 100644 --- a/deploy/qdrant/templates/clusterdefinition.yaml +++ b/deploy/qdrant/templates/clusterdefinition.yaml @@ -31,11 +31,9 @@ spec: service: ports: - name: tcp-qdrant - protocol: TCP port: 6333 targetPort: tcp-qdrant - name: grpc-qdrant - protocol: TCP port: 6334 targetPort: grpc-qdrant volumeTypes: diff --git a/deploy/redis/templates/clusterdefinition.yaml b/deploy/redis/templates/clusterdefinition.yaml index 5bd4f80d1..7ba9b81b7 100644 --- a/deploy/redis/templates/clusterdefinition.yaml +++ b/deploy/redis/templates/clusterdefinition.yaml @@ -57,13 +57,10 @@ spec: service: ports: - name: redis - protocol: TCP port: 6379 targetPort: redis - name: metrics targetPort: metrics - port: 9121 - nodePort: null configSpecs: - name: redis-replication-config templateRef: redis7-config-template @@ -119,7 +116,6 @@ spec: ports: - name: metrics containerPort: 9121 - protocol: TCP livenessProbe: httpGet: path: / @@ -181,7 +177,6 @@ spec: service: ports: - name: redis-sentinel - protocol: TCP targetPort: redis-sentinel port: 26379 configSpecs: @@ -219,7 +214,6 @@ spec: ports: - containerPort: 26379 name: redis-sentinel - protocol: TCP volumeMounts: - name: data mountPath: /data diff --git a/deploy/weaviate/templates/clusterdefinition.yaml b/deploy/weaviate/templates/clusterdefinition.yaml index 34abdb070..5bbf5ab95 100644 --- a/deploy/weaviate/templates/clusterdefinition.yaml +++ b/deploy/weaviate/templates/clusterdefinition.yaml @@ -31,7 +31,6 @@ spec: service: ports: - name: tcp-weaviate - protocol: TCP port: 8080 targetPort: tcp-weaviate volumeTypes: diff --git a/internal/controller/component/component.go b/internal/controller/component/component.go index 8916fe2ea..550704263 100644 --- a/internal/controller/component/component.go +++ b/internal/controller/component/component.go @@ -106,7 +106,7 @@ func BuildComponent( } if clusterCompDefObj.Service != nil { - service := corev1.Service{Spec: *clusterCompDefObj.Service} + service := corev1.Service{Spec: clusterCompDefObj.Service.ToSVCSpec()} service.Spec.Type = corev1.ServiceTypeClusterIP component.Services = append(component.Services, service) @@ -116,7 +116,7 @@ func BuildComponent( Name: item.Name, Annotations: item.Annotations, }, - Spec: *clusterCompDefObj.Service, + Spec: service.Spec, } service.Spec.Type = item.ServiceType component.Services = append(component.Services, service) diff --git a/internal/testutil/apps/clusterdef_factory.go b/internal/testutil/apps/clusterdef_factory.go index 5973d004f..7c8add98f 100644 --- a/internal/testutil/apps/clusterdef_factory.go +++ b/internal/testutil/apps/clusterdef_factory.go @@ -77,8 +77,8 @@ func (factory *MockClusterDefFactory) AddServicePort(port int32) *MockClusterDef if comp == nil { return nil } - comp.Service = &corev1.ServiceSpec{ - Ports: []corev1.ServicePort{{ + comp.Service = &appsv1alpha1.ServiceSpec{ + Ports: []appsv1alpha1.ServicePort{{ Protocol: corev1.ProtocolTCP, Port: port, }}, @@ -160,7 +160,7 @@ func (factory *MockClusterDefFactory) AddHorizontalScalePolicy(policy appsv1alph } func (factory *MockClusterDefFactory) SetConnectionCredential( - connectionCredential map[string]string, svc *corev1.ServiceSpec) *MockClusterDefFactory { + connectionCredential map[string]string, svc *appsv1alpha1.ServiceSpec) *MockClusterDefFactory { factory.get().Spec.ConnectionCredential = connectionCredential factory.SetServiceSpec(svc) return factory @@ -182,7 +182,7 @@ func (factory *MockClusterDefFactory) getLastCompDef() *appsv1alpha1.ClusterComp return &comps[l-1] } -func (factory *MockClusterDefFactory) SetServiceSpec(svc *corev1.ServiceSpec) *MockClusterDefFactory { +func (factory *MockClusterDefFactory) SetServiceSpec(svc *appsv1alpha1.ServiceSpec) *MockClusterDefFactory { comp := factory.get1stCompDef() if comp == nil { return factory diff --git a/internal/testutil/apps/constant.go b/internal/testutil/apps/constant.go index 94b7268c9..40c805f62 100644 --- a/internal/testutil/apps/constant.go +++ b/internal/testutil/apps/constant.go @@ -64,8 +64,8 @@ var ( Name: DefaultNginxContainerName, }}, }, - Service: &corev1.ServiceSpec{ - Ports: []corev1.ServicePort{{ + Service: &appsv1alpha1.ServiceSpec{ + Ports: []appsv1alpha1.ServicePort{{ Protocol: corev1.ProtocolTCP, Port: 80, }}, @@ -86,8 +86,8 @@ var ( // defaultSvc value are corresponding to defaultMySQLContainer.Ports name mapping and // corresponding to defaultConnectionCredential variable placeholder - defaultSvcSpec = corev1.ServiceSpec{ - Ports: []corev1.ServicePort{ + defaultSvcSpec = appsv1alpha1.ServiceSpec{ + Ports: []appsv1alpha1.ServicePort{ { Name: "mysql", TargetPort: intstr.IntOrString{ @@ -175,8 +175,8 @@ var ( UpdateStrategy: appsv1alpha1.BestEffortParallelStrategy, } - defaultMySQLService = corev1.ServiceSpec{ - Ports: []corev1.ServicePort{{ + defaultMySQLService = appsv1alpha1.ServiceSpec{ + Ports: []appsv1alpha1.ServicePort{{ Protocol: corev1.ProtocolTCP, Port: 3306, }}, @@ -199,8 +199,8 @@ var ( }, } - defaultRedisService = corev1.ServiceSpec{ - Ports: []corev1.ServicePort{{ + defaultRedisService = appsv1alpha1.ServiceSpec{ + Ports: []appsv1alpha1.ServicePort{{ Protocol: corev1.ProtocolTCP, Port: 6379, }}, From 98ddbcace9acf9d8aedf8fa98ab1577ae01c52fa Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Tue, 4 Apr 2023 16:51:07 +0800 Subject: [PATCH 43/80] feat: highly available Postgresql using our own image that supports pgvector (#2406) --- deploy/postgresql-patroni-ha/Chart.yaml | 2 +- deploy/postgresql-patroni-ha/values.yaml | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deploy/postgresql-patroni-ha/Chart.yaml b/deploy/postgresql-patroni-ha/Chart.yaml index 40bb628be..1cf6269e0 100644 --- a/deploy/postgresql-patroni-ha/Chart.yaml +++ b/deploy/postgresql-patroni-ha/Chart.yaml @@ -6,7 +6,7 @@ type: application version: 0.5.0-alpha.3 -appVersion: "14.5.0" +appVersion: "15.2.0" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/postgresql-patroni-ha/values.yaml b/deploy/postgresql-patroni-ha/values.yaml index a8614e42b..01afa1177 100644 --- a/deploy/postgresql-patroni-ha/values.yaml +++ b/deploy/postgresql-patroni-ha/values.yaml @@ -12,9 +12,9 @@ ## @param image.debug Specify if debug values should be set ## image: - ## repository: https://hub.docker.com/r/lrengineering/spilo-postgres - repository: lrengineering/spilo-postgres - tag: 14-2.1-p6 + registry: registry.cn-hangzhou.aliyuncs.com + repository: apecloud/spilo + tag: 15.2.0 digest: "" ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' @@ -67,7 +67,7 @@ audit: ## postgresqlSharedPreloadLibraries: "pg_stat_statements, auto_explain" ## Start PostgreSQL pod(s) without limitations on shm memory. -## By default docker and containerd (and possibly other container runtimes) limit `/dev/shm` to `64M` +## By default, docker and containerd (and possibly other container runtimes) limit `/dev/shm` to `64M` ## shmVolume: ## @param shmVolume.enabled Enable emptyDir volume for /dev/shm for PostgreSQL pod(s) From 759344c8899aa7c99b41bfc1c5d5d158dbd0aa49 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Tue, 4 Apr 2023 16:58:54 +0800 Subject: [PATCH 44/80] fix: turning on the tls feature will cause user configuration changes to be rolled back (#2394) --- .../apps/configuration/config_annotation.go | 15 ++++- controllers/apps/configuration/config_util.go | 12 ---- .../reconfigurerequest_controller.go | 7 ++- .../reconfigurerequest_controller_test.go | 44 +++++++++++-- .../apps/operations/reconfigure_util.go | 1 + internal/configuration/constrant.go | 28 +++++++++ internal/configuration/reconfigure_util.go | 35 +++++++++++ internal/constant/const.go | 8 ++- internal/controller/plan/template_wrapper.go | 62 ++++++++++++++++--- internal/controller/types/task.go | 16 +++++ internal/generics/type.go | 13 ++++ 11 files changed, 212 insertions(+), 29 deletions(-) create mode 100644 internal/configuration/constrant.go diff --git a/controllers/apps/configuration/config_annotation.go b/controllers/apps/configuration/config_annotation.go index 09be6ee55..badcbaa9c 100644 --- a/controllers/apps/configuration/config_annotation.go +++ b/controllers/apps/configuration/config_annotation.go @@ -71,14 +71,14 @@ func checkAndApplyConfigsChanged(client client.Client, ctx intctrlutil.RequestCt lastConfig, ok := annotations[constant.LastAppliedConfigAnnotation] if !ok { - return updateAppliedConfigs(client, ctx, cm, configData, ReconfigureFirstConfigType) + return updateAppliedConfigs(client, ctx, cm, configData, cfgcore.ReconfigureCreatedPhase) } return lastConfig == string(configData), nil } // updateAppliedConfigs update hash label and last applied config -func updateAppliedConfigs(cli client.Client, ctx intctrlutil.RequestCtx, config *corev1.ConfigMap, configData []byte, reconfigureType string) (bool, error) { +func updateAppliedConfigs(cli client.Client, ctx intctrlutil.RequestCtx, config *corev1.ConfigMap, configData []byte, reconfigurePhase string) (bool, error) { patch := client.MergeFrom(config.DeepCopy()) if config.ObjectMeta.Annotations == nil { @@ -91,10 +91,19 @@ func updateAppliedConfigs(cli client.Client, ctx intctrlutil.RequestCtx, config return false, err } config.ObjectMeta.Labels[constant.CMInsConfigurationHashLabelKey] = hash - config.ObjectMeta.Labels[constant.CMInsLastReconfigureMethodLabelKey] = reconfigureType + + newReconfigurePhase := config.ObjectMeta.Labels[constant.CMInsLastReconfigurePhaseKey] + if newReconfigurePhase == "" { + newReconfigurePhase = cfgcore.ReconfigureCreatedPhase + } + if cfgcore.ReconfigureNoChangeType != reconfigurePhase && !cfgcore.IsParametersUpdateFromManager(config) { + newReconfigurePhase = reconfigurePhase + } + config.ObjectMeta.Labels[constant.CMInsLastReconfigurePhaseKey] = newReconfigurePhase // delete reconfigure-policy delete(config.ObjectMeta.Annotations, constant.UpgradePolicyAnnotationKey) + delete(config.ObjectMeta.Annotations, constant.KBParameterUpdateSourceAnnotationKey) if err := cli.Patch(ctx.Ctx, config, patch); err != nil { return false, err } diff --git a/controllers/apps/configuration/config_util.go b/controllers/apps/configuration/config_util.go index c0dcc0710..6d64ae597 100644 --- a/controllers/apps/configuration/config_util.go +++ b/controllers/apps/configuration/config_util.go @@ -20,7 +20,6 @@ import ( "context" "fmt" "reflect" - "time" "github.com/StudioSol/set" "github.com/go-logr/logr" @@ -39,17 +38,6 @@ import ( "github.com/apecloud/kubeblocks/internal/generics" ) -const ( - ConfigReconcileInterval = time.Second * 1 - - ReconfigureFirstConfigType = "created" - ReconfigureNoChangeType = "noChange" - ReconfigureAutoReloadType = string(appsv1alpha1.AutoReload) - ReconfigureSimpleType = string(appsv1alpha1.NormalPolicy) - ReconfigureParallelType = string(appsv1alpha1.RestartPolicy) - ReconfigureRollingType = string(appsv1alpha1.RollingPolicy) -) - type ValidateConfigMap func(configTpl, ns string) (*corev1.ConfigMap, error) type ValidateConfigSchema func(tpl *appsv1alpha1.CustomParametersValidation) (bool, error) diff --git a/controllers/apps/configuration/reconfigurerequest_controller.go b/controllers/apps/configuration/reconfigurerequest_controller.go index 8ac52d49f..75740c037 100644 --- a/controllers/apps/configuration/reconfigurerequest_controller.go +++ b/controllers/apps/configuration/reconfigurerequest_controller.go @@ -21,6 +21,7 @@ import ( "encoding/json" "fmt" "strings" + "time" appv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" @@ -46,6 +47,10 @@ type ReconfigureRequestReconciler struct { Recorder record.EventRecorder } +const ( + ConfigReconcileInterval = time.Second * 1 +) + var ConfigurationRequiredLabels = []string{ constant.AppNameLabelKey, constant.AppInstanceLabelKey, @@ -160,7 +165,7 @@ func (r *ReconfigureRequestReconciler) sync(reqCtx intctrlutil.RequestCtx, confi // Not any parameters updated if !configPatch.IsModify { - return r.updateConfigCMStatus(reqCtx, config, ReconfigureNoChangeType) + return r.updateConfigCMStatus(reqCtx, config, cfgcore.ReconfigureNoChangeType) } reqCtx.Log.V(1).Info(fmt.Sprintf("reconfigure params: \n\tadd: %s\n\tdelete: %s\n\tupdate: %s", diff --git a/controllers/apps/configuration/reconfigurerequest_controller_test.go b/controllers/apps/configuration/reconfigurerequest_controller_test.go index 9e1288b12..e48bb3f8d 100644 --- a/controllers/apps/configuration/reconfigurerequest_controller_test.go +++ b/controllers/apps/configuration/reconfigurerequest_controller_test.go @@ -143,15 +143,46 @@ var _ = Describe("Reconfigure Controller", func() { g.Expect(cm.Labels[constant.AppInstanceLabelKey]).To(Equal(clusterObj.Name)) g.Expect(cm.Labels[constant.CMConfigurationTemplateNameLabelKey]).To(Equal(configSpecName)) g.Expect(cm.Labels[constant.CMConfigurationTypeLabelKey]).NotTo(Equal("")) - g.Expect(cm.Labels[constant.CMInsLastReconfigureMethodLabelKey]).To(Equal(ReconfigureFirstConfigType)) + g.Expect(cm.Labels[constant.CMInsLastReconfigurePhaseKey]).To(Equal(cfgcore.ReconfigureCreatedPhase)) configHash = cm.Labels[constant.CMInsConfigurationHashLabelKey] g.Expect(configHash).NotTo(Equal("")) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).To(BeTrue()) + // g.Expect(cm.Annotations[constant.KBParameterUpdateSourceAnnotationKey]).To(Equal(constant.ReconfigureManagerSource)) + }).Should(Succeed()) + + By("manager changes will not change the phase of configmap.") + Eventually(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(configmap), func(cm *corev1.ConfigMap) { + cm.Data["new_data"] = "###" + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureManagerSource) + })).Should(Succeed()) + + Eventually(func(g Gomega) { + cm := &corev1.ConfigMap{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(configmap), cm)).Should(Succeed()) + newHash := cm.Labels[constant.CMInsConfigurationHashLabelKey] + g.Expect(newHash).NotTo(Equal(configHash)) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).To(BeTrue()) + }).Should(Succeed()) + + By("recover normal update parameters") + Eventually(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(configmap), func(cm *corev1.ConfigMap) { + delete(cm.Data, "new_data") + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureManagerSource) + })).Should(Succeed()) + + Eventually(func(g Gomega) { + cm := &corev1.ConfigMap{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(configmap), cm)).Should(Succeed()) + newHash := cm.Labels[constant.CMInsConfigurationHashLabelKey] + g.Expect(newHash).To(Equal(configHash)) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).To(BeTrue()) }).Should(Succeed()) By("Update config, old version: " + configHash) updatedCM := testapps.NewCustomizedObj("resources/mysql-ins-config-update.yaml", &corev1.ConfigMap{}) Eventually(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(configmap), func(cm *corev1.ConfigMap) { cm.Data = updatedCM.Data + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureUserSource) })).Should(Succeed()) By("check config new version") @@ -160,33 +191,38 @@ var _ = Describe("Reconfigure Controller", func() { g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(configmap), cm)).Should(Succeed()) newHash := cm.Labels[constant.CMInsConfigurationHashLabelKey] g.Expect(newHash).NotTo(Equal(configHash)) - g.Expect(cm.Labels[constant.CMInsLastReconfigureMethodLabelKey]).To(Equal(ReconfigureAutoReloadType)) + g.Expect(cm.Labels[constant.CMInsLastReconfigurePhaseKey]).To(Equal(cfgcore.ReconfigureAutoReloadPhase)) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).NotTo(BeTrue()) }).Should(Succeed()) By("invalid Update") invalidUpdatedCM := testapps.NewCustomizedObj("resources/mysql-ins-config-invalid-update.yaml", &corev1.ConfigMap{}) Eventually(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(configmap), func(cm *corev1.ConfigMap) { cm.Data = invalidUpdatedCM.Data + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureUserSource) })).Should(Succeed()) By("check invalid update") Eventually(func(g Gomega) { cm := &corev1.ConfigMap{} g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(configmap), cm)).Should(Succeed()) - g.Expect(cm.Labels[constant.CMInsLastReconfigureMethodLabelKey]).Should(BeEquivalentTo(ReconfigureNoChangeType)) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).NotTo(BeTrue()) + // g.Expect(cm.Labels[constant.CMInsLastReconfigurePhaseKey]).Should(BeEquivalentTo(cfgcore.ReconfigureNoChangeType)) }).Should(Succeed()) By("restart Update") restartUpdatedCM := testapps.NewCustomizedObj("resources/mysql-ins-config-update-with-restart.yaml", &corev1.ConfigMap{}) Eventually(testapps.GetAndChangeObj(&testCtx, client.ObjectKeyFromObject(configmap), func(cm *corev1.ConfigMap) { cm.Data = restartUpdatedCM.Data + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureUserSource) })).Should(Succeed()) By("check invalid update") Eventually(func(g Gomega) { cm := &corev1.ConfigMap{} g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(configmap), cm)).Should(Succeed()) - g.Expect(cm.Labels[constant.CMInsLastReconfigureMethodLabelKey]).Should(BeEquivalentTo(ReconfigureSimpleType)) + g.Expect(cfgcore.IsNotUserReconfigureOperation(cm)).NotTo(BeTrue()) + g.Expect(cm.Labels[constant.CMInsLastReconfigurePhaseKey]).Should(BeEquivalentTo(cfgcore.ReconfigureSimplePhase)) }).Should(Succeed()) }) }) diff --git a/controllers/apps/operations/reconfigure_util.go b/controllers/apps/operations/reconfigure_util.go index 38dc826cb..829403f17 100644 --- a/controllers/apps/operations/reconfigure_util.go +++ b/controllers/apps/operations/reconfigure_util.go @@ -93,6 +93,7 @@ func persistCfgCM(cmObj *corev1.ConfigMap, newCfg map[string]string, cli client. cmObj.Annotations = make(map[string]string) } cmObj.Annotations[constant.LastAppliedOpsCRAnnotation] = opsCrName + cfgcore.SetParametersUpdateSource(cmObj, constant.ReconfigureUserSource) return cli.Patch(ctx, cmObj, patch) } diff --git a/internal/configuration/constrant.go b/internal/configuration/constrant.go new file mode 100644 index 000000000..fa2dea63b --- /dev/null +++ b/internal/configuration/constrant.go @@ -0,0 +1,28 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package configuration + +import appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + +const ( + ReconfigureCreatedPhase = "created" + ReconfigureNoChangeType = "noChange" + ReconfigureAutoReloadPhase = string(appsv1alpha1.AutoReload) + ReconfigureSimplePhase = string(appsv1alpha1.NormalPolicy) + ReconfigureParallelPhase = string(appsv1alpha1.RestartPolicy) + ReconfigureRollingPhase = string(appsv1alpha1.RollingPolicy) +) diff --git a/internal/configuration/reconfigure_util.go b/internal/configuration/reconfigure_util.go index bdf0586ee..9884ea695 100644 --- a/internal/configuration/reconfigure_util.go +++ b/internal/configuration/reconfigure_util.go @@ -21,8 +21,10 @@ import ( "reflect" "github.com/StudioSol/set" + corev1 "k8s.io/api/core/v1" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/constant" ) func getUpdateParameterList(cfg *ConfigPatchInfo) ([]string, error) { @@ -83,3 +85,36 @@ func IsUpdateDynamicParameters(cc *appsv1alpha1.ConfigConstraintSpec, cfg *Confi // restart is the default behavior. return false, nil } + +// IsParametersUpdateFromManager is used to check whether the parameters are updated from manager +func IsParametersUpdateFromManager(cm *corev1.ConfigMap) bool { + annotation := cm.ObjectMeta.Annotations + if annotation == nil { + return false + } + v := annotation[constant.KBParameterUpdateSourceAnnotationKey] + return v == constant.ReconfigureManagerSource +} + +// IsNotUserReconfigureOperation is used to check whether the parameters are updated from operation +func IsNotUserReconfigureOperation(cm *corev1.ConfigMap) bool { + labels := cm.GetLabels() + if labels == nil { + return true + } + lastReconfigurePhase := labels[constant.CMInsLastReconfigurePhaseKey] + return lastReconfigurePhase == "" || ReconfigureCreatedPhase == lastReconfigurePhase +} + +// SetParametersUpdateSource is used to set the parameters update source +// manager: parameter only updated from manager +// external-template: parameter only updated from template +// ops: parameter has updated from operation +func SetParametersUpdateSource(cm *corev1.ConfigMap, source string) { + annotation := cm.GetAnnotations() + if annotation == nil { + annotation = make(map[string]string) + } + annotation[constant.KBParameterUpdateSourceAnnotationKey] = source + cm.SetAnnotations(annotation) +} diff --git a/internal/constant/const.go b/internal/constant/const.go index 4daf4990d..7f78ebab1 100644 --- a/internal/constant/const.go +++ b/internal/constant/const.go @@ -92,6 +92,7 @@ const ( DisableUpgradeInsConfigurationAnnotationKey = "config.kubeblocks.io/disable-reconfigure" UpgradePolicyAnnotationKey = "config.kubeblocks.io/reconfigure-policy" UpgradeRestartAnnotationKey = "config.kubeblocks.io/restart" + KBParameterUpdateSourceAnnotationKey = "config.kubeblocks.io/reconfigure-source" // CMConfigurationTypeLabelKey configmap is config template type, e.g: "tpl", "instance" CMConfigurationTypeLabelKey = "config.kubeblocks.io/config-type" @@ -108,7 +109,8 @@ const ( // CMInsConfigurationLabelKey configmap is configuration file for component // CMInsConfigurationLabelKey = "config.kubeblocks.io/ins-configure" - CMInsLastReconfigureMethodLabelKey = "config.kubeblocks.io/last-applied-reconfigure-policy" + // CMInsLastReconfigurePhaseKey defines the current phase + CMInsLastReconfigurePhaseKey = "config.kubeblocks.io/last-applied-reconfigure-phase" // configuration finalizer ConfigurationTemplateFinalizerName = "config.kubeblocks.io/finalizer" @@ -189,6 +191,10 @@ const ( PodMinReadySecondsEnv = "POD_MIN_READY_SECONDS" ConfigTemplateType = "tpl" ConfigInstanceType = "instance" + + ReconfigureManagerSource = "manager" + ReconfigureUserSource = "ops" + ReconfigureTemplateSource = "external-template" ) const ( diff --git a/internal/controller/plan/template_wrapper.go b/internal/controller/plan/template_wrapper.go index 23c238b8b..dba2c2684 100644 --- a/internal/controller/plan/template_wrapper.go +++ b/internal/controller/plan/template_wrapper.go @@ -20,7 +20,9 @@ import ( "context" "strings" + "github.com/apecloud/kubeblocks/internal/generics" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" @@ -60,21 +62,57 @@ func newTemplateRenderWrapper(templateBuilder *configTemplateBuilder, cluster *a } } +func (wrapper *renderWrapper) enableRerenderTemplateSpec(cfgCMName string, task *intctrltypes.ReconcileTask) (bool, error) { + cmKey := client.ObjectKey{ + Name: cfgCMName, + Namespace: wrapper.cluster.Namespace, + } + + cmObj := &corev1.ConfigMap{} + localObject := task.GetLocalResourceWithObjectKey(cmKey, generics.ToGVK(cmObj)) + if localObject != nil { + return false, nil + } + + cmErr := wrapper.cli.Get(wrapper.ctx, cmKey, cmObj) + if cmErr != nil && !apierrors.IsNotFound(cmErr) { + // An unexpected error occurs + return false, cmErr + } + if cmErr != nil { + // Config is not exists + return true, nil + } + + // Config is exists + return cfgcore.IsNotUserReconfigureOperation(cmObj), nil +} + func (wrapper *renderWrapper) renderConfigTemplate(task *intctrltypes.ReconcileTask) error { + var err error + var enableRerender bool + scheme, _ := appsv1alpha1.SchemeBuilder.Build() for _, configSpec := range task.Component.ConfigTemplates { cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, configSpec.VolumeName) + if enableRerender, err = wrapper.enableRerenderTemplateSpec(cmName, task); err != nil { + return err + } + if !enableRerender { + continue + } // Generate ConfigMap objects for config files - cm, err := generateConfigMapFromTpl(wrapper.templateBuilder, cmName, configSpec.ConfigConstraintRef, configSpec.ComponentTemplateSpec, wrapper.params, wrapper.ctx, wrapper.cli, func(m map[string]string) error { - return validateRenderedData(m, configSpec, wrapper.ctx, wrapper.cli) - }) + cm, err := generateConfigMapFromTpl(wrapper.templateBuilder, cmName, configSpec.ConfigConstraintRef, configSpec.ComponentTemplateSpec, + wrapper.params, wrapper.ctx, wrapper.cli, func(m map[string]string) error { + return validateRenderedData(m, configSpec, wrapper.ctx, wrapper.cli) + }) if err != nil { return err } updateCMConfigSpecLabels(cm, configSpec) - if err := wrapper.addRenderObject(configSpec.ComponentTemplateSpec, cm, scheme); err != nil { + if err := wrapper.addRenderedObject(configSpec.ComponentTemplateSpec, cm, scheme); err != nil { return err } } @@ -85,30 +123,38 @@ func (wrapper *renderWrapper) renderScriptTemplate(task *intctrltypes.ReconcileT scheme, _ := appsv1alpha1.SchemeBuilder.Build() for _, templateSpec := range task.Component.ScriptTemplates { cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, templateSpec.VolumeName) + if task.GetLocalResourceWithObjectKey(client.ObjectKey{ + Name: cmName, + Namespace: wrapper.cluster.Namespace, + }, generics.ToGVK(&corev1.ConfigMap{})) != nil { + continue + } // Generate ConfigMap objects for config files cm, err := generateConfigMapFromTpl(wrapper.templateBuilder, cmName, "", templateSpec, wrapper.params, wrapper.ctx, wrapper.cli, nil) if err != nil { return err } - if err := wrapper.addRenderObject(templateSpec, cm, scheme); err != nil { + if err := wrapper.addRenderedObject(templateSpec, cm, scheme); err != nil { return err } } return nil } -func (wrapper *renderWrapper) addRenderObject(tpl appsv1alpha1.ComponentTemplateSpec, cm *corev1.ConfigMap, scheme *runtime.Scheme) error { +func (wrapper *renderWrapper) addRenderedObject(templateSpec appsv1alpha1.ComponentTemplateSpec, cm *corev1.ConfigMap, scheme *runtime.Scheme) error { // The owner of the configmap object is a cluster of users, // in order to manage the life cycle of configmap if err := controllerutil.SetOwnerReference(wrapper.cluster, cm, scheme); err != nil { return err } + cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureManagerSource) + cmName := cm.Name - wrapper.volumes[cmName] = tpl + wrapper.volumes[cmName] = templateSpec wrapper.renderedObjs = append(wrapper.renderedObjs, cm) - wrapper.templateAnnotations[cfgcore.GenerateTPLUniqLabelKeyWithConfig(tpl.Name)] = cmName + wrapper.templateAnnotations[cfgcore.GenerateTPLUniqLabelKeyWithConfig(templateSpec.Name)] = cmName return nil } diff --git a/internal/controller/types/task.go b/internal/controller/types/task.go index 0fdc227ca..dfa80c648 100644 --- a/internal/controller/types/task.go +++ b/internal/controller/types/task.go @@ -17,11 +17,13 @@ limitations under the License. package types import ( + "k8s.io/apimachinery/pkg/runtime/schema" "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" "github.com/apecloud/kubeblocks/internal/controller/builder" "github.com/apecloud/kubeblocks/internal/controller/component" + "github.com/apecloud/kubeblocks/internal/generics" ) type ReconcileTask struct { @@ -59,3 +61,17 @@ func (r *ReconcileTask) AppendResource(objs ...client.Object) { } *r.Resources = append(*r.Resources, objs...) } + +func (r *ReconcileTask) GetLocalResourceWithObjectKey(objKey client.ObjectKey, gvk schema.GroupVersionKind) client.Object { + if r.Resources == nil { + return nil + } + for _, obj := range *r.Resources { + if obj.GetName() == objKey.Name && obj.GetNamespace() == objKey.Namespace { + if generics.ToGVK(obj) == gvk { + return obj + } + } + } + return nil +} diff --git a/internal/generics/type.go b/internal/generics/type.go index 8524e46df..1095efa6f 100644 --- a/internal/generics/type.go +++ b/internal/generics/type.go @@ -17,12 +17,15 @@ limitations under the License. package generics import ( + "reflect" + snapshotv1 "github.com/kubernetes-csi/external-snapshotter/client/v6/apis/volumesnapshot/v1" appsv1 "k8s.io/api/apps/v1" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" policyv1 "k8s.io/api/policy/v1" storagev1 "k8s.io/api/storage/v1" + "k8s.io/apimachinery/pkg/runtime/schema" "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" @@ -92,3 +95,13 @@ var RestoreJobSignature = func(_ dataprotectionv1alpha1.RestoreJob, _ dataprotec } var AddonSignature = func(_ extensionsv1alpha1.Addon, _ extensionsv1alpha1.AddonList) { } + +func ToGVK(object client.Object) schema.GroupVersionKind { + t := reflect.TypeOf(object) + if t.Kind() != reflect.Pointer { + // Shouldn't ever get here. + return schema.GroupVersionKind{} + } + t = t.Elem() + return corev1.SchemeGroupVersion.WithKind(t.Name()) +} From 32ee676a63f3bc8f172b0d3bf256d88ff11511d9 Mon Sep 17 00:00:00 2001 From: huangzhangshu <109708205+JashBook@users.noreply.github.com> Date: Tue, 4 Apr 2023 18:43:09 +0800 Subject: [PATCH 45/80] chore: not check release note (#2419) --- .github/utils/get_release_version.py | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/utils/get_release_version.py b/.github/utils/get_release_version.py index 839f1fac5..e1bcf4177 100755 --- a/.github/utils/get_release_version.py +++ b/.github/utils/get_release_version.py @@ -32,7 +32,6 @@ githubEnv.write("WITH_RELEASE_NOTES=true\n") else: print("{} is not found".format(releaseNotePath)) - sys.exit(1) print("Release build from {} ...".format(gitRef)) githubEnv.write("REL_VERSION={}\n".format(releaseVersion)) From 7da01ac85a865491bc0f1784acbbcf8cdb677046 Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Tue, 4 Apr 2023 20:13:59 +0800 Subject: [PATCH 46/80] chore: use gitlab helm repo if failed to get ip location (#2421) --- internal/cli/util/util.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/internal/cli/util/util.go b/internal/cli/util/util.go index 5609c9756..89c72f61d 100644 --- a/internal/cli/util/util.go +++ b/internal/cli/util/util.go @@ -521,7 +521,7 @@ func IsSupportReconfigureParams(tpl appsv1alpha1.ComponentConfigSpec, values map } func getIPLocation() (string, error) { - client := &http.Client{} + client := &http.Client{Timeout: 10 * time.Second} req, err := http.NewRequest("GET", "https://ifconfig.io/country_code", nil) if err != nil { return "", err @@ -547,7 +547,8 @@ func GetHelmChartRepoURL() string { } location, _ := getIPLocation() - if location == "CN" { + // if location is CN, or we can not get location, use GitLab helm chart repo + if location == "CN" || location == "" { return types.GitLabHelmChartRepo } return types.KubeBlocksChartURL From 1141545ceb897a79b5d1a2cc9359e71235bf1ef6 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Tue, 4 Apr 2023 23:00:19 +0800 Subject: [PATCH 47/80] feat: refactor reconfigure partII: add sync policy for online update parameters (#2160) (#2257) --- apis/apps/v1alpha1/configconstraint_types.go | 4 + apis/apps/v1alpha1/type.go | 12 +- .../apps.kubeblocks.io_configconstraints.yaml | 4 + .../bases/apps.kubeblocks.io_opsrequests.yaml | 1 + .../apps/configuration/config_annotation.go | 2 +- .../configuration/parallel_upgrade_policy.go | 66 ++++------ .../parallel_upgrade_policy_test.go | 2 +- controllers/apps/configuration/policy_util.go | 59 +++++++-- .../apps/configuration/policy_util_test.go | 35 ++++++ .../apps/configuration/reconfigure_policy.go | 35 ++++-- .../configuration/rolling_upgrade_policy.go | 23 ++-- .../apps/configuration/simple_policy.go | 5 +- .../apps/configuration/sync_upgrade_policy.go | 119 ++++++++++++++++++ .../configuration/sync_upgrade_policy_test.go | 107 ++++++++++++++++ .../templates/configconstraint.yaml | 1 + .../apps.kubeblocks.io_configconstraints.yaml | 4 + .../crds/apps.kubeblocks.io_opsrequests.yaml | 1 + .../configuration/config_manager/builder.go | 4 +- 18 files changed, 406 insertions(+), 78 deletions(-) create mode 100644 controllers/apps/configuration/sync_upgrade_policy.go create mode 100644 controllers/apps/configuration/sync_upgrade_policy_test.go diff --git a/apis/apps/v1alpha1/configconstraint_types.go b/apis/apps/v1alpha1/configconstraint_types.go index 1ee208de3..97efcd418 100644 --- a/apis/apps/v1alpha1/configconstraint_types.go +++ b/apis/apps/v1alpha1/configconstraint_types.go @@ -136,6 +136,10 @@ type TPLScriptTrigger struct { // +kubebuilder:default="default" // +optional Namespace string `json:"namespace,omitempty"` + + // Specify synchronize updates parameters to the config manager. + // +optional + Sync *bool `json:"sync,omitempty"` } type FormatterConfig struct { diff --git a/apis/apps/v1alpha1/type.go b/apis/apps/v1alpha1/type.go index 081534160..07bf3fdb2 100644 --- a/apis/apps/v1alpha1/type.go +++ b/apis/apps/v1alpha1/type.go @@ -336,14 +336,16 @@ const ( // UpgradePolicy defines the policy of reconfiguring. // +enum -// +kubebuilder:validation:Enum={simple,parallel,rolling,autoReload} +// +kubebuilder:validation:Enum={simple,parallel,rolling,autoReload,operatorSyncUpdate} type UpgradePolicy string const ( - NormalPolicy UpgradePolicy = "simple" - RestartPolicy UpgradePolicy = "parallel" - RollingPolicy UpgradePolicy = "rolling" - AutoReload UpgradePolicy = "autoReload" + NonePolicy UpgradePolicy = "none" + NormalPolicy UpgradePolicy = "simple" + RestartPolicy UpgradePolicy = "parallel" + RollingPolicy UpgradePolicy = "rolling" + AutoReload UpgradePolicy = "autoReload" + OperatorSyncUpdate UpgradePolicy = "operatorSyncUpdate" ) // CfgReloadType defines reload method. diff --git a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml index 6860e7d49..ee9f6ff5b 100644 --- a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml +++ b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml @@ -149,6 +149,10 @@ spec: scriptConfigMapRef: description: scriptConfigMapRef used to execute for reload. type: string + sync: + description: Specify synchronize updates parameters to the + config manager. + type: boolean required: - scriptConfigMapRef type: object diff --git a/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml b/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml index 7101384a0..0ab01c7f7 100644 --- a/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml +++ b/config/crd/bases/apps.kubeblocks.io_opsrequests.yaml @@ -703,6 +703,7 @@ spec: - parallel - rolling - autoReload + - operatorSyncUpdate type: string updatedParameters: description: updatedParameters describes the updated parameters. diff --git a/controllers/apps/configuration/config_annotation.go b/controllers/apps/configuration/config_annotation.go index badcbaa9c..e56636c58 100644 --- a/controllers/apps/configuration/config_annotation.go +++ b/controllers/apps/configuration/config_annotation.go @@ -127,7 +127,7 @@ func getLastVersionConfig(cm *corev1.ConfigMap) (map[string]string, error) { func getUpgradePolicy(cm *corev1.ConfigMap) appsv1alpha1.UpgradePolicy { const ( - DefaultUpgradePolicy = appsv1alpha1.NormalPolicy + DefaultUpgradePolicy = appsv1alpha1.NonePolicy ) annotations := cm.GetAnnotations() diff --git a/controllers/apps/configuration/parallel_upgrade_policy.go b/controllers/apps/configuration/parallel_upgrade_policy.go index 8f4c2b93e..c32f39bec 100644 --- a/controllers/apps/configuration/parallel_upgrade_policy.go +++ b/controllers/apps/configuration/parallel_upgrade_policy.go @@ -18,10 +18,10 @@ package configuration import ( corev1 "k8s.io/api/core/v1" - "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgcore "github.com/apecloud/kubeblocks/internal/configuration" + podutil "github.com/apecloud/kubeblocks/internal/controllerutil" ) type parallelUpgradePolicy struct { @@ -32,57 +32,45 @@ func init() { } func (p *parallelUpgradePolicy) Upgrade(params reconfigureParams) (ReturnedStatus, error) { - if finished, err := p.restartPods(params); err != nil { - return makeReturnedStatus(ESAndRetryFailed), err - } else if !finished { - return makeReturnedStatus(ESRetry), nil - } - - return makeReturnedStatus(ESNone), nil -} - -func (p *parallelUpgradePolicy) GetPolicyName() string { - return string(appsv1alpha1.RestartPolicy) -} - -func (p *parallelUpgradePolicy) restartPods(params reconfigureParams) (bool, error) { - var ( - funcs RollingUpgradeFuncs - cType = params.WorkloadType() - configKey = params.getConfigKey() - configVersion = params.getTargetVersionHash() - ) - - updatePodLabelsVersion := func(pod *corev1.Pod, labelKey, labelValue string) error { - patch := client.MergeFrom(pod.DeepCopy()) - if pod.Labels == nil { - pod.Labels = make(map[string]string, 1) - } - pod.Labels[labelKey] = labelValue - return params.Client.Patch(params.Ctx.Ctx, pod, patch) - } + var funcs RollingUpgradeFuncs - switch cType { + switch params.WorkloadType() { + default: + return makeReturnedStatus(ESNotSupport), cfgcore.MakeError("not support component workload type[%s]", params.WorkloadType()) case appsv1alpha1.Consensus: funcs = GetConsensusRollingUpgradeFuncs() case appsv1alpha1.Stateful: funcs = GetStatefulSetRollingUpgradeFuncs() - default: - return false, cfgcore.MakeError("not support component workload type[%s]", cType) + case appsv1alpha1.Replication: + funcs = GetReplicationRollingUpgradeFuncs() } pods, err := funcs.GetPodsFunc(params) if err != nil { - return false, err + return makeReturnedStatus(ESAndRetryFailed), err } + return p.restartPods(params, pods, funcs) +} + +func (p *parallelUpgradePolicy) GetPolicyName() string { + return string(appsv1alpha1.RestartPolicy) +} + +func (p *parallelUpgradePolicy) restartPods(params reconfigureParams, pods []corev1.Pod, funcs RollingUpgradeFuncs) (ReturnedStatus, error) { + var configKey = params.getConfigKey() + var configVersion = params.getTargetVersionHash() + for _, pod := range pods { - if err := funcs.RestartContainerFunc(&pod, params.ContainerNames, params.ReconfigureClientFactory); err != nil { - return false, err + if podutil.IsMatchConfigVersion(&pod, configKey, configVersion) { + continue + } + if err := funcs.RestartContainerFunc(&pod, params.Ctx.Ctx, params.ContainerNames, params.ReconfigureClientFactory); err != nil { + return makeReturnedStatus(ESAndRetryFailed), err } - if err := updatePodLabelsVersion(&pod, configKey, configVersion); err != nil { - return false, err + if err := updatePodLabelsWithConfigVersion(&pod, configKey, configVersion, params.Client, params.Ctx.Ctx); err != nil { + return makeReturnedStatus(ESAndRetryFailed), err } } - return true, nil + return makeReturnedStatus(ESNone), nil } diff --git a/controllers/apps/configuration/parallel_upgrade_policy_test.go b/controllers/apps/configuration/parallel_upgrade_policy_test.go index 738bac40f..beb490a55 100644 --- a/controllers/apps/configuration/parallel_upgrade_policy_test.go +++ b/controllers/apps/configuration/parallel_upgrade_policy_test.go @@ -208,7 +208,7 @@ var _ = Describe("Reconfigure ParallelPolicy", func() { status, err := parallelPolicy.Upgrade(mockParam) Expect(err).ShouldNot(Succeed()) Expect(err.Error()).Should(ContainSubstring("not support component workload type")) - Expect(status.Status).Should(BeEquivalentTo(ESAndRetryFailed)) + Expect(status.Status).Should(BeEquivalentTo(ESNotSupport)) }) }) }) diff --git a/controllers/apps/configuration/policy_util.go b/controllers/apps/configuration/policy_util.go index be9857ef8..d5f131c9b 100644 --- a/controllers/apps/configuration/policy_util.go +++ b/controllers/apps/configuration/policy_util.go @@ -36,40 +36,55 @@ type createReconfigureClient func(addr string) (cfgproto.ReconfigureClient, erro type GetPodsFunc func(params reconfigureParams) ([]corev1.Pod, error) -type RestartContainerFunc func(pod *corev1.Pod, containerName []string, createConnFn createReconfigureClient) error +type RestartContainerFunc func(pod *corev1.Pod, ctx context.Context, containerName []string, createConnFn createReconfigureClient) error +type OnlineUpdatePodFunc func(pod *corev1.Pod, ctx context.Context, createClient createReconfigureClient, configSpec string, updatedParams map[string]string) error type RollingUpgradeFuncs struct { GetPodsFunc GetPodsFunc RestartContainerFunc RestartContainerFunc + OnlineUpdatePodFunc OnlineUpdatePodFunc } func GetConsensusRollingUpgradeFuncs() RollingUpgradeFuncs { return RollingUpgradeFuncs{ GetPodsFunc: getConsensusPods, - RestartContainerFunc: commonStopContainer, + RestartContainerFunc: commonStopContainerWithPod, + OnlineUpdatePodFunc: commonOnlineUpdateWithPod, } } func GetStatefulSetRollingUpgradeFuncs() RollingUpgradeFuncs { return RollingUpgradeFuncs{ GetPodsFunc: getStatefulSetPods, - RestartContainerFunc: commonStopContainer, + RestartContainerFunc: commonStopContainerWithPod, + OnlineUpdatePodFunc: commonOnlineUpdateWithPod, } } func GetReplicationRollingUpgradeFuncs() RollingUpgradeFuncs { return RollingUpgradeFuncs{ GetPodsFunc: getReplicationSetPods, - RestartContainerFunc: commonStopContainer, + RestartContainerFunc: commonStopContainerWithPod, + OnlineUpdatePodFunc: commonOnlineUpdateWithPod, } } -func getReplicationSetPods(params reconfigureParams) ([]corev1.Pod, error) { - var ( - ctx = params.Ctx - cluster = params.Cluster - ) +func GetDeploymentRollingUpgradeFuncs() RollingUpgradeFuncs { + return RollingUpgradeFuncs{ + GetPodsFunc: getDeploymentRollingPods, + RestartContainerFunc: commonStopContainerWithPod, + OnlineUpdatePodFunc: commonOnlineUpdateWithPod, + } +} +func getDeploymentRollingPods(params reconfigureParams) ([]corev1.Pod, error) { + // util.GetComponentPodList support deployment + return getReplicationSetPods(params) +} + +func getReplicationSetPods(params reconfigureParams) ([]corev1.Pod, error) { + var ctx = params.Ctx + var cluster = params.Cluster podList, err := util.GetComponentPodList(ctx.Ctx, params.Client, *cluster, params.ClusterComponent.Name) if err != nil { return nil, err @@ -149,7 +164,29 @@ func getConsensusPods(params reconfigureParams) ([]corev1.Pod, error) { return r, nil } -func commonStopContainer(pod *corev1.Pod, containerNames []string, createClient createReconfigureClient) error { +// TODO commonOnlineUpdateWithPod migrate to sql command pipeline +func commonOnlineUpdateWithPod(pod *corev1.Pod, ctx context.Context, createClient createReconfigureClient, configSpec string, updatedParams map[string]string) error { + client, err := createClient(generateManagerSidecarAddr(pod)) + if err != nil { + return err + } + + response, err := client.OnlineUpgradeParams(ctx, &cfgproto.OnlineUpgradeParamsRequest{ + ConfigSpec: configSpec, + Params: updatedParams, + }) + if err != nil { + return err + } + + errMessage := response.GetErrMessage() + if errMessage != "" { + return cfgcore.MakeError(errMessage) + } + return nil +} + +func commonStopContainerWithPod(pod *corev1.Pod, ctx context.Context, containerNames []string, createClient createReconfigureClient) error { containerIDs := make([]string, 0, len(containerNames)) for _, name := range containerNames { containerID := intctrlutil.GetContainerID(pod, name) @@ -165,7 +202,7 @@ func commonStopContainer(pod *corev1.Pod, containerNames []string, createClient return err } - response, err := client.StopContainer(context.Background(), &cfgproto.StopContainerRequest{ + response, err := client.StopContainer(ctx, &cfgproto.StopContainerRequest{ ContainerIDs: containerIDs, }) if err != nil { diff --git a/controllers/apps/configuration/policy_util_test.go b/controllers/apps/configuration/policy_util_test.go index e12581c4f..cf4949e60 100644 --- a/controllers/apps/configuration/policy_util_test.go +++ b/controllers/apps/configuration/policy_util_test.go @@ -31,6 +31,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/log" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" ) @@ -109,6 +110,40 @@ func withConfigSpec(configSpecName string, data map[string]string) ParamsOps { } } +func withConfigConstraintSpec(formatter *appsv1alpha1.FormatterConfig) ParamsOps { + return func(params *reconfigureParams) { + params.ConfigConstraint = &appsv1alpha1.ConfigConstraintSpec{ + FormatterConfig: formatter, + } + } +} + +func withConfigPatch(patch map[string]string) ParamsOps { + mockEmptyData := func(m map[string]string) map[string]string { + r := make(map[string]string, len(patch)) + for key := range m { + r[key] = "" + } + return r + } + transKeyPair := func(pts map[string]string) map[string]interface{} { + m := make(map[string]interface{}, len(pts)) + for key, value := range pts { + m[key] = value + } + return m + } + return func(params *reconfigureParams) { + cc := params.ConfigConstraint + newConfigData, _ := cfgcore.MergeAndValidateConfigs(*cc, map[string]string{"for_test": ""}, nil, []cfgcore.ParamPairs{{ + Key: "for_test", + UpdatedParams: transKeyPair(patch), + }}) + configPatch, _, _ := cfgcore.CreateConfigPatch(mockEmptyData(newConfigData), newConfigData, cc.FormatterConfig.Format, nil, false) + params.ConfigPatch = configPatch + } +} + func withCDComponent(compType appsv1alpha1.WorkloadType, tpls []appsv1alpha1.ComponentConfigSpec) ParamsOps { return func(params *reconfigureParams) { params.Component = &appsv1alpha1.ClusterComponentDefinition{ diff --git a/controllers/apps/configuration/reconfigure_policy.go b/controllers/apps/configuration/reconfigure_policy.go index 8f10e9ed0..47ef8a358 100644 --- a/controllers/apps/configuration/reconfigure_policy.go +++ b/controllers/apps/configuration/reconfigure_policy.go @@ -197,23 +197,44 @@ func (receiver AutoReloadPolicy) GetPolicyName() string { func NewReconfigurePolicy(cc *appsv1alpha1.ConfigConstraintSpec, cfgPatch *cfgcore.ConfigPatchInfo, policy appsv1alpha1.UpgradePolicy, restart bool) (reconfigurePolicy, error) { if !cfgPatch.IsModify { - // not exec here + // not walk here return nil, cfgcore.MakeError("cfg not modify. [%v]", cfgPatch) } - actionType := policy - if !restart { + if enableAutoDecision(restart, policy) { if dynamicUpdate, err := cfgcore.IsUpdateDynamicParameters(cc, cfgPatch); err != nil { return nil, err } else if dynamicUpdate { - actionType = appsv1alpha1.AutoReload + policy = appsv1alpha1.AutoReload + } + if enableSyncReload(policy, cc.ReloadOptions) { + policy = appsv1alpha1.OperatorSyncUpdate } } - - if action, ok := upgradePolicyMap[actionType]; ok { + if policy == appsv1alpha1.NonePolicy { + policy = appsv1alpha1.NormalPolicy + } + if action, ok := upgradePolicyMap[policy]; ok { return action, nil } - return nil, cfgcore.MakeError("not support upgrade policy:[%s]", actionType) + return nil, cfgcore.MakeError("not support upgrade policy:[%s]", policy) +} + +func enableAutoDecision(restart bool, policy appsv1alpha1.UpgradePolicy) bool { + return !restart && policy == appsv1alpha1.NonePolicy +} + +func enableSyncReload(policyType appsv1alpha1.UpgradePolicy, options *appsv1alpha1.ReloadOptions) bool { + return policyType == appsv1alpha1.AutoReload && enableSyncTrigger(options) +} + +func enableSyncTrigger(options *appsv1alpha1.ReloadOptions) bool { + if options == nil || options.TPLScriptTrigger == nil { + return false + } + + trigger := options.TPLScriptTrigger + return trigger.Sync != nil && *trigger.Sync } func withSucceed(succeedCount int32) func(status *ReturnedStatus) { diff --git a/controllers/apps/configuration/rolling_upgrade_policy.go b/controllers/apps/configuration/rolling_upgrade_policy.go index 44d4ac85f..71341d026 100644 --- a/controllers/apps/configuration/rolling_upgrade_policy.go +++ b/controllers/apps/configuration/rolling_upgrade_policy.go @@ -17,6 +17,7 @@ limitations under the License. package configuration import ( + "context" "os" "github.com/spf13/viper" @@ -95,15 +96,6 @@ func performRollingUpgrade(params reconfigureParams, funcs RollingUpgradeFuncs) configVersion = params.getTargetVersionHash() ) - updatePodLabelsVersion := func(pod *corev1.Pod, labelKey, labelValue string) error { - patch := client.MergeFrom(pod.DeepCopy()) - if pod.Labels == nil { - pod.Labels = make(map[string]string, 1) - } - pod.Labels[labelKey] = labelValue - return params.Client.Patch(params.Ctx.Ctx, pod, patch) - } - if !canPerformUpgrade(pods, params) { return makeReturnedStatus(ESRetry), nil } @@ -125,10 +117,10 @@ func performRollingUpgrade(params reconfigureParams, funcs RollingUpgradeFuncs) params.Ctx.Log.Info("pod is rolling updating.", "pod name", pod.Name) continue } - if err := funcs.RestartContainerFunc(&pod, params.ContainerNames, params.ReconfigureClientFactory); err != nil { + if err := funcs.RestartContainerFunc(&pod, params.Ctx.Ctx, params.ContainerNames, params.ReconfigureClientFactory); err != nil { return makeReturnedStatus(ESAndRetryFailed), err } - if err := updatePodLabelsVersion(&pod, configKey, configVersion); err != nil { + if err := updatePodLabelsWithConfigVersion(&pod, configKey, configVersion, params.Client, params.Ctx.Ctx); err != nil { return makeReturnedStatus(ESAndRetryFailed), err } } @@ -242,3 +234,12 @@ type switchWindow struct { func (w *switchWindow) getWaitRollingPods() []corev1.Pod { return w.pods[w.begin:w.end] } + +func updatePodLabelsWithConfigVersion(pod *corev1.Pod, labelKey, configVersion string, cli client.Client, ctx context.Context) error { + patch := client.MergeFrom(pod.DeepCopy()) + if pod.Labels == nil { + pod.Labels = make(map[string]string, 1) + } + pod.Labels[labelKey] = configVersion + return cli.Patch(ctx, pod, patch) +} diff --git a/controllers/apps/configuration/simple_policy.go b/controllers/apps/configuration/simple_policy.go index 1d2e0e744..3df11dbe9 100644 --- a/controllers/apps/configuration/simple_policy.go +++ b/controllers/apps/configuration/simple_policy.go @@ -54,8 +54,9 @@ func rollingStatefulSets(param reconfigureParams) (ReturnedStatus, error) { client = param.Client newVersion = param.getTargetVersionHash() configKey = param.getConfigKey() - progress = cfgcore.NotStarted - retStatus = ESRetry + + retStatus = ESRetry + progress = cfgcore.NotStarted ) if configKey == "" { diff --git a/controllers/apps/configuration/sync_upgrade_policy.go b/controllers/apps/configuration/sync_upgrade_policy.go new file mode 100644 index 000000000..1cbd15433 --- /dev/null +++ b/controllers/apps/configuration/sync_upgrade_policy.go @@ -0,0 +1,119 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package configuration + +import ( + corev1 "k8s.io/api/core/v1" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + cfgutil "github.com/apecloud/kubeblocks/internal/configuration" + podutil "github.com/apecloud/kubeblocks/internal/controllerutil" +) + +type syncPolicy struct { +} + +func init() { + RegisterPolicy(appsv1alpha1.OperatorSyncUpdate, &syncPolicy{}) +} + +func (o *syncPolicy) GetPolicyName() string { + return string(appsv1alpha1.OperatorSyncUpdate) +} + +func (o *syncPolicy) Upgrade(params reconfigureParams) (ReturnedStatus, error) { + configPatch := params.ConfigPatch + if !configPatch.IsModify { + return makeReturnedStatus(ESNone), nil + } + + updatedParameters := getOnlineUpdateParams(configPatch, params.ConfigConstraint.FormatterConfig) + if len(updatedParameters) == 0 { + return makeReturnedStatus(ESNone), nil + } + + var funcs RollingUpgradeFuncs + switch params.WorkloadType() { + default: + return makeReturnedStatus(ESNotSupport), cfgutil.MakeError("not support component workload type[%s]", params.WorkloadType()) + case appsv1alpha1.Stateless: + funcs = GetDeploymentRollingUpgradeFuncs() + case appsv1alpha1.Consensus: + funcs = GetConsensusRollingUpgradeFuncs() + case appsv1alpha1.Stateful: + funcs = GetStatefulSetRollingUpgradeFuncs() + case appsv1alpha1.Replication: + funcs = GetReplicationRollingUpgradeFuncs() + } + + pods, err := funcs.GetPodsFunc(params) + if err != nil { + return makeReturnedStatus(ESAndRetryFailed), err + } + return sync(params, updatedParameters, pods, funcs) +} + +func sync(params reconfigureParams, updatedParameters map[string]string, pods []corev1.Pod, funcs RollingUpgradeFuncs) (ReturnedStatus, error) { + var ( + r = ESNone + total = int32(len(pods)) + replicas = int32(params.getTargetReplicas()) + progress = cfgutil.NotStarted + + err error + ctx = params.Ctx.Ctx + configKey = params.getConfigKey() + versionHash = params.getTargetVersionHash() + ) + + for _, pod := range pods { + if podutil.IsMatchConfigVersion(&pod, configKey, versionHash) { + progress++ + continue + } + if !podutil.PodIsReady(&pod) { + continue + } + err = funcs.OnlineUpdatePodFunc(&pod, ctx, params.ReconfigureClientFactory, params.ConfigSpecName, updatedParameters) + if err != nil { + return makeReturnedStatus(ESAndRetryFailed), err + } + err = updatePodLabelsWithConfigVersion(&pod, configKey, versionHash, params.Client, ctx) + if err != nil { + return makeReturnedStatus(ESAndRetryFailed), err + } + progress++ + } + + if total != progress || replicas != total { + r = ESRetry + } + return makeReturnedStatus(r, withExpected(replicas), withSucceed(progress)), nil +} + +func getOnlineUpdateParams(configPatch *cfgutil.ConfigPatchInfo, formatConfig *appsv1alpha1.FormatterConfig) map[string]string { + r := make(map[string]string) + parameters := cfgutil.GenerateVisualizedParamsList(configPatch, formatConfig, nil) + for _, key := range parameters { + if key.UpdateType == cfgutil.UpdatedType { + for _, p := range key.Parameters { + r[p.Key] = p.Value + } + } + } + return r +} diff --git a/controllers/apps/configuration/sync_upgrade_policy_test.go b/controllers/apps/configuration/sync_upgrade_policy_test.go new file mode 100644 index 000000000..6108e9c5a --- /dev/null +++ b/controllers/apps/configuration/sync_upgrade_policy_test.go @@ -0,0 +1,107 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package configuration + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/runtime" + + "github.com/golang/mock/gomock" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + cfgproto "github.com/apecloud/kubeblocks/internal/configuration/proto" + mock_proto "github.com/apecloud/kubeblocks/internal/configuration/proto/mocks" + testutil "github.com/apecloud/kubeblocks/internal/testutil/k8s" +) + +var operatorSyncPolicy = &syncPolicy{} + +var _ = Describe("Reconfigure OperatorSyncPolicy", func() { + + var ( + k8sMockClient *testutil.K8sClientMockHelper + reconfigureClient *mock_proto.MockReconfigureClient + ) + + BeforeEach(func() { + k8sMockClient = testutil.NewK8sMockClient() + reconfigureClient = mock_proto.NewMockReconfigureClient(k8sMockClient.Controller()) + }) + + AfterEach(func() { + k8sMockClient.Finish() + }) + + Context("sync reconfigure policy test", func() { + It("Should success without error", func() { + By("check policy name") + Expect(operatorSyncPolicy.GetPolicyName()).Should(BeEquivalentTo("operatorSyncUpdate")) + + By("prepare reconfigure policy params") + mockParam := newMockReconfigureParams("operatorSyncPolicy", k8sMockClient.Client(), + withGRPCClient(func(addr string) (cfgproto.ReconfigureClient, error) { + return reconfigureClient, nil + }), + withMockStatefulSet(3, nil), + withConfigSpec("for_test", map[string]string{"a": "c b e f"}), + withConfigConstraintSpec(&appsv1alpha1.FormatterConfig{Format: appsv1alpha1.RedisCfg}), + withConfigPatch(map[string]string{ + "a": "c b e f", + }), + withClusterComponent(3), + withCDComponent(appsv1alpha1.Consensus, []appsv1alpha1.ComponentConfigSpec{{ + ComponentTemplateSpec: appsv1alpha1.ComponentTemplateSpec{ + Name: "for_test", + VolumeName: "test_volume", + }, + }})) + + By("mock client get pod caller") + k8sMockClient.MockListMethod(testutil.WithListReturned( + testutil.WithConstructListSequenceResult([][]runtime.Object{ + fromPodObjectList(newMockPodsWithStatefulSet(&mockParam.ComponentUnits[0], 3, + withReadyPod(0, 1))), + fromPodObjectList(newMockPodsWithStatefulSet(&mockParam.ComponentUnits[0], 3, + withReadyPod(0, 3))), + }), + testutil.WithAnyTimes())) + + By("mock client patch caller") + // mock client update caller + k8sMockClient.MockPatchMethod(testutil.WithSucceed(testutil.WithMinTimes(3))) + + By("mock remote online update caller") + reconfigureClient.EXPECT().OnlineUpgradeParams(gomock.Any(), gomock.Any()).Return( + &cfgproto.OnlineUpgradeParamsResponse{}, nil). + MinTimes(3) + + status, err := operatorSyncPolicy.Upgrade(mockParam) + Expect(err).Should(Succeed()) + Expect(status.Status).Should(BeEquivalentTo(ESRetry)) + Expect(status.SucceedCount).Should(BeEquivalentTo(1)) + Expect(status.ExpectedCount).Should(BeEquivalentTo(3)) + + status, err = operatorSyncPolicy.Upgrade(mockParam) + Expect(err).Should(Succeed()) + Expect(status.Status).Should(BeEquivalentTo(ESNone)) + Expect(status.SucceedCount).Should(BeEquivalentTo(3)) + Expect(status.ExpectedCount).Should(BeEquivalentTo(3)) + }) + }) + +}) diff --git a/deploy/apecloud-mysql/templates/configconstraint.yaml b/deploy/apecloud-mysql/templates/configconstraint.yaml index 2551779a4..3e3e49385 100644 --- a/deploy/apecloud-mysql/templates/configconstraint.yaml +++ b/deploy/apecloud-mysql/templates/configconstraint.yaml @@ -10,6 +10,7 @@ spec: # tplRef: mysql-3node-tpl-8.0 reloadOptions: tplScriptTrigger: + sync: true scriptConfigMapRef: mysql-reload-script namespace: {{ .Release.Namespace }} diff --git a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml index 6860e7d49..ee9f6ff5b 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml @@ -149,6 +149,10 @@ spec: scriptConfigMapRef: description: scriptConfigMapRef used to execute for reload. type: string + sync: + description: Specify synchronize updates parameters to the + config manager. + type: boolean required: - scriptConfigMapRef type: object diff --git a/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml b/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml index 7101384a0..0ab01c7f7 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml @@ -703,6 +703,7 @@ spec: - parallel - rolling - autoReload + - operatorSyncUpdate type: string updatedParameters: description: updatedParameters describes the updated parameters. diff --git a/internal/configuration/config_manager/builder.go b/internal/configuration/config_manager/builder.go index f7b418035..d0a3fe4a1 100644 --- a/internal/configuration/config_manager/builder.go +++ b/internal/configuration/config_manager/builder.go @@ -63,7 +63,9 @@ func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []cor } args := buildConfigManagerCommonArgs(volumeDirs) - args = append(args, "--operator-update-enable") + if options.Sync != nil && *options.Sync { + args = append(args, "--operator-update-enable") + } args = append(args, "--tcp", viper.GetString(constant.ConfigManagerGPRCPortEnv)) args = append(args, "--notify-type", string(appsv1alpha1.TPLScriptType)) args = append(args, "--tpl-config", filepath.Join(scriptVolumePath, configTemplateName)) From eece554f7f6edcf2f870248ceb110b9c3b9d0361 Mon Sep 17 00:00:00 2001 From: chantu Date: Thu, 6 Apr 2023 11:26:38 +0800 Subject: [PATCH 48/80] fix: helm upgrade nil pointer error (#2416) --- internal/cli/util/helm/helm.go | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/internal/cli/util/helm/helm.go b/internal/cli/util/helm/helm.go index 0492e63c8..a67775d6f 100644 --- a/internal/cli/util/helm/helm.go +++ b/internal/cli/util/helm/helm.go @@ -406,7 +406,9 @@ func (i *InstallOpts) tryUpgrade(cfg *action.Configuration) (string, error) { } else { client.Version = installed.Chart.AppVersion() } - client.ReuseValues = true + // do not use helm's ReuseValues, do it ourselves, helm's default upgrade also set it to false + // if ReuseValues set to true, helm will use old values instead of new ones, which will cause nil pointer error if new values added. + client.ReuseValues = false cp, err := client.ChartPathOptions.LocateChart(i.Chart, settings) if err != nil { @@ -418,6 +420,17 @@ func (i *InstallOpts) tryUpgrade(cfg *action.Configuration) (string, error) { if err != nil { return "", err } + // get coalesced values of current chart + currentValues, err := chartutil.CoalesceValues(installed.Chart, installed.Config) + if err != nil { + return "", err + } + // merge current values into vals, so current release's user values can be kept + installed.Chart.Values = currentValues + vals, err = chartutil.CoalesceValues(installed.Chart, vals) + if err != nil { + return "", err + } // Check Chart dependencies to make sure all are present in /charts chartRequested, err := loader.Load(cp) From e41826fa66fc2eb5b86bd55ff42fdf6f993e77b3 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Thu, 6 Apr 2023 12:58:55 +0800 Subject: [PATCH 49/80] feat: config render support secret and configmap for build-in getEnvByName (#2324) (#2338) --- internal/controller/plan/builtin_env.go | 314 ++++++++++++++++++ internal/controller/plan/builtin_env_test.go | 278 ++++++++++++++++ internal/controller/plan/builtin_functions.go | 14 - internal/controller/plan/config_template.go | 10 +- .../controller/plan/config_template_test.go | 6 +- internal/controller/plan/prepare.go | 2 +- 6 files changed, 602 insertions(+), 22 deletions(-) create mode 100644 internal/controller/plan/builtin_env.go create mode 100644 internal/controller/plan/builtin_env_test.go diff --git a/internal/controller/plan/builtin_env.go b/internal/controller/plan/builtin_env.go new file mode 100644 index 000000000..18e8593cf --- /dev/null +++ b/internal/controller/plan/builtin_env.go @@ -0,0 +1,314 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package plan + +import ( + b64 "encoding/base64" + "regexp" + "strings" + + "github.com/StudioSol/set" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/kubectl/pkg/util/resource" + coreclient "sigs.k8s.io/controller-runtime/pkg/client" + + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" + "github.com/apecloud/kubeblocks/internal/constant" + "github.com/apecloud/kubeblocks/internal/controller/component" + intctrltypes "github.com/apecloud/kubeblocks/internal/controller/types" + "github.com/apecloud/kubeblocks/internal/generics" +) + +type envBuildInFunc func(container interface{}, envName string) (string, error) + +type envWrapper struct { + // prevent circular references. + referenceCount int + *configTemplateBuilder + + // configmap or secret not yet submitted. + localObjects *intctrltypes.ReconcileTask + // cache remoted configmap and secret. + cache map[schema.GroupVersionKind]map[coreclient.ObjectKey]coreclient.Object +} + +const maxReferenceCount = 10 + +func wrapGetEnvByName(templateBuilder *configTemplateBuilder, localObjects *intctrltypes.ReconcileTask) envBuildInFunc { + wrapper := &envWrapper{ + configTemplateBuilder: templateBuilder, + localObjects: localObjects, + cache: make(map[schema.GroupVersionKind]map[coreclient.ObjectKey]coreclient.Object), + } + return func(args interface{}, envName string) (string, error) { + container, err := fromJSONObject[corev1.Container](args) + if err != nil { + return "", err + } + return wrapper.getEnvByName(container, envName) + } +} + +func (w *envWrapper) getEnvByName(container *corev1.Container, envName string) (string, error) { + for _, v := range container.Env { + if v.Name != envName { + continue + } + switch { + case v.ValueFrom == nil: + return w.checkAndReplaceEnv(v.Value, container) + case v.ValueFrom.ConfigMapKeyRef != nil: + return w.configMapValue(v.ValueFrom.ConfigMapKeyRef, container) + case v.ValueFrom.SecretKeyRef != nil: + return w.secretValue(v.ValueFrom.SecretKeyRef, container) + case v.ValueFrom.FieldRef != nil: + return fieldRefValue(v.ValueFrom.FieldRef, w.podSpec) + case v.ValueFrom.ResourceFieldRef != nil: + return resourceRefValue(v.ValueFrom.ResourceFieldRef, w.podSpec.Containers, container) + } + } + return w.getEnvFromResources(container.EnvFrom, envName, container) +} + +func (w *envWrapper) getEnvFromResources(envSources []corev1.EnvFromSource, envName string, container *corev1.Container) (string, error) { + for _, source := range envSources { + if value, err := w.getEnvFromResource(source, envName, container); err != nil { + return "", err + } else if value != "" { + return w.checkAndReplaceEnv(value, container) + } + } + return "", nil +} + +func (w *envWrapper) getEnvFromResource(envSource corev1.EnvFromSource, envName string, container *corev1.Container) (string, error) { + fromConfigMap := func(configmapRef *corev1.ConfigMapEnvSource) (string, error) { + return w.configMapValue(&corev1.ConfigMapKeySelector{ + Key: envName, + LocalObjectReference: corev1.LocalObjectReference{Name: configmapRef.Name}, + }, container) + } + fromSecret := func(secretRef *corev1.SecretEnvSource) (string, error) { + return w.secretValue(&corev1.SecretKeySelector{ + Key: envName, + LocalObjectReference: corev1.LocalObjectReference{Name: secretRef.Name}, + }, container) + } + + switch { + default: + return "", nil + case envSource.ConfigMapRef != nil: + return fromConfigMap(envSource.ConfigMapRef) + case envSource.SecretRef != nil: + return fromSecret(envSource.SecretRef) + } +} + +func (w *envWrapper) secretValue(secretRef *corev1.SecretKeySelector, container *corev1.Container) (string, error) { + secretPlaintext := func(m map[string]string) (string, error) { + if v, ok := m[secretRef.Key]; ok { + return w.checkAndReplaceEnv(v, container) + } + return "", nil + } + secretCiphertext := func(m map[string][]byte) (string, error) { + if v, ok := m[secretRef.Key]; ok { + return decodeString(v) + } + return "", nil + } + + if w.cli == nil { + return "", cfgcore.MakeError("not support secret[%s] value in local mode, cli is nil", secretRef.Name) + } + + secretName, err := w.checkAndReplaceEnv(secretRef.Name, container) + if err != nil { + return "", err + } + secretKey := coreclient.ObjectKey{ + Name: secretName, + Namespace: w.namespace, + } + secret, err := getResourceObject(w, &corev1.Secret{}, secretKey) + if err != nil { + return "", err + } + if secret.StringData != nil { + return secretPlaintext(secret.StringData) + } + if secret.Data != nil { + return secretCiphertext(secret.Data) + } + return "", nil +} + +func (w *envWrapper) configMapValue(configmapRef *corev1.ConfigMapKeySelector, container *corev1.Container) (string, error) { + if w.cli == nil { + return "", cfgcore.MakeError("not support configmap[%s] value in local mode, cli is nil", configmapRef.Name) + } + + cmName, err := w.checkAndReplaceEnv(configmapRef.Name, container) + if err != nil { + return "", err + } + cmKey := coreclient.ObjectKey{ + Name: cmName, + Namespace: w.namespace, + } + cm, err := getResourceObject(w, &corev1.ConfigMap{}, cmKey) + if err != nil { + return "", err + } + return cm.Data[configmapRef.Key], nil +} + +func (w *envWrapper) getResourceFromLocal(key coreclient.ObjectKey, gvk schema.GroupVersionKind) coreclient.Object { + if _, ok := w.cache[gvk]; !ok { + w.cache[gvk] = make(map[coreclient.ObjectKey]coreclient.Object) + } + if v, ok := w.cache[gvk][key]; ok { + return v + } + if w.localObjects == nil { + return nil + } + return w.localObjects.GetLocalResourceWithObjectKey(key, gvk) +} + +var envPlaceHolderRegexp = regexp.MustCompile(`\$\(\w+\)`) + +func (w *envWrapper) checkAndReplaceEnv(value string, container *corev1.Container) (string, error) { + // env value replace,e.g: $(CONN_CREDENTIAL_SECRET_NAME), $(KB_CLUSTER_COMP_NAME) + // - name: KB_POD_FQDN + // value: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc + // + // - name: MYSQL_ROOT_USER + // valueFrom: + // secretKeyRef: + // key: username + // name: $(CONN_CREDENTIAL_SECRET_NAME) + // var := "$(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc" + // + // loop reference + // - name: LOOP_REF_A + // value: $(LOOP_REF_B) + // - name: LOOP_REF_B + // value: $(LOOP_REF_A) + + if len(value) == 0 || strings.IndexByte(value, '$') < 0 { + return value, nil + } + envHolderVec := envPlaceHolderRegexp.FindAllString(value, -1) + if len(envHolderVec) == 0 { + return value, nil + } + return w.doEnvReplace(set.NewLinkedHashSetString(envHolderVec...), value, container) +} + +func (w *envWrapper) doEnvReplace(replacedVars *set.LinkedHashSetString, oldValue string, container *corev1.Container) (string, error) { + var ( + clusterName = w.localObjects.Cluster.Name + componentName = w.localObjects.Component.Name + builtInEnvMap = component.GetReplacementMapForBuiltInEnv(clusterName, componentName) + ) + + builtInEnvMap[constant.ConnCredentialPlaceHolder] = component.GenerateConnCredential(w.localObjects.Cluster.Name) + kbInnerEnvReplaceFn := func(envName string, strToReplace string) string { + return strings.ReplaceAll(strToReplace, envName, builtInEnvMap[envName]) + } + + if !w.incAndCheckReferenceCount() { + return "", cfgcore.MakeError("too many reference count, maybe there is a loop reference: [%s] more than %d times ", oldValue, w.referenceCount) + } + + replacedValue := oldValue + for envHolder := range replacedVars.Iter() { + if len(envHolder) <= 3 { + continue + } + if _, ok := builtInEnvMap[envHolder]; ok { + replacedValue = kbInnerEnvReplaceFn(envHolder, replacedValue) + continue + } + envName := envHolder[2 : len(envHolder)-1] + envValue, err := w.getEnvByName(container, envName) + if err != nil { + w.decReferenceCount() + return envValue, err + } + replacedValue = strings.ReplaceAll(replacedValue, envHolder, envValue) + } + w.decReferenceCount() + return replacedValue, nil +} + +func (w *envWrapper) incReferenceCount() { + w.referenceCount++ +} + +func (w *envWrapper) decReferenceCount() { + w.referenceCount-- +} + +func (w *envWrapper) incAndCheckReferenceCount() bool { + w.incReferenceCount() + return w.referenceCount <= maxReferenceCount +} + +func getResourceObject[T generics.Object, PT generics.PObject[T]](w *envWrapper, obj PT, key coreclient.ObjectKey) (PT, error) { + gvk := generics.ToGVK(obj) + object := w.getResourceFromLocal(key, gvk) + if v, ok := object.(PT); ok { + return v, nil + } + if err := w.cli.Get(w.ctx, key, obj); err != nil { + return nil, err + } + w.cache[gvk][key] = obj + return obj, nil +} + +func decodeString(encoded []byte) (string, error) { + decoded, err := b64.StdEncoding.DecodeString(string(encoded)) + if err != nil { + return "", err + } + return string(decoded), nil +} + +func resourceRefValue(resourceRef *corev1.ResourceFieldSelector, containers []corev1.Container, curContainer *corev1.Container) (string, error) { + if resourceRef.ContainerName == "" { + return containerResourceRefValue(resourceRef, curContainer) + } + for _, v := range containers { + if v.Name == resourceRef.ContainerName { + return containerResourceRefValue(resourceRef, &v) + } + } + return "", cfgcore.MakeError("not found named[%s] container", resourceRef.ContainerName) +} + +func containerResourceRefValue(fieldSelector *corev1.ResourceFieldSelector, c *corev1.Container) (string, error) { + return resource.ExtractContainerResourceValue(fieldSelector, c) +} + +func fieldRefValue(podReference *corev1.ObjectFieldSelector, podSpec *corev1.PodSpec) (string, error) { + return "", cfgcore.MakeError("not support pod field ref") +} diff --git a/internal/controller/plan/builtin_env_test.go b/internal/controller/plan/builtin_env_test.go new file mode 100644 index 000000000..c4bf0c374 --- /dev/null +++ b/internal/controller/plan/builtin_env_test.go @@ -0,0 +1,278 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package plan + +import ( + "strconv" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + coreclient "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + ctrlcomp "github.com/apecloud/kubeblocks/internal/controller/component" + intctrltypes "github.com/apecloud/kubeblocks/internal/controller/types" + testutil "github.com/apecloud/kubeblocks/internal/testutil/k8s" +) + +var _ = Describe("tpl env template", func() { + + patroniTemplate := ` +bootstrap: + initdb: + - auth-host: md5 + - auth-local: trust +` + + var ( + podSpec *corev1.PodSpec + cfgTemplate []appsv1alpha1.ComponentConfigSpec + component *ctrlcomp.SynthesizedComponent + cluster *appsv1alpha1.Cluster + + mockClient *testutil.K8sClientMockHelper + ) + + BeforeEach(func() { + mockClient = testutil.NewK8sMockClient() + + mockClient.MockGetMethod(testutil.WithGetReturned(testutil.WithConstructSimpleGetResult([]coreclient.Object{ + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: "patroni-template-config", + Namespace: "default", + }, + Data: map[string]string{ + "postgresql.yaml": patroniTemplate, + }}, + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: "my-config-env", + Namespace: "default", + }, + Data: map[string]string{ + "KB_MYSQL_0_HOSTNAME": "my-mysql-0.my-mysql-headless", + "KB_MYSQL_FOLLOWERS": "", + "KB_MYSQL_LEADER": "my-mysql-0", + "KB_MYSQL_N": "1", + "KB_MYSQL_RECREATE": "false", + "LOOP_REFERENCE_A": "$(LOOP_REFERENCE_B)", + "LOOP_REFERENCE_B": "$(LOOP_REFERENCE_C)", + "LOOP_REFERENCE_C": "$(LOOP_REFERENCE_A)", + }}, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: "my-conn-credential", + Namespace: "default", + }, + Data: map[string][]byte{ + "password": []byte("NHpycWZsMnI="), + "username": []byte("cm9vdA=="), + }}, + }), testutil.WithAnyTimes())) + + // 2 configmap and 2 secret + // Add any setup steps that needs to be executed before each test + podSpec = &corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "mytest", + Env: []corev1.EnvVar{ + { + Name: "KB_CLUSTER_NAME", + Value: "my", + }, + { + Name: "KB_COMP_NAME", + Value: "mysql", + }, + { + Name: "MEMORY_SIZE", + ValueFrom: &corev1.EnvVarSource{ + ResourceFieldRef: &corev1.ResourceFieldSelector{ + ContainerName: "mytest", + Resource: "limits.memory", + }, + }, + }, + { + Name: "CPU", + ValueFrom: &corev1.EnvVarSource{ + ResourceFieldRef: &corev1.ResourceFieldSelector{ + Resource: "limits.cpu", + }, + }, + }, + { + Name: "CPU2", + ValueFrom: &corev1.EnvVarSource{ + ResourceFieldRef: &corev1.ResourceFieldSelector{ + ContainerName: "not_exist_container", + Resource: "limits.memory", + }, + }, + }, + { + Name: "MYSQL_USER", + ValueFrom: &corev1.EnvVarSource{ + SecretKeyRef: &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: "my-conn-credential", + }, + Key: "username", + }, + }, + }, + { + Name: "MYSQL_PASSWORD", + ValueFrom: &corev1.EnvVarSource{ + SecretKeyRef: &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: "$(CONN_CREDENTIAL_SECRET_NAME)", + }, + Key: "password", + }, + }, + }, + { + Name: "SPILO_CONFIGURATION", + ValueFrom: &corev1.EnvVarSource{ + ConfigMapKeyRef: &corev1.ConfigMapKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: "patroni-template-config", + }, + Key: "postgresql.yaml", + }, + }, + }, + }, + EnvFrom: []corev1.EnvFromSource{ + { + ConfigMapRef: &corev1.ConfigMapEnvSource{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: "my-config-env", + }, + }, + }, + { + SecretRef: &corev1.SecretEnvSource{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: "my-secret-env", + }, + }, + }, + }, + Resources: corev1.ResourceRequirements{ + Limits: map[corev1.ResourceName]resource.Quantity{ + corev1.ResourceMemory: resource.MustParse("8Gi"), + corev1.ResourceCPU: resource.MustParse("4"), + }, + }, + }, + { + Name: "invalid_contaienr", + }, + }, + } + component = &ctrlcomp.SynthesizedComponent{ + Name: "mysql", + } + cluster = &appsv1alpha1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "my", + }, + } + cfgTemplate = []appsv1alpha1.ComponentConfigSpec{{ + ComponentTemplateSpec: appsv1alpha1.ComponentTemplateSpec{ + Name: "mysql-config-8.0.2", + TemplateRef: "mysql-config-8.0.2", + VolumeName: "config1", + }, + ConfigConstraintRef: "mysql-config-8.0.2", + }} + }) + + AfterEach(func() { + mockClient.Finish() + }) + + // for test GetContainerWithVolumeMount + Context("ConfigTemplateBuilder built-in env test", func() { + It("test built-in function", func() { + cfgBuilder := newTemplateBuilder( + "my_test", + "default", + &appsv1alpha1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "my_test", + Namespace: "default", + }, + }, + nil, ctx, mockClient.Client(), + ) + + task := intctrltypes.InitReconcileTask(nil, nil, cluster, component) + task.AppendResource(&corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: "patroni-template-config", + Namespace: "default", + }, + Data: map[string]string{ + "postgresql.yaml": patroniTemplate, + }}) + Expect(cfgBuilder.injectBuiltInObjectsAndFunctions(podSpec, cfgTemplate, component, task)).Should(BeNil()) + + rendered, err := cfgBuilder.render(map[string]string{ + // KB_CLUSTER_NAME, KB_COMP_NAME from env + // MYSQL_USER,MYSQL_PASSWORD from valueFrom secret key + // SPILO_CONFIGURATION from valueFrom configmap key + // KB_MYSQL_LEADER from envFrom configmap + // MEMORY_SIZE, CPU from resourceFieldRef + "my": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"KB_CLUSTER_NAME\" }}", + "mysql": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"KB_COMP_NAME\" }}", + "root": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"MYSQL_USER\" }}", + "4zrqfl2r": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"MYSQL_PASSWORD\" }}", + patroniTemplate: "{{ getEnvByName ( index $.podSpec.containers 0 ) \"SPILO_CONFIGURATION\" }}", + "my-mysql-0": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"KB_MYSQL_LEADER\" }}", + + strconv.Itoa(4): "{{ getEnvByName ( index $.podSpec.containers 0 ) \"CPU\" }}", + strconv.Itoa(8 * 1024 * 1024 * 1024): "{{ getEnvByName ( index $.podSpec.containers 0 ) \"MEMORY_SIZE\" }}", + }) + + Expect(err).Should(Succeed()) + for key, value := range rendered { + Expect(key).Should(BeEquivalentTo(value)) + } + + _, err = cfgBuilder.render(map[string]string{ + "error": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"CPU2\" }}", + }) + Expect(err).ShouldNot(Succeed()) + Expect(err.Error()).Should(ContainSubstring("not found named[not_exist_container] container")) + + _, err = cfgBuilder.render(map[string]string{ + "error_loop_reference": "{{ getEnvByName ( index $.podSpec.containers 0 ) \"LOOP_REFERENCE_A\" }}", + }) + Expect(err).ShouldNot(Succeed()) + Expect(err.Error()).Should(ContainSubstring("too many reference count, maybe there is a loop reference")) + }) + }) +}) diff --git a/internal/controller/plan/builtin_functions.go b/internal/controller/plan/builtin_functions.go index 7d6e03f4e..183c1e393 100644 --- a/internal/controller/plan/builtin_functions.go +++ b/internal/controller/plan/builtin_functions.go @@ -202,20 +202,6 @@ func getPVCByName(args []interface{}, volumeName string) (interface{}, error) { return nil, nil } -// getEnvByName for general built-in -func getEnvByName(args interface{}, envName string) (string, error) { - container, err := fromJSONObject[corev1.Container](args) - if err != nil { - return "", err - } - for _, v := range container.Env { - if v.Name == envName { - return v.Value, nil - } - } - return "", nil -} - // getContainerMemory for general built-in func getContainerMemory(args interface{}) (int64, error) { container, err := fromJSONObject[corev1.Container](args) diff --git a/internal/controller/plan/config_template.go b/internal/controller/plan/config_template.go index ddf8a4af6..88376f5e9 100644 --- a/internal/controller/plan/config_template.go +++ b/internal/controller/plan/config_template.go @@ -26,6 +26,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" "github.com/apecloud/kubeblocks/internal/controller/client" "github.com/apecloud/kubeblocks/internal/controller/component" + intctrltypes "github.com/apecloud/kubeblocks/internal/controller/types" intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" "github.com/apecloud/kubeblocks/internal/gotemplate" ) @@ -157,23 +158,24 @@ func (c *configTemplateBuilder) builtinObjectsAsValues() (*gotemplate.TplValues, func (c *configTemplateBuilder) injectBuiltInObjectsAndFunctions( podSpec *corev1.PodSpec, configs []appsv1alpha1.ComponentConfigSpec, - component *component.SynthesizedComponent) error { + component *component.SynthesizedComponent, + task *intctrltypes.ReconcileTask) error { if err := c.injectBuiltInObjects(podSpec, component, configs); err != nil { return err } - if err := c.injectBuiltInFunctions(component); err != nil { + if err := c.injectBuiltInFunctions(component, task); err != nil { return err } return nil } -func (c *configTemplateBuilder) injectBuiltInFunctions(component *component.SynthesizedComponent) error { +func (c *configTemplateBuilder) injectBuiltInFunctions(component *component.SynthesizedComponent, task *intctrltypes.ReconcileTask) error { // TODO add built-in function c.builtInFunctions = &gotemplate.BuiltInObjectsFunc{ builtInMysqlCalBufferFunctionName: calDBPoolSize, builtInGetVolumeFunctionName: getVolumeMountPathByName, builtInGetPvcFunctionName: getPVCByName, - builtInGetEnvFunctionName: getEnvByName, + builtInGetEnvFunctionName: wrapGetEnvByName(c, task), builtInGetPortFunctionName: getPortByName, builtInGetArgFunctionName: getArgByName, builtInGetContainerFunctionName: getPodContainerByName, diff --git a/internal/controller/plan/config_template_test.go b/internal/controller/plan/config_template_test.go index 56a18b3be..eef8b270a 100644 --- a/internal/controller/plan/config_template_test.go +++ b/internal/controller/plan/config_template_test.go @@ -180,7 +180,7 @@ single_thread_memory = 294912 nil, nil, nil) Expect(cfgBuilder.injectBuiltInObjectsAndFunctions( - podSpec, cfgTemplate, component)).Should(BeNil()) + podSpec, cfgTemplate, component, nil)).Should(BeNil()) cfgBuilder.componentValues.Resource = &ResourceDefinition{ MemorySize: 8 * 1024 * 1024 * 1024, @@ -208,7 +208,7 @@ single_thread_memory = 294912 nil, nil, nil, ) - Expect(cfgBuilder.injectBuiltInObjectsAndFunctions(podSpec, cfgTemplate, component)).Should(BeNil()) + Expect(cfgBuilder.injectBuiltInObjectsAndFunctions(podSpec, cfgTemplate, component, nil)).Should(BeNil()) rendered, err := cfgBuilder.render(map[string]string{ "a": "{{ getVolumePathByName ( index $.podSpec.containers 0 ) \"log\" }}", @@ -269,7 +269,7 @@ single_thread_memory = 294912 nil, nil, nil, ) - Expect(cfgBuilder.injectBuiltInObjectsAndFunctions(podSpec, cfgTemplate, component)).Should(BeNil()) + Expect(cfgBuilder.injectBuiltInObjectsAndFunctions(podSpec, cfgTemplate, component, nil)).Should(BeNil()) tests := []struct { name string diff --git a/internal/controller/plan/prepare.go b/internal/controller/plan/prepare.go index 9b4c54fe8..b15c75436 100644 --- a/internal/controller/plan/prepare.go +++ b/internal/controller/plan/prepare.go @@ -313,7 +313,7 @@ func buildCfg(task *intctrltypes.ReconcileTask, // New ConfigTemplateBuilder templateBuilder := newTemplateBuilder(clusterName, namespaceName, task.Cluster, task.ClusterVersion, ctx, cli) // Prepare built-in objects and built-in functions - if err := templateBuilder.injectBuiltInObjectsAndFunctions(podSpec, task.Component.ConfigTemplates, task.Component); err != nil { + if err := templateBuilder.injectBuiltInObjectsAndFunctions(podSpec, task.Component.ConfigTemplates, task.Component, task); err != nil { return nil, err } From f761e7de6208516a5186171e0de69225341e6f1f Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Thu, 6 Apr 2023 15:14:31 +0800 Subject: [PATCH 50/80] feat: refactor reconfigure partIII: Secondary rendering of the config files by patching parameters to the config files on the pod. (#2160) (#2258) --- cmd/reloader/app/proxy.go | 2 +- internal/cli/cmd/cluster/operations.go | 10 +-- internal/configuration/config.go | 10 +++ .../configuration/config_manager/builder.go | 6 +- .../config_manager/builder_test.go | 6 +- .../configuration/config_manager/handler.go | 2 +- .../config_manager/handler_util.go | 3 +- .../config_manager/reload_util.go | 28 +++++++-- .../config_manager/reload_util_test.go | 2 +- internal/configuration/config_util.go | 46 ++++++++++---- internal/configuration/config_util_test.go | 62 +++++++++++++++++++ internal/controller/builder/builder.go | 2 +- internal/controller/builder/builder_test.go | 2 +- internal/controller/plan/prepare.go | 18 +++--- 14 files changed, 151 insertions(+), 48 deletions(-) diff --git a/cmd/reloader/app/proxy.go b/cmd/reloader/app/proxy.go index e1d677bb3..a93cf97d8 100644 --- a/cmd/reloader/app/proxy.go +++ b/cmd/reloader/app/proxy.go @@ -103,7 +103,7 @@ func (r *reconfigureProxy) initOnlineUpdater(opt *VolumeWatcherOpts) error { return nil } - updater, err := cfgcm.OnlineUpdateParamsHandle(opt.TPLScriptPath) + updater, err := cfgcm.OnlineUpdateParamsHandle(opt.TPLScriptPath, opt.FormatterConfig) if err != nil { return cfgcore.WrapError(err, "failed to create online updater") } diff --git a/internal/cli/cmd/cluster/operations.go b/internal/cli/cmd/cluster/operations.go index 4f493add6..1f2dc2ae4 100644 --- a/internal/cli/cmd/cluster/operations.go +++ b/internal/cli/cmd/cluster/operations.go @@ -200,14 +200,6 @@ func (o *OperationsOptions) validateReconfiguring() error { } func (o *OperationsOptions) validateConfigParams(tpl *appsv1alpha1.ComponentConfigSpec) error { - transKeyPair := func(pts map[string]string) map[string]interface{} { - m := make(map[string]interface{}, len(pts)) - for key, value := range pts { - m[key] = value - } - return m - } - configConstraintKey := client.ObjectKey{ Namespace: "", Name: tpl.ConfigConstraintRef, @@ -219,7 +211,7 @@ func (o *OperationsOptions) validateConfigParams(tpl *appsv1alpha1.ComponentConf newConfigData, err := cfgcore.MergeAndValidateConfigs(configConstraint.Spec, map[string]string{o.CfgFile: ""}, tpl.Keys, []cfgcore.ParamPairs{{ Key: o.CfgFile, - UpdatedParams: transKeyPair(o.KeyValues), + UpdatedParams: cfgcore.FromStringMap(o.KeyValues), }}) if err != nil { return err diff --git a/internal/configuration/config.go b/internal/configuration/config.go index b2510602c..cd50ee978 100644 --- a/internal/configuration/config.go +++ b/internal/configuration/config.go @@ -295,6 +295,16 @@ func NewCfgOptions(filename string, options ...Option) CfgOpOption { return context } +func WithFormatterConfig(formatConfig *appsv1alpha1.FormatterConfig) Option { + return func(ctx *CfgOpOption) { + if formatConfig.Format == appsv1alpha1.Ini && formatConfig.IniConfig != nil { + ctx.IniContext = &IniContext{ + SectionName: formatConfig.IniConfig.SectionName, + } + } + } +} + func (c *cfgWrapper) Query(jsonpath string, option CfgOpOption) ([]byte, error) { if option.AllSearch && c.fileCount > 1 { return c.queryAllCfg(jsonpath, option) diff --git a/internal/configuration/config_manager/builder.go b/internal/configuration/config_manager/builder.go index d0a3fe4a1..6800e8af9 100644 --- a/internal/configuration/config_manager/builder.go +++ b/internal/configuration/config_manager/builder.go @@ -40,7 +40,7 @@ const ( scriptVolumePath = "/opt/config/reload" ) -func BuildConfigManagerContainerArgs(reloadOptions *appsv1alpha1.ReloadOptions, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *ConfigManagerParams) error { +func BuildConfigManagerContainerArgs(reloadOptions *appsv1alpha1.ReloadOptions, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams) error { switch { case reloadOptions.UnixSignalTrigger != nil: manager.Args = buildSignalArgs(*reloadOptions.UnixSignalTrigger, volumeDirs) @@ -53,7 +53,7 @@ func BuildConfigManagerContainerArgs(reloadOptions *appsv1alpha1.ReloadOptions, return cfgutil.MakeError("not support reload.") } -func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *ConfigManagerParams) error { +func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams) error { scriptCMName := fmt.Sprintf("%s-%s", options.ScriptConfigMapRef, manager.Cluster.GetName()) if err := checkOrCreateScriptCM(options, client.ObjectKey{ Namespace: manager.Cluster.GetNamespace(), @@ -124,7 +124,7 @@ func checkOrCreateScriptCM(options *appsv1alpha1.TPLScriptTrigger, scriptCMKey c return nil } -func buildShellArgs(options appsv1alpha1.ShellTrigger, volumeDirs []corev1.VolumeMount, manager *ConfigManagerParams) error { +func buildShellArgs(options appsv1alpha1.ShellTrigger, volumeDirs []corev1.VolumeMount, manager *CfgManagerBuildParams) error { command := strings.Trim(options.Exec, " \t") if command == "" { return cfgutil.MakeError("invalid command: [%s]", options.Exec) diff --git a/internal/configuration/config_manager/builder_test.go b/internal/configuration/config_manager/builder_test.go index 158bae6e7..f974f6786 100644 --- a/internal/configuration/config_manager/builder_test.go +++ b/internal/configuration/config_manager/builder_test.go @@ -55,7 +55,7 @@ var _ = Describe("ConfigManager Test", func() { volumeDirs []corev1.VolumeMount cli client.Client ctx context.Context - param *ConfigManagerParams + param *CfgManagerBuildParams } tests := []struct { name string @@ -120,7 +120,7 @@ var _ = Describe("ConfigManager Test", func() { }}, cli: mockK8sCli.Client(), ctx: context.TODO(), - param: &ConfigManagerParams{ + param: &CfgManagerBuildParams{ Cluster: &appsv1alpha1.Cluster{ ObjectMeta: metav1.ObjectMeta{ Name: "abcd", @@ -139,7 +139,7 @@ var _ = Describe("ConfigManager Test", func() { for _, tt := range tests { param := tt.args.param if param == nil { - param = &ConfigManagerParams{} + param = &CfgManagerBuildParams{} } err := BuildConfigManagerContainerArgs(tt.args.reloadOptions, tt.args.volumeDirs, tt.args.cli, tt.args.ctx, param) Expect(err != nil).Should(BeEquivalentTo(tt.wantErr)) diff --git a/internal/configuration/config_manager/handler.go b/internal/configuration/config_manager/handler.go index d50156a7e..f9cfc19d3 100644 --- a/internal/configuration/config_manager/handler.go +++ b/internal/configuration/config_manager/handler.go @@ -153,7 +153,7 @@ func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, if err != nil { return err } - if err := wrapGoTemplateRun(tplScripts, string(tplContent), updatedParams); err != nil { + if err := wrapGoTemplateRun(tplScripts, string(tplContent), updatedParams, formatConfig); err != nil { return err } return backupLastConfigFiles(currFiles, backupPath) diff --git a/internal/configuration/config_manager/handler_util.go b/internal/configuration/config_manager/handler_util.go index 6481508ad..7dfd9dfa7 100644 --- a/internal/configuration/config_manager/handler_util.go +++ b/internal/configuration/config_manager/handler_util.go @@ -29,7 +29,8 @@ import ( cfgutil "github.com/apecloud/kubeblocks/internal/configuration" ) -type ConfigManagerParams struct { +// CfgManagerBuildParams is the params for building config manager sidecar +type CfgManagerBuildParams struct { ManagerName string `json:"name"` Image string `json:"sidecarImage"` Args []string `json:"args"` diff --git a/internal/configuration/config_manager/reload_util.go b/internal/configuration/config_manager/reload_util.go index a0ccefcc3..6812144dc 100644 --- a/internal/configuration/config_manager/reload_util.go +++ b/internal/configuration/config_manager/reload_util.go @@ -44,7 +44,7 @@ const ( type DynamicUpdater = func(updatedParams map[string]string) error -func OnlineUpdateParamsHandle(tplScript string) (DynamicUpdater, error) { +func OnlineUpdateParamsHandle(tplScript string, formatConfig *appsv1alpha1.FormatterConfig) (DynamicUpdater, error) { tplContent, err := os.ReadFile(tplScript) if err != nil { return nil, err @@ -53,7 +53,7 @@ func OnlineUpdateParamsHandle(tplScript string) (DynamicUpdater, error) { return nil, err } return func(updatedParams map[string]string) error { - return wrapGoTemplateRun(tplScript, string(tplContent), updatedParams) + return wrapGoTemplateRun(tplScript, string(tplContent), updatedParams, formatConfig) }, nil } @@ -64,7 +64,7 @@ func checkTPLScript(tplName string, tplContent string) error { return err } -func wrapGoTemplateRun(tplName string, tplContent string, updatedParams map[string]string) error { +func wrapGoTemplateRun(tplName string, tplContent string, updatedParams map[string]string, formatConfig *appsv1alpha1.FormatterConfig) error { var ( err error commandChannel DynamicParamUpdater @@ -77,12 +77,12 @@ func wrapGoTemplateRun(tplName string, tplContent string, updatedParams map[stri logger.Info(fmt.Sprintf("update global dynamic params: %v", updatedParams)) values := gotemplate.ConstructFunctionArgList(updatedParams) - engine := gotemplate.NewTplEngine(&values, constructReloadBuiltinFuncs(commandChannel), tplName, nil, nil) + engine := gotemplate.NewTplEngine(&values, constructReloadBuiltinFuncs(commandChannel, formatConfig), tplName, nil, nil) _, err = engine.Render(tplContent) return err } -func constructReloadBuiltinFuncs(cc DynamicParamUpdater) *gotemplate.BuiltInObjectsFunc { +func constructReloadBuiltinFuncs(cc DynamicParamUpdater, formatConfig *appsv1alpha1.FormatterConfig) *gotemplate.BuiltInObjectsFunc { return &gotemplate.BuiltInObjectsFunc{ builtInExecFunctionName: func(command string, args ...string) (string, error) { execCommand := exec.Command(command, args...) @@ -95,6 +95,24 @@ func constructReloadBuiltinFuncs(cc DynamicParamUpdater) *gotemplate.BuiltInObje logger.V(1).Info(fmt.Sprintf("sql: [%s], result: [%v]", sql, r)) return err }, + builtInParamsPatchFunctionName: func(updatedParams map[string]string, basefile, newfile string) error { + logger.V(1).Info(fmt.Sprintf("update params: %v, basefile: %s, newfile: %s", updatedParams, basefile, newfile)) + if len(updatedParams) == 0 { + if basefile == newfile { + return nil + } + return copyFileContents(basefile, newfile) + } + b, err := os.ReadFile(basefile) + if err != nil { + return err + } + newConfig, err := cfgutil.ApplyConfigPatch(b, updatedParams, formatConfig) + if err != nil { + return err + } + return os.WriteFile(newfile, []byte(newConfig), os.ModePerm) + }, } } diff --git a/internal/configuration/config_manager/reload_util_test.go b/internal/configuration/config_manager/reload_util_test.go index c77d90937..70b32ffbe 100644 --- a/internal/configuration/config_manager/reload_util_test.go +++ b/internal/configuration/config_manager/reload_util_test.go @@ -79,7 +79,7 @@ func TestCreateUpdatedParamsPatch(t *testing.T) { } func TestConstructReloadBuiltinFuncs(t *testing.T) { - require.NotNil(t, constructReloadBuiltinFuncs(nil)) + require.NotNil(t, constructReloadBuiltinFuncs(nil, nil)) } func prepareTestData(t *testing.T, dir1 string, dir2 string) string { diff --git a/internal/configuration/config_util.go b/internal/configuration/config_util.go index b93c9bfbf..a46e17d36 100644 --- a/internal/configuration/config_util.go +++ b/internal/configuration/config_util.go @@ -42,29 +42,19 @@ func MergeAndValidateConfigs(configConstraint appsv1alpha1.ConfigConstraintSpec, ) cmKeySet := FromCMKeysSelector(cmKey) - configOption := CfgOption{ + configLoaderOption := CfgOption{ Type: CfgCmType, Log: log.FromContext(context.TODO()), CfgType: fc.Format, ConfigResource: FromConfigData(baseConfigs, cmKeySet), } - if configOperator, err = NewConfigLoader(configOption); err != nil { + if configOperator, err = NewConfigLoader(configLoaderOption); err != nil { return nil, err } - // process special formatter options - mergedOptions := func(ctx *CfgOpOption) { - // process special formatter - if fc.Format == appsv1alpha1.Ini && fc.IniConfig != nil { - ctx.IniContext = &IniContext{ - SectionName: fc.IniConfig.SectionName, - } - } - } - // merge param to config file for _, params := range updatedParams { - if err := configOperator.MergeFrom(params.UpdatedParams, NewCfgOptions(params.Key, mergedOptions)); err != nil { + if err := configOperator.MergeFrom(params.UpdatedParams, NewCfgOptions(params.Key, WithFormatterConfig(fc))); err != nil { return nil, err } updatedKeys.Add(params.Key) @@ -112,3 +102,33 @@ func fromUpdatedConfig(m map[string]string, sets *set.LinkedHashSetString) map[s } return r } + +// FromStringMap converts a map[string]string to a map[string]interface{} +func FromStringMap(m map[string]string) map[string]interface{} { + r := make(map[string]interface{}, len(m)) + for key, v := range m { + r[key] = v + } + return r +} + +func ApplyConfigPatch(baseCfg []byte, updatedParameters map[string]string, formatConfig *appsv1alpha1.FormatterConfig) (string, error) { + configLoaderOption := CfgOption{ + Type: CfgRawType, + Log: log.FromContext(context.TODO()), + CfgType: formatConfig.Format, + RawData: baseCfg, + } + configWrapper, err := NewConfigLoader(configLoaderOption) + if err != nil { + return "", err + } + + mergedOptions := NewCfgOptions("", WithFormatterConfig(formatConfig)) + err = configWrapper.MergeFrom(FromStringMap(updatedParameters), mergedOptions) + if err != nil { + return "", err + } + mergedConfig := configWrapper.getConfigObject(mergedOptions) + return mergedConfig.Marshal() +} diff --git a/internal/configuration/config_util_test.go b/internal/configuration/config_util_test.go index 8d73336da..723ca45ba 100644 --- a/internal/configuration/config_util_test.go +++ b/internal/configuration/config_util_test.go @@ -236,3 +236,65 @@ func TestMergeUpdatedConfig(t *testing.T) { }) } } + +func TestApplyConfigPatch(t *testing.T) { + type args struct { + baseCfg []byte + updatedParameters map[string]string + formatConfig *v1alpha1.FormatterConfig + } + tests := []struct { + name string + args args + want string + wantErr bool + }{{ + name: "normal_test", + args: args{ + baseCfg: []byte(`[test] +test=test`), + updatedParameters: map[string]string{ + "a": "b", + "max_connections": "600", + }, + formatConfig: &v1alpha1.FormatterConfig{ + Format: v1alpha1.Ini, + FormatterOptions: v1alpha1.FormatterOptions{ + IniConfig: &v1alpha1.IniConfig{ + SectionName: "test", + }}}, + }, + want: `[test] +a=b +max_connections=600 +test=test +`, + wantErr: false, + }, { + name: "normal_test", + args: args{ + baseCfg: []byte(` `), + updatedParameters: map[string]string{ + "a": "b", + "c": "d e f g", + }, + formatConfig: &v1alpha1.FormatterConfig{ + Format: v1alpha1.RedisCfg, + }, + }, + want: "a b\nc d e f g", + wantErr: false, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ApplyConfigPatch(tt.args.baseCfg, tt.args.updatedParameters, tt.args.formatConfig) + if (err != nil) != tt.wantErr { + t.Errorf("ApplyConfigPatch() error = %v, wantErr %v", err, tt.wantErr) + return + } + if got != tt.want { + t.Errorf("ApplyConfigPatch() got = %v, want %v", got, tt.want) + } + }) + } +} diff --git a/internal/controller/builder/builder.go b/internal/controller/builder/builder.go index 7e73e90bd..cf2fdde97 100644 --- a/internal/controller/builder/builder.go +++ b/internal/controller/builder/builder.go @@ -660,7 +660,7 @@ func BuildConfigMapWithTemplate( return &cm, nil } -func BuildCfgManagerContainer(sidecarRenderedParam *cfgcm.ConfigManagerParams) (*corev1.Container, error) { +func BuildCfgManagerContainer(sidecarRenderedParam *cfgcm.CfgManagerBuildParams) (*corev1.Container, error) { const tplFile = "config_manager_sidecar.cue" cueFS, _ := debme.FS(cueTemplates, "cue") cueTpl, err := getCacheCUETplValue(tplFile, func() (*intctrlutil.CUETpl, error) { diff --git a/internal/controller/builder/builder_test.go b/internal/controller/builder/builder_test.go index 065dc55fb..a55ebaf9d 100644 --- a/internal/controller/builder/builder_test.go +++ b/internal/controller/builder/builder_test.go @@ -468,7 +468,7 @@ var _ = Describe("builder", func() { }) It("builds config manager sidecar container correctly", func() { - sidecarRenderedParam := &cfgcm.ConfigManagerParams{ + sidecarRenderedParam := &cfgcm.CfgManagerBuildParams{ ManagerName: "cfgmgr", CharacterType: "mysql", SecreteName: "test-secret", diff --git a/internal/controller/plan/prepare.go b/internal/controller/plan/prepare.go index b15c75436..e9eb6ca61 100644 --- a/internal/controller/plan/prepare.go +++ b/internal/controller/plan/prepare.go @@ -372,25 +372,25 @@ func updateConfigManagerWithComponent(podSpec *corev1.PodSpec, cfgTemplates []ap var ( err error - volumeDirs []corev1.VolumeMount - configManagerParams *cfgcm.ConfigManagerParams + volumeDirs []corev1.VolumeMount + buildParams *cfgcm.CfgManagerBuildParams ) if volumeDirs = getUsingVolumesByCfgTemplates(podSpec, cfgTemplates); len(volumeDirs) == 0 { return nil } - if configManagerParams, err = buildConfigManagerParams(cli, ctx, cfgTemplates, volumeDirs, params); err != nil { + if buildParams, err = buildConfigManagerParams(cli, ctx, cfgTemplates, volumeDirs, params); err != nil { return err } - if configManagerParams == nil { + if buildParams == nil { return nil } - container, err := builder.BuildCfgManagerContainer(configManagerParams) + container, err := builder.BuildCfgManagerContainer(buildParams) if err != nil { return err } - updateTPLScriptVolume(podSpec, configManagerParams) + updateTPLScriptVolume(podSpec, buildParams) // Add sidecar to podTemplate podSpec.Containers = append(podSpec.Containers, *container) @@ -400,7 +400,7 @@ func updateConfigManagerWithComponent(podSpec *corev1.PodSpec, cfgTemplates []ap return nil } -func updateTPLScriptVolume(podSpec *corev1.PodSpec, configManager *cfgcm.ConfigManagerParams) { +func updateTPLScriptVolume(podSpec *corev1.PodSpec, configManager *cfgcm.CfgManagerBuildParams) { scriptVolume := configManager.ScriptVolume if scriptVolume == nil { return @@ -451,8 +451,8 @@ func getUsingVolumesByCfgTemplates(podSpec *corev1.PodSpec, cfgTemplates []appsv return volumeDirs } -func buildConfigManagerParams(cli client.Client, ctx context.Context, cfgTemplates []appsv1alpha1.ComponentConfigSpec, volumeDirs []corev1.VolumeMount, params builder.BuilderParams) (*cfgcm.ConfigManagerParams, error) { - configManagerParams := &cfgcm.ConfigManagerParams{ +func buildConfigManagerParams(cli client.Client, ctx context.Context, cfgTemplates []appsv1alpha1.ComponentConfigSpec, volumeDirs []corev1.VolumeMount, params builder.BuilderParams) (*cfgcm.CfgManagerBuildParams, error) { + configManagerParams := &cfgcm.CfgManagerBuildParams{ ManagerName: constant.ConfigSidecarName, CharacterType: params.Component.CharacterType, SecreteName: component.GenerateConnCredential(params.Cluster.Name), From ff37d4321ed495bf8900729bc4ad72fbb2c58fb0 Mon Sep 17 00:00:00 2001 From: chantu Date: Thu, 6 Apr 2023 15:15:35 +0800 Subject: [PATCH 51/80] fix: pg remove metrics service (#2392) --- deploy/postgresql/templates/clusterdefinition.yaml | 3 --- 1 file changed, 3 deletions(-) diff --git a/deploy/postgresql/templates/clusterdefinition.yaml b/deploy/postgresql/templates/clusterdefinition.yaml index aa0cd001d..26572f2a8 100644 --- a/deploy/postgresql/templates/clusterdefinition.yaml +++ b/deploy/postgresql/templates/clusterdefinition.yaml @@ -55,9 +55,6 @@ spec: - name: tcp-postgresql port: 5432 targetPort: tcp-postgresql - - name: http-metrics-postgresql - port: 9187 - targetPort: http-metrics volumeTypes: - name: data type: data From fe2fd37c60eff759eb9f97e2aa676ade901164bd Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Thu, 6 Apr 2023 16:48:37 +0800 Subject: [PATCH 52/80] feat: refactor reconfigure partIV: Support script to mount to sidecar(configmanager) with configmap (#2160) (#2279) --- apis/apps/v1alpha1/configconstraint_types.go | 11 ++ .../apps.kubeblocks.io_configconstraints.yaml | 11 ++ controllers/apps/configuration/config_util.go | 15 +-- .../apps/configuration/config_util_test.go | 2 +- .../apps.kubeblocks.io_configconstraints.yaml | 11 ++ .../configuration/config_manager/builder.go | 118 ++++++++++++++---- .../config_manager/builder_test.go | 110 +++++++++++++++- internal/controller/plan/prepare.go | 16 ++- 8 files changed, 252 insertions(+), 42 deletions(-) diff --git a/apis/apps/v1alpha1/configconstraint_types.go b/apis/apps/v1alpha1/configconstraint_types.go index 97efcd418..e888b977b 100644 --- a/apis/apps/v1alpha1/configconstraint_types.go +++ b/apis/apps/v1alpha1/configconstraint_types.go @@ -123,6 +123,17 @@ type ShellTrigger struct { // exec used to execute for reload. // +kubebuilder:validation:Required Exec string `json:"exec"` + + // scriptConfigMapRef used to execute for reload. + // +kubebuilder:validation:Required + ScriptConfigMapRef string `json:"scriptConfigMapRef"` + + // Specify the namespace of the referenced the tpl script ConfigMap object. + // An empty namespace is equivalent to the "default" namespace. + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:default="default" + // +optional + Namespace string `json:"namespace,omitempty"` } type TPLScriptTrigger struct { diff --git a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml index ee9f6ff5b..474796337 100644 --- a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml +++ b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml @@ -133,8 +133,19 @@ spec: exec: description: exec used to execute for reload. type: string + namespace: + default: default + description: Specify the namespace of the referenced the tpl + script ConfigMap object. An empty namespace is equivalent + to the "default" namespace. + maxLength: 63 + type: string + scriptConfigMapRef: + description: scriptConfigMapRef used to execute for reload. + type: string required: - exec + - scriptConfigMapRef type: object tplScriptTrigger: description: goTplTrigger performs the reload command. diff --git a/controllers/apps/configuration/config_util.go b/controllers/apps/configuration/config_util.go index 6d64ae597..1f7bd6ec3 100644 --- a/controllers/apps/configuration/config_util.go +++ b/controllers/apps/configuration/config_util.go @@ -433,23 +433,24 @@ func NeedReloadVolume(config appsv1alpha1.ComponentConfigSpec) bool { return config.ConfigConstraintRef != "" } -func GetReloadOptions(cli client.Client, ctx context.Context, configSpecs []appsv1alpha1.ComponentConfigSpec) (*appsv1alpha1.ReloadOptions, error) { +func GetReloadOptions(cli client.Client, ctx context.Context, configSpecs []appsv1alpha1.ComponentConfigSpec) (*appsv1alpha1.ReloadOptions, *appsv1alpha1.FormatterConfig, error) { for _, configSpec := range configSpecs { if !NeedReloadVolume(configSpec) { continue } - cfgConst := &appsv1alpha1.ConfigConstraint{} - if err := cli.Get(ctx, client.ObjectKey{ + ccKey := client.ObjectKey{ Namespace: "", Name: configSpec.ConfigConstraintRef, - }, cfgConst); err != nil { - return nil, cfgcore.WrapError(err, "failed to get ConfigConstraint, key[%v]", configSpec) + } + cfgConst := &appsv1alpha1.ConfigConstraint{} + if err := cli.Get(ctx, ccKey, cfgConst); err != nil { + return nil, nil, cfgcore.WrapError(err, "failed to get ConfigConstraint, key[%v]", ccKey) } if cfgConst.Spec.ReloadOptions != nil { - return cfgConst.Spec.ReloadOptions, nil + return cfgConst.Spec.ReloadOptions, cfgConst.Spec.FormatterConfig, nil } } - return nil, nil + return nil, nil, nil } func getComponentFromClusterDefinition( diff --git a/controllers/apps/configuration/config_util_test.go b/controllers/apps/configuration/config_util_test.go index 68e0c578b..33868a45f 100644 --- a/controllers/apps/configuration/config_util_test.go +++ b/controllers/apps/configuration/config_util_test.go @@ -363,7 +363,7 @@ var _ = Describe("ConfigWrapper util test", func() { }, testutil.WithMaxTimes(len(tests)))) for _, tt := range tests { - got, err := GetReloadOptions(k8sMockClient.Client(), ctx, tt.tpls) + got, _, err := GetReloadOptions(k8sMockClient.Client(), ctx, tt.tpls) Expect(err != nil).Should(BeEquivalentTo(tt.wantErr)) Expect(reflect.DeepEqual(got, tt.want)).Should(BeTrue()) } diff --git a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml index ee9f6ff5b..474796337 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml @@ -133,8 +133,19 @@ spec: exec: description: exec used to execute for reload. type: string + namespace: + default: default + description: Specify the namespace of the referenced the tpl + script ConfigMap object. An empty namespace is equivalent + to the "default" namespace. + maxLength: 63 + type: string + scriptConfigMapRef: + description: scriptConfigMapRef used to execute for reload. + type: string required: - exec + - scriptConfigMapRef type: object tplScriptTrigger: description: goTplTrigger performs the reload command. diff --git a/internal/configuration/config_manager/builder.go b/internal/configuration/config_manager/builder.go index 6800e8af9..1a56ce26f 100644 --- a/internal/configuration/config_manager/builder.go +++ b/internal/configuration/config_manager/builder.go @@ -23,8 +23,11 @@ import ( "strings" "github.com/spf13/viper" + "gopkg.in/yaml.v2" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + apiruntime "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" @@ -34,31 +37,44 @@ import ( ) const ( - scriptName = "reload.tpl" - configTemplateName = "reload.yaml" - scriptVolumeName = "reload-manager-reload" - scriptVolumePath = "/opt/config/reload" + configTemplateName = "reload.yaml" + scriptVolumeName = "reload-manager-reload" + scriptVolumePath = "/opt/config/reload" + scriptConfigField = "scripts" + formatterConfigField = "formatterConfig" ) -func BuildConfigManagerContainerArgs(reloadOptions *appsv1alpha1.ReloadOptions, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams) error { +func BuildConfigManagerContainerArgs(reloadOptions *appsv1alpha1.ReloadOptions, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams, formatterConfig *appsv1alpha1.FormatterConfig) error { switch { case reloadOptions.UnixSignalTrigger != nil: manager.Args = buildSignalArgs(*reloadOptions.UnixSignalTrigger, volumeDirs) return nil case reloadOptions.ShellTrigger != nil: - return buildShellArgs(*reloadOptions.ShellTrigger, volumeDirs, manager) + return buildShellArgs(*reloadOptions.ShellTrigger, volumeDirs, manager, cli, ctx) case reloadOptions.TPLScriptTrigger != nil: - return buildTPLScriptArgs(reloadOptions.TPLScriptTrigger, volumeDirs, cli, ctx, manager) + return buildTPLScriptArgs(reloadOptions.TPLScriptTrigger, volumeDirs, cli, ctx, manager, formatterConfig) } return cfgutil.MakeError("not support reload.") } -func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams) error { - scriptCMName := fmt.Sprintf("%s-%s", options.ScriptConfigMapRef, manager.Cluster.GetName()) - if err := checkOrCreateScriptCM(options, client.ObjectKey{ +func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []corev1.VolumeMount, cli client.Client, ctx context.Context, manager *CfgManagerBuildParams, formatterConfig *appsv1alpha1.FormatterConfig) error { + reloadYamlFn := func(cm *corev1.ConfigMap) error { + newData, err := checkAndUpdateReloadYaml(cm.Data, configTemplateName, formatterConfig) + if err != nil { + return err + } + cm.Data = newData + return nil + } + referenceCMKey := client.ObjectKey{ + Namespace: options.Namespace, + Name: options.ScriptConfigMapRef, + } + scriptCMKey := client.ObjectKey{ Namespace: manager.Cluster.GetNamespace(), - Name: scriptCMName, - }, cli, ctx, manager.Cluster); err != nil { + Name: fmt.Sprintf("%s-%s", options.ScriptConfigMapRef, manager.Cluster.GetName()), + } + if err := checkOrCreateScriptCM(referenceCMKey, scriptCMKey, cli, ctx, manager.Cluster, reloadYamlFn); err != nil { return err } @@ -70,6 +86,12 @@ func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []cor args = append(args, "--notify-type", string(appsv1alpha1.TPLScriptType)) args = append(args, "--tpl-config", filepath.Join(scriptVolumePath, configTemplateName)) manager.Args = args + + buildReloadScriptVolume(scriptCMKey.Name, manager) + return nil +} + +func buildReloadScriptVolume(scriptCMName string, manager *CfgManagerBuildParams) { manager.Volumes = append(manager.Volumes, corev1.VolumeMount{ Name: scriptVolumeName, MountPath: scriptVolumePath, @@ -82,38 +104,33 @@ func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []cor }, }, } - return nil } -func checkOrCreateScriptCM(options *appsv1alpha1.TPLScriptTrigger, scriptCMKey client.ObjectKey, cli client.Client, ctx context.Context, cluster *appsv1alpha1.Cluster) error { +func checkOrCreateScriptCM(referenceCM client.ObjectKey, scriptCMKey client.ObjectKey, cli client.Client, ctx context.Context, cluster *appsv1alpha1.Cluster, fn func(cm *corev1.ConfigMap) error) error { var ( err error - ccCM = corev1.ConfigMap{} + refCM = corev1.ConfigMap{} sidecarCM = corev1.ConfigMap{} ) - if err = cli.Get(ctx, client.ObjectKey{ - Namespace: options.Namespace, - Name: options.ScriptConfigMapRef, - }, &ccCM); err != nil { + if err = cli.Get(ctx, referenceCM, &refCM); err != nil { return err } - if _, ok := ccCM.Data[scriptName]; !ok { - return cfgutil.MakeError("configmap not exist script: %s", scriptName) - } - if err = cli.Get(ctx, scriptCMKey, &sidecarCM); err != nil { if !apierrors.IsNotFound(err) { return err } scheme, _ := appsv1alpha1.SchemeBuilder.Build() - sidecarCM.Data = ccCM.Data - sidecarCM.SetLabels(ccCM.GetLabels()) + sidecarCM.Data = refCM.Data + if fn != nil && fn(&sidecarCM) != nil { + return err + } + sidecarCM.SetLabels(refCM.GetLabels()) sidecarCM.SetName(scriptCMKey.Name) sidecarCM.SetNamespace(scriptCMKey.Namespace) - sidecarCM.SetLabels(ccCM.Labels) + sidecarCM.SetLabels(refCM.Labels) if err := controllerutil.SetOwnerReference(cluster, &sidecarCM, scheme); err != nil { return err } @@ -124,7 +141,33 @@ func checkOrCreateScriptCM(options *appsv1alpha1.TPLScriptTrigger, scriptCMKey c return nil } -func buildShellArgs(options appsv1alpha1.ShellTrigger, volumeDirs []corev1.VolumeMount, manager *CfgManagerBuildParams) error { +func checkAndUpdateReloadYaml(data map[string]string, reloadConfig string, formatterConfig *appsv1alpha1.FormatterConfig) (map[string]string, error) { + configObject := make(map[string]interface{}) + if content, ok := data[reloadConfig]; ok { + if err := yaml.Unmarshal([]byte(content), &configObject); err != nil { + return nil, err + } + } + if res, _, _ := unstructured.NestedFieldNoCopy(configObject, scriptConfigField); res == nil { + return nil, cfgutil.MakeError("reload.yaml required field: %s", scriptConfigField) + } + + formatObject, err := apiruntime.DefaultUnstructuredConverter.ToUnstructured(formatterConfig) + if err != nil { + return nil, err + } + if err := unstructured.SetNestedField(configObject, formatObject, formatterConfigField); err != nil { + return nil, err + } + b, err := yaml.Marshal(configObject) + if err != nil { + return nil, err + } + data[reloadConfig] = string(b) + return data, nil +} + +func buildShellArgs(options appsv1alpha1.ShellTrigger, volumeDirs []corev1.VolumeMount, manager *CfgManagerBuildParams, cli client.Client, ctx context.Context) error { command := strings.Trim(options.Exec, " \t") if command == "" { return cfgutil.MakeError("invalid command: [%s]", options.Exec) @@ -133,6 +176,27 @@ func buildShellArgs(options appsv1alpha1.ShellTrigger, volumeDirs []corev1.Volum args = append(args, "--notify-type", string(appsv1alpha1.ShellType)) args = append(args, "---command", command) manager.Args = args + + if options.ScriptConfigMapRef == "" { + return nil + } + + return buildShellScriptCM(options, manager, cli, ctx) +} + +func buildShellScriptCM(options appsv1alpha1.ShellTrigger, manager *CfgManagerBuildParams, cli client.Client, ctx context.Context) error { + referenceCMKey := client.ObjectKey{ + Namespace: options.Namespace, + Name: options.ScriptConfigMapRef, + } + scriptsCMKey := client.ObjectKey{ + Namespace: manager.Cluster.GetNamespace(), + Name: fmt.Sprintf("%s-%s", options.ScriptConfigMapRef, manager.Cluster.GetName()), + } + if err := checkOrCreateScriptCM(referenceCMKey, scriptsCMKey, cli, ctx, manager.Cluster, nil); err != nil { + return err + } + buildReloadScriptVolume(scriptsCMKey.Name, manager) return nil } diff --git a/internal/configuration/config_manager/builder_test.go b/internal/configuration/config_manager/builder_test.go index f974f6786..edde3b3f1 100644 --- a/internal/configuration/config_manager/builder_test.go +++ b/internal/configuration/config_manager/builder_test.go @@ -18,10 +18,14 @@ package configmanager import ( "context" + "reflect" + "testing" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" + "github.com/stretchr/testify/assert" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/yaml" corev1 "k8s.io/api/core/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -105,6 +109,36 @@ var _ = Describe("ConfigManager Test", func() { `--volume-dir`, `/postgresql/conf`, `---command`, `pwd`, }, + }, { + name: "buildCfgContainerParams", + args: args{ + reloadOptions: &appsv1alpha1.ReloadOptions{ + ShellTrigger: &appsv1alpha1.ShellTrigger{ + Exec: "pwd", + Namespace: "default", + ScriptConfigMapRef: "script_cm", + }}, + volumeDirs: []corev1.VolumeMount{ + { + MountPath: "/postgresql/conf", + Name: "pg_config", + }}, + cli: mockK8sCli.Client(), + ctx: context.TODO(), + param: &CfgManagerBuildParams{ + Cluster: &appsv1alpha1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "abcd", + Namespace: "default", + }, + }, + }, + }, + expectedArgs: []string{ + `--notify-type`, `exec`, + `--volume-dir`, `/postgresql/conf`, + `---command`, `pwd`, + }, }, { name: "buildCfgContainerParams", args: args{ @@ -141,7 +175,7 @@ var _ = Describe("ConfigManager Test", func() { if param == nil { param = &CfgManagerBuildParams{} } - err := BuildConfigManagerContainerArgs(tt.args.reloadOptions, tt.args.volumeDirs, tt.args.cli, tt.args.ctx, param) + err := BuildConfigManagerContainerArgs(tt.args.reloadOptions, tt.args.volumeDirs, tt.args.cli, tt.args.ctx, param, nil) Expect(err != nil).Should(BeEquivalentTo(tt.wantErr)) if !tt.wantErr { for _, arg := range tt.expectedArgs { @@ -153,3 +187,77 @@ var _ = Describe("ConfigManager Test", func() { }) }) + +func TestCheckAndUpdateReloadYaml(t *testing.T) { + customEqual := func(l, r map[string]string) bool { + if len(l) != len(r) { + return false + } + var err error + for k, v := range l { + var lv any + var rv any + err = yaml.Unmarshal([]byte(v), &lv) + assert.Nil(t, err) + err = yaml.Unmarshal([]byte(r[k]), &rv) + assert.Nil(t, err) + if !reflect.DeepEqual(lv, rv) { + return false + } + } + return true + } + + type args struct { + data map[string]string + reloadConfig string + formatterConfig *appsv1alpha1.FormatterConfig + } + tests := []struct { + name string + args args + want map[string]string + wantErr bool + }{{ + name: "testCheckAndUpdateReloadYaml", + args: args{ + data: map[string]string{"reload.yaml": ` +fileRegex: my.cnf +scripts: reload.tpl +`}, + reloadConfig: "reload.yaml", + formatterConfig: &appsv1alpha1.FormatterConfig{ + Format: appsv1alpha1.Ini, + }, + }, + wantErr: false, + want: map[string]string{"reload.yaml": ` +scripts: reload.tpl +fileRegex: my.cnf +formatterConfig: + format: ini +`, + }, + }, { + name: "testCheckAndUpdateReloadYaml", + args: args{ + data: map[string]string{}, + reloadConfig: "reload.yaml", + formatterConfig: &appsv1alpha1.FormatterConfig{Format: appsv1alpha1.Ini}, + }, + wantErr: true, + want: map[string]string{}, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := checkAndUpdateReloadYaml(tt.args.data, tt.args.reloadConfig, tt.args.formatterConfig) + if (err != nil) != tt.wantErr { + t.Errorf("checkAndUpdateReloadYaml() error = %v, wantErr %v", err, tt.wantErr) + return + } + if !customEqual(got, tt.want) { + t.Errorf("checkAndUpdateReloadYaml() got = %v, want %v", got, tt.want) + } + }) + } +} diff --git a/internal/controller/plan/prepare.go b/internal/controller/plan/prepare.go index e9eb6ca61..04efdfdcd 100644 --- a/internal/controller/plan/prepare.go +++ b/internal/controller/plan/prepare.go @@ -451,7 +451,13 @@ func getUsingVolumesByCfgTemplates(podSpec *corev1.PodSpec, cfgTemplates []appsv return volumeDirs } -func buildConfigManagerParams(cli client.Client, ctx context.Context, cfgTemplates []appsv1alpha1.ComponentConfigSpec, volumeDirs []corev1.VolumeMount, params builder.BuilderParams) (*cfgcm.CfgManagerBuildParams, error) { +func buildConfigManagerParams(cli client.Client, ctx context.Context, configSpec []appsv1alpha1.ComponentConfigSpec, volumeDirs []corev1.VolumeMount, params builder.BuilderParams) (*cfgcm.CfgManagerBuildParams, error) { + var ( + err error + reloadOptions *appsv1alpha1.ReloadOptions + formatterConfig *appsv1alpha1.FormatterConfig + ) + configManagerParams := &cfgcm.CfgManagerBuildParams{ ManagerName: constant.ConfigSidecarName, CharacterType: params.Component.CharacterType, @@ -461,15 +467,13 @@ func buildConfigManagerParams(cli client.Client, ctx context.Context, cfgTemplat Cluster: params.Cluster, } - var err error - var reloadOptions *appsv1alpha1.ReloadOptions - if reloadOptions, err = cfgutil.GetReloadOptions(cli, ctx, cfgTemplates); err != nil { + if reloadOptions, formatterConfig, err = cfgutil.GetReloadOptions(cli, ctx, configSpec); err != nil { return nil, err } - if reloadOptions == nil { + if reloadOptions == nil || formatterConfig == nil { return nil, nil } - if err = cfgcm.BuildConfigManagerContainerArgs(reloadOptions, volumeDirs, cli, ctx, configManagerParams); err != nil { + if err = cfgcm.BuildConfigManagerContainerArgs(reloadOptions, volumeDirs, cli, ctx, configManagerParams, formatterConfig); err != nil { return nil, err } return configManagerParams, nil From bb7603e89fe0ebaeee24842bc88ba7e658398aa8 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Thu, 6 Apr 2023 20:24:16 +0800 Subject: [PATCH 53/80] feat: refactor reconfigure partV: Support pg patroni ha (#2160) (#2308) --- apis/apps/v1alpha1/configconstraint_types.go | 4 + cmd/reloader/app/cmd.go | 6 +- cmd/reloader/app/flags.go | 2 + cmd/reloader/app/proxy.go | 6 +- .../apps.kubeblocks.io_configconstraints.yaml | 45 +++++++ controllers/apps/configuration/policy_util.go | 36 +++-- .../apps/configuration/policy_util_test.go | 1 + .../apps/configuration/sync_upgrade_policy.go | 34 ++++- .../configuration/sync_upgrade_policy_test.go | 66 +++++++++- .../apecloud-mysql/scripts/mysql-reload.tpl | 4 +- .../apps.kubeblocks.io_configconstraints.yaml | 45 +++++++ .../config/pg14-config-effect-scope.yaml | 67 +++------- .../scripts/patroni-reload.tpl | 17 +++ .../scripts/restart-parameter.yaml | 50 +++++++ .../templates/clusterdefinition.yaml | 2 +- .../templates/configconstraint.yaml | 21 +-- .../templates/patroni-reload.yaml | 15 +++ .../configuration/config_manager/builder.go | 5 +- .../config_manager/builder_test.go | 30 +++++ .../dynamic_paramter_updater.go | 123 +++++++++++++++--- .../configuration/config_manager/files.go | 44 +++++++ .../config_manager/files_test.go | 74 +++++++++++ .../configuration/config_manager/handler.go | 11 +- .../config_manager/handler_test.go | 5 +- .../config_manager/reload_util.go | 36 ++--- .../config_manager/reload_util_test.go | 101 +++++++++++++- .../config_manager/volume_watcher.go | 8 +- .../config_manager/volume_watcher_test.go | 4 +- internal/gotemplate/functional.go | 13 ++ internal/gotemplate/tpl_engine.go | 6 +- 30 files changed, 758 insertions(+), 123 deletions(-) create mode 100644 deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl create mode 100644 deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml create mode 100644 deploy/postgresql-patroni-ha/templates/patroni-reload.yaml create mode 100644 internal/configuration/config_manager/files.go create mode 100644 internal/configuration/config_manager/files_test.go diff --git a/apis/apps/v1alpha1/configconstraint_types.go b/apis/apps/v1alpha1/configconstraint_types.go index e888b977b..fa0b9eb5d 100644 --- a/apis/apps/v1alpha1/configconstraint_types.go +++ b/apis/apps/v1alpha1/configconstraint_types.go @@ -52,6 +52,10 @@ type ConfigConstraintSpec struct { // +optional ImmutableParameters []string `json:"immutableParameters,omitempty"` + // selector is used to match the label on the pod, + // for example, a pod of the primary is match on the patroni cluster. + Selector *metav1.LabelSelector `json:"selector,omitempty"` + // formatterConfig describes the format of the configuration file, the controller // 1. parses configuration file // 2. analyzes the modified parameters diff --git a/cmd/reloader/app/cmd.go b/cmd/reloader/app/cmd.go index c7ba59fa8..6947a6a2a 100644 --- a/cmd/reloader/app/cmd.go +++ b/cmd/reloader/app/cmd.go @@ -176,6 +176,8 @@ func checkOptions(opt *VolumeWatcherOpts) error { type TplScriptConfig struct { Scripts string `json:"scripts"` FileRegex string `json:"fileRegex"` + DataType string `json:"dataType"` + DSN string `json:"dsn"` FormatterConfig appsv1alpha1.FormatterConfig `json:"formatterConfig"` } @@ -198,6 +200,8 @@ func checkTPLScriptOptions(opt *VolumeWatcherOpts) error { } opt.FormatterConfig = &tplConfig.FormatterConfig + opt.DSN = tplConfig.DSN + opt.DataType = tplConfig.DataType opt.FileRegex = tplConfig.FileRegex opt.TPLScriptPath = filepath.Join(filepath.Dir(opt.TPLConfig), tplConfig.Scripts) return nil @@ -239,7 +243,7 @@ func createHandlerWithVolumeWatch(opt *VolumeWatcherOpts) (cfgcore.WatchEventHan case ShellTool: return cfgcore.CreateExecHandler(opt.Command) case TPLScript: - return cfgcore.CreateTPLScriptHandler(opt.TPLScriptPath, opt.VolumeDirs, opt.FileRegex, opt.BackupPath, opt.FormatterConfig) + return cfgcore.CreateTPLScriptHandler(opt.TPLScriptPath, opt.VolumeDirs, opt.FileRegex, opt.BackupPath, opt.FormatterConfig, opt.DataType, opt.DSN) case SQL, WebHook: return nil, cfgutil.MakeError("event type[%s]: not yet, but in the future", opt.NotifyHandType.String()) default: diff --git a/cmd/reloader/app/flags.go b/cmd/reloader/app/flags.go index b86c45d30..4217c11f0 100644 --- a/cmd/reloader/app/flags.go +++ b/cmd/reloader/app/flags.go @@ -113,6 +113,8 @@ type VolumeWatcherOpts struct { BackupPath string FormatterConfig *appsv1alpha1.FormatterConfig TPLScriptPath string + DataType string + DSN string LogLevel string NotifyHandType NotifyEventType diff --git a/cmd/reloader/app/proxy.go b/cmd/reloader/app/proxy.go index a93cf97d8..cb2d440a5 100644 --- a/cmd/reloader/app/proxy.go +++ b/cmd/reloader/app/proxy.go @@ -84,7 +84,7 @@ func (r *reconfigureProxy) StopContainer(ctx context.Context, request *cfgproto. return &cfgproto.StopContainerResponse{}, nil } -func (r *reconfigureProxy) OnlineUpgradeParams(_ context.Context, request *cfgproto.OnlineUpgradeParamsRequest) (*cfgproto.OnlineUpgradeParamsResponse, error) { +func (r *reconfigureProxy) OnlineUpgradeParams(ctx context.Context, request *cfgproto.OnlineUpgradeParamsRequest) (*cfgproto.OnlineUpgradeParamsResponse, error) { if r.updater == nil { return nil, cfgcore.MakeError("online updater is not initialized.") } @@ -92,7 +92,7 @@ func (r *reconfigureProxy) OnlineUpgradeParams(_ context.Context, request *cfgpr if len(params) == 0 { return nil, cfgcore.MakeError("update params not empty.") } - if err := r.updater(params); err != nil { + if err := r.updater(ctx, params); err != nil { return nil, err } return &cfgproto.OnlineUpgradeParamsResponse{}, nil @@ -103,7 +103,7 @@ func (r *reconfigureProxy) initOnlineUpdater(opt *VolumeWatcherOpts) error { return nil } - updater, err := cfgcm.OnlineUpdateParamsHandle(opt.TPLScriptPath, opt.FormatterConfig) + updater, err := cfgcm.OnlineUpdateParamsHandle(opt.TPLScriptPath, opt.FormatterConfig, opt.DataType, opt.DSN) if err != nil { return cfgcore.WrapError(err, "failed to create online updater") } diff --git a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml index 474796337..5a6a5d3b0 100644 --- a/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml +++ b/config/crd/bases/apps.kubeblocks.io_configconstraints.yaml @@ -216,6 +216,51 @@ spec: - signal type: object type: object + selector: + description: selector is used to match the label on the pod, for example, + a pod of the primary is match on the patroni cluster. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object staticParameters: description: staticParameters, list of StaticParameter, modifications of them trigger a process restart. diff --git a/controllers/apps/configuration/policy_util.go b/controllers/apps/configuration/policy_util.go index d5f131c9b..4a1f9f0e5 100644 --- a/controllers/apps/configuration/policy_util.go +++ b/controllers/apps/configuration/policy_util.go @@ -19,7 +19,9 @@ package configuration import ( "context" "fmt" + "net" "sort" + "strconv" "github.com/spf13/viper" corev1 "k8s.io/api/core/v1" @@ -166,7 +168,11 @@ func getConsensusPods(params reconfigureParams) ([]corev1.Pod, error) { // TODO commonOnlineUpdateWithPod migrate to sql command pipeline func commonOnlineUpdateWithPod(pod *corev1.Pod, ctx context.Context, createClient createReconfigureClient, configSpec string, updatedParams map[string]string) error { - client, err := createClient(generateManagerSidecarAddr(pod)) + address, err := cfgManagerGrpcURL(pod) + if err != nil { + return err + } + client, err := createClient(address) if err != nil { return err } @@ -196,8 +202,12 @@ func commonStopContainerWithPod(pod *corev1.Pod, ctx context.Context, containerN containerIDs = append(containerIDs, containerID) } + address, err := cfgManagerGrpcURL(pod) + if err != nil { + return err + } // stop container - client, err := createClient(generateManagerSidecarAddr(pod)) + client, err := createClient(address) if err != nil { return err } @@ -216,10 +226,20 @@ func commonStopContainerWithPod(pod *corev1.Pod, ctx context.Context, containerN return nil } -func generateManagerSidecarAddr(pod *corev1.Pod) string { - var ( - podAddress = pod.Status.PodIP - podPort = viper.GetInt32(constant.ConfigManagerGPRCPortEnv) - ) - return fmt.Sprintf("%s:%d", podAddress, podPort) +func cfgManagerGrpcURL(pod *corev1.Pod) (string, error) { + podPort := viper.GetInt(constant.ConfigManagerGPRCPortEnv) + return getURLFromPod(pod, podPort) +} + +func getURLFromPod(pod *corev1.Pod, portPort int) (string, error) { + ip := net.ParseIP(pod.Status.PodIP) + if ip == nil { + return "", cfgcore.MakeError("%s is not a valid IP", pod.Status.PodIP) + } + + // Sanity check PodIP + if ip.To4() == nil && ip.To16() == nil { + return "", fmt.Errorf("%s is not a valid IPv4/IPv6 address", pod.Status.PodIP) + } + return net.JoinHostPort(ip.String(), strconv.Itoa(portPort)), nil } diff --git a/controllers/apps/configuration/policy_util_test.go b/controllers/apps/configuration/policy_util_test.go index cf4949e60..ca5dc03da 100644 --- a/controllers/apps/configuration/policy_util_test.go +++ b/controllers/apps/configuration/policy_util_test.go @@ -186,6 +186,7 @@ func newMockPodsWithStatefulSet(sts *appsv1.StatefulSet, replicas int, options . for i := 0; i < replicas; i++ { pods[i] = newMockPod(sts.Name+"-"+fmt.Sprint(i), &sts.Spec.Template.Spec) pods[i].OwnerReferences = []metav1.OwnerReference{newControllerRef(sts, stsSchemaKind)} + pods[i].Status.PodIP = "1.1.1.1" } for _, customFn := range options { for i := range pods { diff --git a/controllers/apps/configuration/sync_upgrade_policy.go b/controllers/apps/configuration/sync_upgrade_policy.go index 1cbd15433..6cfed37b7 100644 --- a/controllers/apps/configuration/sync_upgrade_policy.go +++ b/controllers/apps/configuration/sync_upgrade_policy.go @@ -18,6 +18,8 @@ package configuration import ( corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgutil "github.com/apecloud/kubeblocks/internal/configuration" @@ -67,6 +69,21 @@ func (o *syncPolicy) Upgrade(params reconfigureParams) (ReturnedStatus, error) { return sync(params, updatedParameters, pods, funcs) } +func matchLabel(pods []corev1.Pod, selector *metav1.LabelSelector) ([]corev1.Pod, error) { + var result []corev1.Pod + + match, err := metav1.LabelSelectorAsSelector(selector) + if err != nil { + return nil, cfgutil.WrapError(err, "failed to convert selector: %v", selector) + } + for _, pod := range pods { + if match.Matches(labels.Set(pod.Labels)) { + result = append(result, pod) + } + } + return result, nil +} + func sync(params reconfigureParams, updatedParameters map[string]string, pods []corev1.Pod, funcs RollingUpgradeFuncs) (ReturnedStatus, error) { var ( r = ESNone @@ -80,7 +97,20 @@ func sync(params reconfigureParams, updatedParameters map[string]string, pods [] versionHash = params.getTargetVersionHash() ) + if params.ConfigConstraint.Selector != nil { + pods, err = matchLabel(pods, params.ConfigConstraint.Selector) + } + if err != nil { + return makeReturnedStatus(ESAndRetryFailed), err + } + if len(pods) == 0 { + params.Ctx.Log.Info("no pods to update, and retry, selector: %v, current all pod: %v", params.ConfigConstraint.Selector) + return makeReturnedStatus(ESRetry), nil + } + + requireUpdatedCount := int32(len(pods)) for _, pod := range pods { + params.Ctx.Log.V(1).Info("sync pod: %s", pod.Name) if podutil.IsMatchConfigVersion(&pod, configKey, versionHash) { progress++ continue @@ -99,10 +129,10 @@ func sync(params reconfigureParams, updatedParameters map[string]string, pods [] progress++ } - if total != progress || replicas != total { + if requireUpdatedCount != progress || replicas != total { r = ESRetry } - return makeReturnedStatus(r, withExpected(replicas), withSucceed(progress)), nil + return makeReturnedStatus(r, withExpected(requireUpdatedCount), withSucceed(progress)), nil } func getOnlineUpdateParams(configPatch *cfgutil.ConfigPatchInfo, formatConfig *appsv1alpha1.FormatterConfig) map[string]string { diff --git a/controllers/apps/configuration/sync_upgrade_policy_test.go b/controllers/apps/configuration/sync_upgrade_policy_test.go index 6108e9c5a..24ff2d9d0 100644 --- a/controllers/apps/configuration/sync_upgrade_policy_test.go +++ b/controllers/apps/configuration/sync_upgrade_policy_test.go @@ -19,9 +19,11 @@ package configuration import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" - "k8s.io/apimachinery/pkg/runtime" "github.com/golang/mock/gomock" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgproto "github.com/apecloud/kubeblocks/internal/configuration/proto" @@ -104,4 +106,66 @@ var _ = Describe("Reconfigure OperatorSyncPolicy", func() { }) }) + Context("sync reconfigure policy with selector test", func() { + It("Should success without error", func() { + By("check policy name") + Expect(operatorSyncPolicy.GetPolicyName()).Should(BeEquivalentTo("operatorSyncUpdate")) + + By("prepare reconfigure policy params") + mockParam := newMockReconfigureParams("operatorSyncPolicy", k8sMockClient.Client(), + withGRPCClient(func(addr string) (cfgproto.ReconfigureClient, error) { + return reconfigureClient, nil + }), + withMockStatefulSet(3, nil), + withConfigSpec("for_test", map[string]string{"a": "c b e f"}), + withConfigConstraintSpec(&appsv1alpha1.FormatterConfig{Format: appsv1alpha1.RedisCfg}), + withConfigPatch(map[string]string{ + "a": "c b e f", + }), + withClusterComponent(3), + withCDComponent(appsv1alpha1.Consensus, []appsv1alpha1.ComponentConfigSpec{{ + ComponentTemplateSpec: appsv1alpha1.ComponentTemplateSpec{ + Name: "for_test", + VolumeName: "test_volume", + }, + }})) + + // add selector + mockParam.ConfigConstraint.Selector = &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "primary": "true", + }, + } + + By("mock client get pod caller") + k8sMockClient.MockListMethod(testutil.WithListReturned( + testutil.WithConstructListReturnedResult( + fromPodObjectList(newMockPodsWithStatefulSet(&mockParam.ComponentUnits[0], 3, + withReadyPod(0, 1), func(pod *corev1.Pod, index int) { + if index == 0 { + if pod.Labels == nil { + pod.Labels = make(map[string]string) + } + pod.Labels["primary"] = "true" + } + }))), + testutil.WithAnyTimes())) + + By("mock client patch caller") + // mock client update caller + k8sMockClient.MockPatchMethod(testutil.WithSucceed(testutil.WithTimes(1))) + + By("mock remote online update caller") + reconfigureClient.EXPECT().OnlineUpgradeParams(gomock.Any(), gomock.Any()).Return( + &cfgproto.OnlineUpgradeParamsResponse{}, nil). + Times(1) + + status, err := operatorSyncPolicy.Upgrade(mockParam) + Expect(err).Should(Succeed()) + Expect(status.Status).Should(BeEquivalentTo(ESNone)) + Expect(status.SucceedCount).Should(BeEquivalentTo(1)) + Expect(status.ExpectedCount).Should(BeEquivalentTo(1)) + }) + }) + }) diff --git a/deploy/apecloud-mysql/scripts/mysql-reload.tpl b/deploy/apecloud-mysql/scripts/mysql-reload.tpl index 7ec40d316..8a2184509 100644 --- a/deploy/apecloud-mysql/scripts/mysql-reload.tpl +++ b/deploy/apecloud-mysql/scripts/mysql-reload.tpl @@ -28,8 +28,8 @@ {{- end }} {{- end }} {{- if ge $var_int 0 }} - {{- exec_sql ( printf "SET GLOBAL %s = %d" $pk $var_int ) }} + {{- execSql ( printf "SET GLOBAL %s = %d" $pk $var_int ) }} {{- else }} - {{- exec_sql ( printf "SET GLOBAL %s = '%s'" $pk $pv ) }} + {{- execSql ( printf "SET GLOBAL %s = '%s'" $pk $pv ) }} {{- end }} {{- end }} \ No newline at end of file diff --git a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml index 474796337..5a6a5d3b0 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_configconstraints.yaml @@ -216,6 +216,51 @@ spec: - signal type: object type: object + selector: + description: selector is used to match the label on the pod, for example, + a pod of the primary is match on the patroni cluster. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. + The requirements are ANDed. + items: + description: A label selector requirement is a selector that + contains values, a key, and an operator that relates the key + and values. + properties: + key: + description: key is the label key that the selector applies + to. + type: string + operator: + description: operator represents a key's relationship to + a set of values. Valid operators are In, NotIn, Exists + and DoesNotExist. + type: string + values: + description: values is an array of string values. If the + operator is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single + {key,value} in the matchLabels map is equivalent to an element + of matchExpressions, whose key field is "key", the operator + is "In", and the values array contains only "value". The requirements + are ANDed. + type: object + type: object staticParameters: description: staticParameters, list of StaticParameter, modifications of them trigger a process restart. diff --git a/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml b/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml index 3aeb21fe8..6e4d69046 100644 --- a/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml +++ b/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml @@ -1,47 +1,22 @@ +# Patroni bootstrap parameters staticParameters: -# - autovacuum_freeze_max_age -# - autovacuum_max_workers -# - autovacuum_multixact_freeze_max_age -# - config_file -# - cron.database_name -# - cron.log_run -# - cron.log_statement -# - cron.max_running_jobs -# - cron.use_background_workers -# - data_directory -# - hba_file -# - huge_pages -# - ident_file -# - ignore_invalid_pages -# - listen_addresses -# - logging_collector -# - max_connections -# - max_files_per_process -# - max_locks_per_transaction -# - max_logical_replication_workers -# - max_pred_locks_per_transaction -# - max_prepared_transactions -# - max_replication_slots -# - max_wal_senders -# - max_worker_processes -# - min_dynamic_shared_memory -# - old_snapshot_threshold -# - pglogical.batch_inserts -# - pglogical.synchronous_commit -# - pglogical.use_spi -# - pg_prewarm.autoprewarm -# - pg_stat_statements.max -# - port -# - postgis.gdal_enabled_drivers -# - recovery_init_sync_method -# - session_preload_libraries -# - shared_buffers -# - shared_preload_libraries -# - superuser_reserved_connections -# - track_activity_query_size -# - track_commit_timestamp -# - unix_socket_directories -# - unix_socket_group -# - unix_socket_permissions -# - wal_buffers -# - wal_compression \ No newline at end of file + - archive_command + - shared_buffers + - logging_collector + - log_destination + - log_directory + - log_filename + - log_file_mode + - log_rotation_age + - log_truncate_on_rotation + - ssl + - ssl_ca_file + - ssl_crl_file + - ssl_cert_file + - ssl_key_file + - shared_preload_libraries + - bg_mon.listen_address + - bg_mon.history_buckets + - pg_stat_statements.track_utility + - extwlist.extensions + - extwlist.custom_path \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl b/deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl new file mode 100644 index 000000000..6669eea7b --- /dev/null +++ b/deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl @@ -0,0 +1,17 @@ +{{- $bootstrap := $.Files.Get "bootstrap.yaml" | fromYamlArray }} +{{- $command := "reload" }} +{{- range $pk, $_ := $.arg0 }} + {{- if has $pk $bootstrap }} + {{- $command = "restart" }} + {{ break }} + {{- end }} +{{- end }} +{{ $params := dict "parameters" $.arg0 }} +{{- $err := execSql ( dict "postgresql" $params | toJson ) "config" }} +{{- if $err }} + {{- failed $err }} +{{- end }} +{{- $err := execSql "" $command }} +{{- if $err }} + {{- failed $err }} +{{- end }} diff --git a/deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml b/deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml new file mode 100644 index 000000000..eabd4c5ae --- /dev/null +++ b/deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml @@ -0,0 +1,50 @@ +#parameters: +- archive_mode +- autovacuum_freeze_max_age +- autovacuum_max_workers +- autovacuum_multixact_freeze_max_age +- config_file +- cron.database_name +- cron.log_run +- cron.log_statement +- cron.max_running_jobs +- cron.use_background_workers +- data_directory +- hba_file +- huge_pages +- huge_page_size +- ident_file +- ignore_invalid_pages +- listen_addresses +- logging_collector +- max_connections +- max_files_per_process +- max_locks_per_transaction +- max_logical_replication_workers +- max_pred_locks_per_transaction +- max_prepared_transactions +- max_replication_slots +- max_wal_senders +- max_worker_processes +- min_dynamic_shared_memory +- old_snapshot_threshold +- pglogical.batch_inserts +- pglogical.synchronous_commit +- pglogical.use_spi +- pg_prewarm.autoprewarm +- pg_stat_statements.max +- port +- postgis.gdal_enabled_drivers +- recovery_init_sync_method +- session_preload_libraries +- shared_buffers +- shared_preload_libraries +- superuser_reserved_connections +- track_activity_query_size +- track_commit_timestamp +- unix_socket_directories +- unix_socket_group +- unix_socket_permissions +- wal_buffers +- wal_compression +- wal_decode_buffer_size \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml index 62841e385..d3a7728da 100644 --- a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml +++ b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml @@ -7,7 +7,7 @@ metadata: spec: connectionCredential: username: postgres - password: {{ (include "postgresql.postgresPassword" .) | quote }} + password: "$(RANDOM_PASSWD)" endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-postgresql)" host: "$(SVC_FQDN)" port: "$(SVC_PORT_tcp-postgresql)" diff --git a/deploy/postgresql-patroni-ha/templates/configconstraint.yaml b/deploy/postgresql-patroni-ha/templates/configconstraint.yaml index 076c860e5..b9445e995 100644 --- a/deploy/postgresql-patroni-ha/templates/configconstraint.yaml +++ b/deploy/postgresql-patroni-ha/templates/configconstraint.yaml @@ -7,9 +7,9 @@ metadata: {{- include "postgresql.labels" . | nindent 4 }} spec: reloadOptions: - unixSignalTrigger: - signal: SIGHUP - processName: postgres + tplScriptTrigger: + scriptConfigMapRef: patroni-reload-script + namespace: {{ .Release.Namespace }} # top level mysql configuration type cfgSchemaTopLevelName: PGParameter @@ -31,16 +31,17 @@ spec: {{- end }} {{- end}} - ## reload parameters - ## dynamicParameters - {{- if hasKey $cc "dynamicParameters" }} - dynamicParameters: - {{- $params := get $cc "dynamicParameters" }} + ## define immutable parameter list, this feature is not currently supported. + {{- if hasKey $cc "immutableParameters" }} + immutableParameters: + {{- $params := get $cc "immutableParameters" }} {{- range $params }} - {{ . }} {{- end }} - {{- end}} + {{- end}} + + - # configuration file format + # configuration file format formatterConfig: format: properties diff --git a/deploy/postgresql-patroni-ha/templates/patroni-reload.yaml b/deploy/postgresql-patroni-ha/templates/patroni-reload.yaml new file mode 100644 index 000000000..a2ccab659 --- /dev/null +++ b/deploy/postgresql-patroni-ha/templates/patroni-reload.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: patroni-reload-script + labels: + {{- include "postgresql.labels" . | nindent 4 }} +data: + patroni_reload.tpl: |- + {{- .Files.Get "scripts/patroni-reload.tpl" | nindent 4 }} + bootstrap.yaml: |- + {{- .Files.Get "scripts/restart-parameter.yaml" | nindent 4 }} + reload.yaml: |- + scripts: patroni_reload.tpl + dataType: patroni + dsn: localhost:8008 \ No newline at end of file diff --git a/internal/configuration/config_manager/builder.go b/internal/configuration/config_manager/builder.go index 1a56ce26f..31a872ed9 100644 --- a/internal/configuration/config_manager/builder.go +++ b/internal/configuration/config_manager/builder.go @@ -78,9 +78,12 @@ func buildTPLScriptArgs(options *appsv1alpha1.TPLScriptTrigger, volumeDirs []cor return err } - args := buildConfigManagerCommonArgs(volumeDirs) + var args []string if options.Sync != nil && *options.Sync { args = append(args, "--operator-update-enable") + args = append(args, "--log-level", viper.GetString(constant.ConfigManagerLogLevel)) + } else { + args = buildConfigManagerCommonArgs(volumeDirs) } args = append(args, "--tcp", viper.GetString(constant.ConfigManagerGPRCPortEnv)) args = append(args, "--notify-type", string(appsv1alpha1.TPLScriptType)) diff --git a/internal/configuration/config_manager/builder_test.go b/internal/configuration/config_manager/builder_test.go index edde3b3f1..8e8ebf7cf 100644 --- a/internal/configuration/config_manager/builder_test.go +++ b/internal/configuration/config_manager/builder_test.go @@ -141,6 +141,36 @@ var _ = Describe("ConfigManager Test", func() { }, }, { name: "buildCfgContainerParams", + args: args{ + reloadOptions: &appsv1alpha1.ReloadOptions{ + TPLScriptTrigger: &appsv1alpha1.TPLScriptTrigger{ + ScriptConfigMapRef: "script_cm", + Namespace: "default", + Sync: func() *bool { b := true; return &b }()}}, + volumeDirs: []corev1.VolumeMount{ + { + MountPath: "/postgresql/conf", + Name: "pg_config", + }}, + cli: mockK8sCli.Client(), + ctx: context.TODO(), + param: &CfgManagerBuildParams{ + Cluster: &appsv1alpha1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "abcd", + Namespace: "default", + }, + }, + }, + }, + expectedArgs: []string{ + `--notify-type`, `tpl`, + `--tpl-config`, `/opt/config/reload/reload.yaml`, + `--operator-update-enable`, + }, + wantErr: false, + }, { + name: "buildCfgContainerParamsWithOutSync", args: args{ reloadOptions: &appsv1alpha1.ReloadOptions{ TPLScriptTrigger: &appsv1alpha1.TPLScriptTrigger{ diff --git a/internal/configuration/config_manager/dynamic_paramter_updater.go b/internal/configuration/config_manager/dynamic_paramter_updater.go index 823619092..0f1bf02fc 100644 --- a/internal/configuration/config_manager/dynamic_paramter_updater.go +++ b/internal/configuration/config_manager/dynamic_paramter_updater.go @@ -20,18 +20,22 @@ import ( "context" "database/sql" "fmt" + "io/ioutil" + "net/http" "os" "strconv" + "strings" "time" _ "github.com/go-sql-driver/mysql" + "github.com/spf13/viper" cfgcore "github.com/apecloud/kubeblocks/internal/configuration" ) // DynamicParamUpdater is designed to adapt to the dapper implementation. type DynamicParamUpdater interface { - ExecCommand(command string) (string, error) + ExecCommand(ctx context.Context, command string, args ...string) (string, error) Close() } @@ -40,32 +44,44 @@ type mysqlCommandChannel struct { } const ( - DBType = "DB_TYPE" + dbType = "DB_TYPE" - MYSQL = "mysql" - MYSQLDsnEnv = "DATA_SOURCE_NAME" + connectTimeout = 5 * time.Second + + mysql = "mysql" + patroni = "patroni" + mysqlDsnEnv = "DATA_SOURCE_NAME" + patroniRestAPIURL = "PATRONI_REST_API_URL" ) -func NewCommandChannel(dbType string) (DynamicParamUpdater, error) { +func NewCommandChannel(ctx context.Context, dataType, dsn string) (DynamicParamUpdater, error) { // TODO using dapper command channel - logger.V(1).Info(fmt.Sprintf("new command channel. [%s]", dbType)) - switch dbType { - case MYSQL: - return NewMysqlConnection() + if dataType == "" { + dataType = viper.GetString(dbType) + } + + logger.V(1).Info(fmt.Sprintf("new command channel. [%s]", dataType)) + switch strings.ToLower(dataType) { + case mysql: + return NewMysqlConnection(ctx, dsn) + case patroni: + return NewPGPatroniConnection(dsn) default: // TODO mock db begin support dapper } - return nil, cfgcore.MakeError("not support type[%s]", dbType) + return nil, cfgcore.MakeError("not support type[%s]", dataType) } -func NewMysqlConnection() (DynamicParamUpdater, error) { +func NewMysqlConnection(ctx context.Context, dsn string) (DynamicParamUpdater, error) { logger.V(1).Info("connecting mysql.") - dsn := os.Getenv(MYSQLDsnEnv) + if dsn == "" { + dsn = os.Getenv(mysqlDsnEnv) + } if dsn == "" { return nil, cfgcore.MakeError("require DATA_SOURCE_NAME env.") } - db, err := sql.Open(MYSQL, dsn) + db, err := sql.Open(mysql, dsn) if err != nil { return nil, cfgcore.WrapError(err, "failed to opening connection to mysql.") } @@ -74,7 +90,7 @@ func NewMysqlConnection() (DynamicParamUpdater, error) { // Set max lifetime for a connection. db.SetConnMaxLifetime(1 * time.Minute) - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + ctx, cancel := context.WithTimeout(ctx, connectTimeout) defer cancel() if err := db.PingContext(ctx); err != nil { logger.Error(err, "failed to pinging mysqld.") @@ -84,8 +100,8 @@ func NewMysqlConnection() (DynamicParamUpdater, error) { return &mysqlCommandChannel{db: db}, nil } -func (m *mysqlCommandChannel) ExecCommand(sql string) (string, error) { - ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) +func (m *mysqlCommandChannel) ExecCommand(ctx context.Context, sql string, _ ...string) (string, error) { + ctx, cancel := context.WithTimeout(ctx, connectTimeout) defer cancel() r, err := m.db.ExecContext(ctx, sql) @@ -103,3 +119,78 @@ func (m *mysqlCommandChannel) Close() { m.db.Close() logger.V(1).Info("closed mysql connection.") } + +type pgPatroniCommandChannel struct { + restapiURL string +} + +func (p *pgPatroniCommandChannel) ExecCommand(ctx context.Context, command string, args ...string) (string, error) { + const ( + Config = "config" + Reload = "reload" + Restart = "restart" + ) + + if len(args) == 0 { + return "", cfgcore.MakeError("require patroni functional.") + } + + functional := args[0] + restPath := strings.Join([]string{p.restapiURL, functional}, "/") + logger.V(1).Info(fmt.Sprintf("exec patroni command: %s", functional)) + switch functional { + case Config: + return sendRestRequest(ctx, command, restPath, "PATCH") + case Restart, Reload: + return sendRestRequest(ctx, command, restPath, "POST") + } + return "", cfgcore.MakeError("not support patroni functional[%s]", args[0]) +} + +func (p *pgPatroniCommandChannel) Close() { +} + +func NewPGPatroniConnection(hostURL string) (DynamicParamUpdater, error) { + logger.V(1).Info("connecting patroni.") + if hostURL == "" { + hostURL = os.Getenv(patroniRestAPIURL) + } + if hostURL == "" { + return nil, cfgcore.MakeError("require PATRONI_REST_API_URL env.") + } + + return &pgPatroniCommandChannel{restapiURL: formatRestAPIPath(hostURL)}, nil +} + +func sendRestRequest(ctx context.Context, body string, url string, method string) (string, error) { + client := &http.Client{} + ctx, cancel := context.WithTimeout(ctx, connectTimeout) + defer cancel() + // create new HTTP PATCH request with JSON payload + req, err := http.NewRequestWithContext(ctx, method, url, strings.NewReader(body)) + if err != nil { + return "", err + } + + // set content-type header to JSON + req.Header.Set("Content-Type", "application/json") + resp, err := client.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + + response, err := ioutil.ReadAll(resp.Body) + if err != nil { + return "", err + } + return string(response), nil +} + +func formatRestAPIPath(path string) string { + const restAPIPrefix = "http://" + if strings.HasPrefix(path, restAPIPrefix) { + return path + } + return fmt.Sprintf("%s%s", restAPIPrefix, path) +} diff --git a/internal/configuration/config_manager/files.go b/internal/configuration/config_manager/files.go new file mode 100644 index 000000000..8343cc96f --- /dev/null +++ b/internal/configuration/config_manager/files.go @@ -0,0 +1,44 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package configmanager + +import ( + "errors" + "os" + "path/filepath" + + cfgutil "github.com/apecloud/kubeblocks/internal/configuration" +) + +type files string + +func newFiles(basePath string) files { + return files(basePath) +} + +func (f files) Get(filename string) (string, error) { + path := filepath.Join(string(f), filename) + if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) { + return "", cfgutil.MakeError("file %s not found", filename) + } + + b, err := os.ReadFile(path) + if err != nil { + return "", cfgutil.WrapError(err, "failed to read file %s", filename) + } + return string(b), nil +} diff --git a/internal/configuration/config_manager/files_test.go b/internal/configuration/config_manager/files_test.go new file mode 100644 index 000000000..7b1c3089e --- /dev/null +++ b/internal/configuration/config_manager/files_test.go @@ -0,0 +1,74 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package configmanager + +import ( + "os" + "testing" + + "github.com/stretchr/testify/require" +) + +func Test_newFiles(t *testing.T) { + pwd, err := os.Getwd() + if err != nil { + t.Errorf("failed to Getwd directory") + } + + tmpDir, err := os.MkdirTemp(os.TempDir(), "files-test-") + require.Nil(t, err) + defer os.RemoveAll(tmpDir) + createTestConfigureDirectory(t, tmpDir, "my.test.yaml", "a: b") + + type args struct { + basePath string + } + tests := []struct { + name string + args args + file string + want string + wantErr bool + }{{ + name: "testFiles", + args: args{ + basePath: pwd, + }, + file: "not_file", + wantErr: true, + }, { + name: "testFiles", + args: args{ + basePath: tmpDir, + }, + file: "my.test.yaml", + want: "a: b", + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + files := newFiles(tt.args.basePath) + v, err := files.Get(tt.file) + if (err != nil) != tt.wantErr { + t.Errorf("Get() error = %v, wantErr %v", err, tt.wantErr) + return + } + if !tt.wantErr && tt.want != v { + t.Errorf("Get() is not expected %s : %s", v, tt.want) + } + }) + } +} diff --git a/internal/configuration/config_manager/handler.go b/internal/configuration/config_manager/handler.go index f9cfc19d3..4cdb19113 100644 --- a/internal/configuration/config_manager/handler.go +++ b/internal/configuration/config_manager/handler.go @@ -17,6 +17,7 @@ limitations under the License. package configmanager import ( + "context" "fmt" "os" "os/exec" @@ -86,7 +87,7 @@ func CreateSignalHandler(sig appsv1alpha1.SignalType, processName string) (Watch logger.Error(err, "failed to create signal handler") return nil, err } - return func(event fsnotify.Event) error { + return func(_ context.Context, event fsnotify.Event) error { pid, err := findPidFromProcessName(processName) if err != nil { return err @@ -102,7 +103,7 @@ func CreateExecHandler(command string) (WatchEventHandler, error) { return nil, cfgutil.MakeError("invalid command: %s", command) } cmd := exec.Command(args[0], args[1:]...) - return func(_ fsnotify.Event) error { + return func(_ context.Context, _ fsnotify.Event) error { stdout, err := cfgcontainer.ExecShellCommand(cmd) if err == nil { logger.V(1).Info(fmt.Sprintf("exec: [%s], result: [%s]", command, stdout)) @@ -116,7 +117,7 @@ func IsValidUnixSignal(sig appsv1alpha1.SignalType) bool { return ok } -func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, backupPath string, formatConfig *appsv1alpha1.FormatterConfig) (WatchEventHandler, error) { +func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, backupPath string, formatConfig *appsv1alpha1.FormatterConfig, dataType string, dsn string) (WatchEventHandler, error) { logger.V(1).Info(fmt.Sprintf("config file regex: %s", fileRegex)) logger.V(1).Info(fmt.Sprintf("config file reload script: %s", tplScripts)) if _, err := os.Stat(tplScripts); err != nil { @@ -136,7 +137,7 @@ func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, if err := backupConfigFiles(dirs, filter, backupPath); err != nil { return nil, err } - return func(event fsnotify.Event) error { + return func(ctx context.Context, event fsnotify.Event) error { var ( lastVersion = []string{backupPath} currVersion = []string{filepath.Dir(event.Name)} @@ -153,7 +154,7 @@ func CreateTPLScriptHandler(tplScripts string, dirs []string, fileRegex string, if err != nil { return err } - if err := wrapGoTemplateRun(tplScripts, string(tplContent), updatedParams, formatConfig); err != nil { + if err := wrapGoTemplateRun(ctx, tplScripts, string(tplContent), updatedParams, formatConfig, dataType, dsn); err != nil { return err } return backupLastConfigFiles(currFiles, backupPath) diff --git a/internal/configuration/config_manager/handler_test.go b/internal/configuration/config_manager/handler_test.go index 9ff8a18b5..c74645e8e 100644 --- a/internal/configuration/config_manager/handler_test.go +++ b/internal/configuration/config_manager/handler_test.go @@ -17,6 +17,7 @@ limitations under the License. package configmanager import ( + "context" "fmt" "io/fs" "os" @@ -70,7 +71,7 @@ func TestCreateExecHandler(t *testing.T) { require.ErrorContains(t, err, "invalid command") c, err := CreateExecHandler("go version") require.Nil(t, err) - require.Nil(t, c(fsnotify.Event{})) + require.Nil(t, c(context.Background(), fsnotify.Event{})) } func TestCreateTPLScriptHandler(t *testing.T) { @@ -82,7 +83,7 @@ func TestCreateTPLScriptHandler(t *testing.T) { tplFile := filepath.Join(tmpDir, "test.tpl") require.Nil(t, os.WriteFile(tplFile, []byte("xxx"), fs.ModePerm)) - _, err = CreateTPLScriptHandler(tplFile, []string{filepath.Join(tmpDir, "config")}, "", filepath.Join(tmpDir, "backup"), nil) + _, err = CreateTPLScriptHandler(tplFile, []string{filepath.Join(tmpDir, "config")}, "", filepath.Join(tmpDir, "backup"), nil, "", "") require.Nil(t, err) } diff --git a/internal/configuration/config_manager/reload_util.go b/internal/configuration/config_manager/reload_util.go index 6812144dc..2d54ba547 100644 --- a/internal/configuration/config_manager/reload_util.go +++ b/internal/configuration/config_manager/reload_util.go @@ -17,6 +17,7 @@ limitations under the License. package configmanager import ( + "context" "fmt" "io" "io/fs" @@ -26,8 +27,6 @@ import ( "regexp" "text/template/parse" - "github.com/spf13/viper" - appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cfgutil "github.com/apecloud/kubeblocks/internal/configuration" cfgcontainer "github.com/apecloud/kubeblocks/internal/configuration/container" @@ -38,22 +37,24 @@ type regexFilter = func(fileName string) bool const ( builtInExecFunctionName = "exec" - builtInUpdateVariableFunctionName = "exec_sql" - builtInParamsPatchFunctionName = "patch_params" + builtInUpdateVariableFunctionName = "execSql" + builtInParamsPatchFunctionName = "patchParams" + + buildInFilesObjectName = "Files" ) -type DynamicUpdater = func(updatedParams map[string]string) error +type DynamicUpdater = func(ctx context.Context, updatedParams map[string]string) error -func OnlineUpdateParamsHandle(tplScript string, formatConfig *appsv1alpha1.FormatterConfig) (DynamicUpdater, error) { - tplContent, err := os.ReadFile(tplScript) +func OnlineUpdateParamsHandle(tplScriptPath string, formatConfig *appsv1alpha1.FormatterConfig, dataType, dsn string) (DynamicUpdater, error) { + tplContent, err := os.ReadFile(tplScriptPath) if err != nil { return nil, err } - if err := checkTPLScript(tplScript, string(tplContent)); err != nil { + if err := checkTPLScript(tplScriptPath, string(tplContent)); err != nil { return nil, err } - return func(updatedParams map[string]string) error { - return wrapGoTemplateRun(tplScript, string(tplContent), updatedParams, formatConfig) + return func(ctx context.Context, updatedParams map[string]string) error { + return wrapGoTemplateRun(ctx, tplScriptPath, string(tplContent), updatedParams, formatConfig, dataType, dsn) }, nil } @@ -64,34 +65,35 @@ func checkTPLScript(tplName string, tplContent string) error { return err } -func wrapGoTemplateRun(tplName string, tplContent string, updatedParams map[string]string, formatConfig *appsv1alpha1.FormatterConfig) error { +func wrapGoTemplateRun(ctx context.Context, tplScriptPath string, tplContent string, updatedParams map[string]string, formatConfig *appsv1alpha1.FormatterConfig, dataType string, dsn string) error { var ( err error commandChannel DynamicParamUpdater ) - if commandChannel, err = NewCommandChannel(viper.GetString(DBType)); err != nil { + if commandChannel, err = NewCommandChannel(ctx, dataType, dsn); err != nil { return err } defer commandChannel.Close() logger.Info(fmt.Sprintf("update global dynamic params: %v", updatedParams)) values := gotemplate.ConstructFunctionArgList(updatedParams) - engine := gotemplate.NewTplEngine(&values, constructReloadBuiltinFuncs(commandChannel, formatConfig), tplName, nil, nil) + values[buildInFilesObjectName] = newFiles(filepath.Dir(tplScriptPath)) + engine := gotemplate.NewTplEngine(&values, constructReloadBuiltinFuncs(ctx, commandChannel, formatConfig), tplScriptPath, nil, nil) _, err = engine.Render(tplContent) return err } -func constructReloadBuiltinFuncs(cc DynamicParamUpdater, formatConfig *appsv1alpha1.FormatterConfig) *gotemplate.BuiltInObjectsFunc { +func constructReloadBuiltinFuncs(ctx context.Context, cc DynamicParamUpdater, formatConfig *appsv1alpha1.FormatterConfig) *gotemplate.BuiltInObjectsFunc { return &gotemplate.BuiltInObjectsFunc{ builtInExecFunctionName: func(command string, args ...string) (string, error) { - execCommand := exec.Command(command, args...) + execCommand := exec.CommandContext(ctx, command, args...) stdout, err := cfgcontainer.ExecShellCommand(execCommand) logger.V(1).Info(fmt.Sprintf("command: [%s], output: %s, err: %v", execCommand.String(), stdout, err)) return stdout, err }, - builtInUpdateVariableFunctionName: func(sql string) error { - r, err := cc.ExecCommand(sql) + builtInUpdateVariableFunctionName: func(sql string, args ...string) error { + r, err := cc.ExecCommand(ctx, sql, args...) logger.V(1).Info(fmt.Sprintf("sql: [%s], result: [%v]", sql, r)) return err }, diff --git a/internal/configuration/config_manager/reload_util_test.go b/internal/configuration/config_manager/reload_util_test.go index 70b32ffbe..f02b2386f 100644 --- a/internal/configuration/config_manager/reload_util_test.go +++ b/internal/configuration/config_manager/reload_util_test.go @@ -17,10 +17,14 @@ limitations under the License. package configmanager import ( + "context" "io/fs" + "net/http" + "net/http/httptest" "os" "path/filepath" "reflect" + "strings" "testing" "github.com/stretchr/testify/require" @@ -79,7 +83,7 @@ func TestCreateUpdatedParamsPatch(t *testing.T) { } func TestConstructReloadBuiltinFuncs(t *testing.T) { - require.NotNil(t, constructReloadBuiltinFuncs(nil, nil)) + require.NotNil(t, constructReloadBuiltinFuncs(ctx, nil, nil)) } func prepareTestData(t *testing.T, dir1 string, dir2 string) string { @@ -119,3 +123,98 @@ max_connections=666 require.Nil(t, os.WriteFile(filepath.Join(tmpDir, dir2, testFileName), []byte(v2), fs.ModePerm)) return tmpDir } + +func TestOnlineUpdateParamsHandle(t *testing.T) { + server := mockRestAPIServer(t) + defer server.Close() + + partroniPath := "reload.tpl" + tmpTestData := mockPatroniTestData(t, partroniPath) + defer os.RemoveAll(tmpTestData) + + type args struct { + tplScriptPath string + formatConfig *appsv1alpha1.FormatterConfig + dataType string + dsn string + } + tests := []struct { + name string + args args + want DynamicUpdater + wantErr bool + }{{ + name: "online_update_params_handle_test", + args: args{ + tplScriptPath: filepath.Join(tmpTestData, partroniPath), + formatConfig: &appsv1alpha1.FormatterConfig{ + Format: appsv1alpha1.Properties, + }, + dsn: server.URL, + dataType: "patroni", + }, + }} + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := OnlineUpdateParamsHandle(tt.args.tplScriptPath, tt.args.formatConfig, tt.args.dataType, tt.args.dsn) + if (err != nil) != tt.wantErr { + t.Errorf("OnlineUpdateParamsHandle() error = %v, wantErr %v", err, tt.wantErr) + return + } + err = got(context.Background(), map[string]string{"key_buffer_size": "128M", "max_connections": "666"}) + require.Nil(t, err) + }) + } +} + +func mockRestAPIServer(t *testing.T) *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch strings.TrimSpace(r.URL.Path) { + default: + w.WriteHeader(http.StatusBadRequest) + _, err := w.Write([]byte(`failed`)) + require.Nil(t, err) + case "/config", "/restart": + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + _, err := w.Write([]byte(`success`)) + require.Nil(t, err) + } + })) +} + +func mockPatroniTestData(t *testing.T, reloadScript string) string { + tmpDir, err := os.MkdirTemp(os.TempDir(), "pg-patroni-test-") + require.Nil(t, err) + + err = os.WriteFile(filepath.Join(tmpDir, reloadScript), []byte(` +{{- $bootstrap := $.Files.Get "bootstrap.yaml" | fromYamlArray }} +{{- $command := "reload" }} +{{- range $pk, $_ := $.arg0 }} + {{- if has $pk $bootstrap }} + {{- $command = "restart" }} + {{ break }} + {{- end }} +{{- end }} +{{ $params := dict "parameters" $.arg0 }} +{{- $err := execSql ( dict "postgresql" $params | toJson ) "config" }} +{{- if $err }} + {{- failed $err }} +{{- end }} +{{- $err := execSql "" $command }} +{{- if $err }} + {{- failed $err }} +{{- end }} +`), fs.ModePerm) + require.Nil(t, err) + + err = os.WriteFile(filepath.Join(tmpDir, "bootstrap.yaml"), []byte(` +- archive_mode +- autovacuum_freeze_max_age +- autovacuum_max_workers +- max_connections +`), fs.ModePerm) + require.Nil(t, err) + + return tmpDir +} diff --git a/internal/configuration/config_manager/volume_watcher.go b/internal/configuration/config_manager/volume_watcher.go index b7a5e6215..007f7e3bb 100644 --- a/internal/configuration/config_manager/volume_watcher.go +++ b/internal/configuration/config_manager/volume_watcher.go @@ -26,7 +26,7 @@ import ( cfgcore "github.com/apecloud/kubeblocks/internal/configuration" ) -type WatchEventHandler func(event fsnotify.Event) error +type WatchEventHandler func(ctx context.Context, event fsnotify.Event) error type NotifyEventFilter func(event fsnotify.Event) (bool, error) const ( @@ -125,7 +125,7 @@ func (w *ConfigMapVolumeWatcher) loopNotifyEvent(watcher *fsnotify.Watcher, ctx continue } w.log.Debugf("volume configmap updated. event: %s, path: %s", event.Op.String(), event.Name) - runWithRetry(w.handler, event, w.retryCount, w.log) + runWithRetry(w.ctx, w.handler, event, w.retryCount, w.log) case err := <-watcher.Errors: w.log.Error(err) case <-ctx.Done(): @@ -135,10 +135,10 @@ func (w *ConfigMapVolumeWatcher) loopNotifyEvent(watcher *fsnotify.Watcher, ctx } } -func runWithRetry(handler WatchEventHandler, event fsnotify.Event, retryCount int, logger *zap.SugaredLogger) { +func runWithRetry(ctx context.Context, handler WatchEventHandler, event fsnotify.Event, retryCount int, logger *zap.SugaredLogger) { var err error for { - if err = handler(event); err == nil { + if err = handler(ctx, event); err == nil { return } retryCount-- diff --git a/internal/configuration/config_manager/volume_watcher_test.go b/internal/configuration/config_manager/volume_watcher_test.go index 4495a68a2..9cbac366d 100644 --- a/internal/configuration/config_manager/volume_watcher_test.go +++ b/internal/configuration/config_manager/volume_watcher_test.go @@ -45,7 +45,7 @@ func TestConfigMapVolumeWatcherFailed(t *testing.T) { defer volumeWatcher.Close() require.EqualError(t, volumeWatcher.Run(), "require process event handler.") - volumeWatcher.AddHandler(func(event fsnotify.Event) error { + volumeWatcher.AddHandler(func(_ context.Context, event fsnotify.Event) error { return nil }) require.Regexp(t, "no such file or directory", volumeWatcher.Run().Error()) @@ -75,7 +75,7 @@ func TestConfigMapVolumeWatcher(t *testing.T) { regexFilter, err := CreateCfgRegexFilter(`.*`) require.Nil(t, err) volumeWatcher.SetRetryCount(2). - AddHandler(func(event fsnotify.Event) error { + AddHandler(func(_ context.Context, event fsnotify.Event) error { zapLog.Info(fmt.Sprintf("handl volume event: %v", event)) retryCount++ // mock failed to handle diff --git a/internal/gotemplate/functional.go b/internal/gotemplate/functional.go index fac38eb4d..d0bc082f6 100644 --- a/internal/gotemplate/functional.go +++ b/internal/gotemplate/functional.go @@ -21,6 +21,7 @@ import ( "regexp" "strconv" + "gopkg.in/yaml.v2" corev1 "k8s.io/api/core/v1" cfgcore "github.com/apecloud/kubeblocks/internal/configuration" @@ -66,3 +67,15 @@ func regexStringSubmatch(regex string, s string) ([]string, error) { } return r.FindStringSubmatch(s), nil } + +func fromYAML(str string) (map[string]interface{}, error) { + m := map[string]interface{}{} + err := yaml.Unmarshal([]byte(str), &m) + return m, err +} + +func fromYAMLArray(str string) ([]interface{}, error) { + var a []interface{} + err := yaml.Unmarshal([]byte(str), &a) + return a, err +} diff --git a/internal/gotemplate/tpl_engine.go b/internal/gotemplate/tpl_engine.go index 233aa8429..23a068853 100644 --- a/internal/gotemplate/tpl_engine.go +++ b/internal/gotemplate/tpl_engine.go @@ -42,7 +42,9 @@ const ( ) const ( - goTemplateExtendBuildInRegexSubString = "regexStringSubmatch" + goTemplateExtendBuildInRegexSubString = "regexStringSubmatch" + goTemplateExtendBuildInFromYamlString = "fromYaml" + goTemplateExtendBuildInFromYamlArrayString = "fromYamlArray" ) type TplValues map[string]interface{} @@ -157,6 +159,8 @@ func (t *TplEngine) initSystemFunMap(funcs template.FuncMap) { // Wrap regex.FindStringSubmatch funcs[goTemplateExtendBuildInRegexSubString] = regexStringSubmatch + funcs[goTemplateExtendBuildInFromYamlString] = fromYAML + funcs[goTemplateExtendBuildInFromYamlArrayString] = fromYAMLArray t.tpl.Option(DefaultTemplateOps) t.tpl.Funcs(funcs) From a8308363ba938b104cd80562e722c10ecc77997a Mon Sep 17 00:00:00 2001 From: chantu Date: Fri, 7 Apr 2023 11:53:38 +0800 Subject: [PATCH 54/80] chore: rename conn-credential with engine type (#2407) --- apis/apps/v1alpha1/clusterdefinition_types.go | 6 ++++++ .../crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml | 6 ++++++ .../apecloud-mysql-scale/templates/clusterdefinition.yaml | 1 + deploy/apecloud-mysql/templates/clusterdefinition.yaml | 1 + deploy/clickhouse/templates/clusterdefinition.yaml | 1 + deploy/etcd/templates/clusterdefinition.yaml | 1 + .../helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml | 6 ++++++ deploy/kafka/templates/clusterdefinition.yaml | 1 + deploy/mongodb/templates/clusterdefinition.yaml | 1 + .../templates/clusterdefinition.yaml | 1 + deploy/postgresql/templates/clusterdefinition.yaml | 1 + deploy/qdrant/templates/clusterdefinition.yaml | 1 + deploy/redis/templates/clusterdefinition.yaml | 1 + deploy/weaviate/templates/clusterdefinition.yaml | 1 + internal/controller/builder/builder_test.go | 8 ++++++++ .../controller/builder/cue/conn_credential_template.cue | 3 +++ 16 files changed, 40 insertions(+) diff --git a/apis/apps/v1alpha1/clusterdefinition_types.go b/apis/apps/v1alpha1/clusterdefinition_types.go index 2e693349b..e9bd2ce22 100644 --- a/apis/apps/v1alpha1/clusterdefinition_types.go +++ b/apis/apps/v1alpha1/clusterdefinition_types.go @@ -27,6 +27,12 @@ import ( // ClusterDefinitionSpec defines the desired state of ClusterDefinition type ClusterDefinitionSpec struct { + // Cluster definition type defines well known application cluster type, e.g. mysql/redis/mongodb + // +kubebuilder:validation:MaxLength=24 + // +kubebuilder:validation:Pattern:=`^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$` + // +optional + Type string `json:"type,omitempty"` + // componentDefs provides cluster components definitions. // +kubebuilder:validation:Required // +kubebuilder:validation:MinItems=1 diff --git a/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml b/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml index 599bccf91..2916d11b2 100644 --- a/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml +++ b/config/crd/bases/apps.kubeblocks.io_clusterdefinitions.yaml @@ -8728,6 +8728,12 @@ spec: "mysql", "targetPort": "mysqlContainerPort", "port": 3306 }, and "$(SVC_PORT_mysql)" in the connection credential value is 3306.' type: object + type: + description: Cluster definition type defines well known application + cluster type, e.g. mysql/redis/mongodb + maxLength: 24 + pattern: ^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$ + type: string required: - componentDefs type: object diff --git a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml index 7f97de009..3c9cbb8db 100644 --- a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "apecloud-mysql.labels" . | nindent 4 }} spec: + type: mysql connectionCredential: username: root password: root #"$(RANDOM_PASSWD)" diff --git a/deploy/apecloud-mysql/templates/clusterdefinition.yaml b/deploy/apecloud-mysql/templates/clusterdefinition.yaml index f345d4b1f..087f8b3b8 100644 --- a/deploy/apecloud-mysql/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "apecloud-mysql.labels" . | nindent 4 }} spec: + type: mysql connectionCredential: username: root password: "$(RANDOM_PASSWD)" diff --git a/deploy/clickhouse/templates/clusterdefinition.yaml b/deploy/clickhouse/templates/clusterdefinition.yaml index 1cae0bf96..4fab723dc 100644 --- a/deploy/clickhouse/templates/clusterdefinition.yaml +++ b/deploy/clickhouse/templates/clusterdefinition.yaml @@ -11,6 +11,7 @@ metadata: annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} {{- end }} spec: + type: clickhouse connectionCredential: username: "admin" admin-password: "$(RANDOM_PASSWD)" diff --git a/deploy/etcd/templates/clusterdefinition.yaml b/deploy/etcd/templates/clusterdefinition.yaml index ce92152d5..445f6cac9 100644 --- a/deploy/etcd/templates/clusterdefinition.yaml +++ b/deploy/etcd/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "etcd.labels" . | nindent 4}} spec: + type: etcd componentDefs: - name: etcd characterType: etcd diff --git a/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml b/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml index 599bccf91..2916d11b2 100644 --- a/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml +++ b/deploy/helm/crds/apps.kubeblocks.io_clusterdefinitions.yaml @@ -8728,6 +8728,12 @@ spec: "mysql", "targetPort": "mysqlContainerPort", "port": 3306 }, and "$(SVC_PORT_mysql)" in the connection credential value is 3306.' type: object + type: + description: Cluster definition type defines well known application + cluster type, e.g. mysql/redis/mongodb + maxLength: 24 + pattern: ^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$ + type: string required: - componentDefs type: object diff --git a/deploy/kafka/templates/clusterdefinition.yaml b/deploy/kafka/templates/clusterdefinition.yaml index 042795763..61f4f5642 100644 --- a/deploy/kafka/templates/clusterdefinition.yaml +++ b/deploy/kafka/templates/clusterdefinition.yaml @@ -11,6 +11,7 @@ metadata: annotations: {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }} {{- end }} spec: + type: kafka connectionCredential: superusers: "User:admin" endpoint: "$(SVC_FQDN):$(SVC_PORT_kafka-ctrlr)" diff --git a/deploy/mongodb/templates/clusterdefinition.yaml b/deploy/mongodb/templates/clusterdefinition.yaml index 8a84e3415..a74a93036 100644 --- a/deploy/mongodb/templates/clusterdefinition.yaml +++ b/deploy/mongodb/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "mongodb.labels" . | nindent 4 }} spec: + type: mongodb connectionCredential: username: admin password: "" diff --git a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml index d3a7728da..2015c1273 100644 --- a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml +++ b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "postgresql.labels" . | nindent 4 }} spec: + type: postgresql connectionCredential: username: postgres password: "$(RANDOM_PASSWD)" diff --git a/deploy/postgresql/templates/clusterdefinition.yaml b/deploy/postgresql/templates/clusterdefinition.yaml index 26572f2a8..61c05ff01 100644 --- a/deploy/postgresql/templates/clusterdefinition.yaml +++ b/deploy/postgresql/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "postgresql.labels" . | nindent 4 }} spec: + type: postgresql connectionCredential: username: postgres password: {{ (include "postgresql.postgresPassword" .) | quote }} diff --git a/deploy/qdrant/templates/clusterdefinition.yaml b/deploy/qdrant/templates/clusterdefinition.yaml index 169468f60..e3f9533ab 100644 --- a/deploy/qdrant/templates/clusterdefinition.yaml +++ b/deploy/qdrant/templates/clusterdefinition.yaml @@ -6,6 +6,7 @@ metadata: labels: {{- include "qdrant.labels" . | nindent 4 }} spec: + type: qdrant connectionCredential: username: root password: "$(RANDOM_PASSWD)" diff --git a/deploy/redis/templates/clusterdefinition.yaml b/deploy/redis/templates/clusterdefinition.yaml index 7ba9b81b7..b84c3ce45 100644 --- a/deploy/redis/templates/clusterdefinition.yaml +++ b/deploy/redis/templates/clusterdefinition.yaml @@ -5,6 +5,7 @@ metadata: labels: {{- include "redis.labels" . | nindent 4 }} spec: + type: redis connectionCredential: username: "" password: "" diff --git a/deploy/weaviate/templates/clusterdefinition.yaml b/deploy/weaviate/templates/clusterdefinition.yaml index 5bbf5ab95..500dd3f66 100644 --- a/deploy/weaviate/templates/clusterdefinition.yaml +++ b/deploy/weaviate/templates/clusterdefinition.yaml @@ -6,6 +6,7 @@ metadata: labels: {{- include "weaviate.labels" . | nindent 4 }} spec: + type: weaviate connectionCredential: username: root password: "$(RANDOM_PASSWD)" diff --git a/internal/controller/builder/builder_test.go b/internal/controller/builder/builder_test.go index a55ebaf9d..fc1830815 100644 --- a/internal/controller/builder/builder_test.go +++ b/internal/controller/builder/builder_test.go @@ -223,6 +223,14 @@ var _ = Describe("builder", func() { credential, err := BuildConnCredential(*params) Expect(err).Should(BeNil()) Expect(credential).ShouldNot(BeNil()) + Expect(credential.Labels["apps.kubeblocks.io/cluster-type"]).Should(BeEmpty()) + By("setting type") + characterType := "test-character-type" + params.ClusterDefinition.Spec.Type = characterType + credential, err = BuildConnCredential(*params) + Expect(err).Should(BeNil()) + Expect(credential).ShouldNot(BeNil()) + Expect(credential.Labels["apps.kubeblocks.io/cluster-type"]).Should(Equal(characterType)) // "username": "root", // "SVC_FQDN": "$(SVC_FQDN)", // "RANDOM_PASSWD": "$(RANDOM_PASSWD)", diff --git a/internal/controller/builder/cue/conn_credential_template.cue b/internal/controller/builder/cue/conn_credential_template.cue index 81743e0af..6f8508711 100644 --- a/internal/controller/builder/cue/conn_credential_template.cue +++ b/internal/controller/builder/cue/conn_credential_template.cue @@ -38,6 +38,9 @@ secret: { "app.kubernetes.io/name": "\(clusterdefinition.metadata.name)" "app.kubernetes.io/instance": cluster.metadata.name "app.kubernetes.io/managed-by": "kubeblocks" + if clusterdefinition.spec.type != _|_ { + "apps.kubeblocks.io/cluster-type": clusterdefinition.spec.type + } } } } From ddff3958ed4950c6a63d376f0ffc7cd658d8b6f6 Mon Sep 17 00:00:00 2001 From: Ziang Guo Date: Fri, 7 Apr 2023 16:12:55 +0800 Subject: [PATCH 55/80] fix: install aws-loadbalancer-controller failed (#2438) --- .../templates/addons/aws-loadbalancer-controller-addon.yaml | 2 +- deploy/helm/values.yaml | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml b/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml index 90244fccc..14137410d 100644 --- a/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml +++ b/deploy/helm/templates/addons/aws-loadbalancer-controller-addon.yaml @@ -22,7 +22,7 @@ spec: key: values-kubeblocks-override.yaml setValues: - - clusterName={{ index .Values "aws-loadbalancer-controller" "cluterName " }} + - clusterName={{ index .Values "aws-loadbalancer-controller" "clusterName" }} valuesMapping: valueMap: diff --git a/deploy/helm/values.yaml b/deploy/helm/values.yaml index a129d1e92..a9902fe04 100644 --- a/deploy/helm/values.yaml +++ b/deploy/helm/values.yaml @@ -1651,6 +1651,7 @@ csi-hostpath-driver: default: true aws-loadbalancer-controller: + clusterName: "" enabled: false replicaCount: 1 tolerations: From 1a63484e6f86efa8cc11366fdb4ee0e92e900639 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Fri, 7 Apr 2023 17:19:08 +0800 Subject: [PATCH 56/80] chore: replace the volume name with template spec name and improved the prompt for kbcli (#2429) Co-authored-by: sophon-zt --- .../apps/configuration/reconfigure_policy.go | 7 +--- .../apps/configuration/simple_policy.go | 4 -- .../apps/configuration/simple_policy_test.go | 38 +++++++++---------- controllers/apps/operations/reconfigure.go | 2 +- .../apps/operations/reconfigure_test.go | 2 +- docs/user_docs/cli/kbcli_cluster_configure.md | 8 ++-- .../cli/kbcli_cluster_describe-config.md | 12 +++--- .../cli/kbcli_cluster_explain-config.md | 18 ++++----- .../cli/cmd/cluster/describe_configure.go | 34 ++++++++--------- internal/cli/cmd/cluster/describe_ops.go | 5 +-- internal/cli/cmd/cluster/errors.go | 2 +- internal/cli/cmd/cluster/operations.go | 10 ++--- internal/cli/cmd/cluster/operations_test.go | 2 +- internal/configuration/type.go | 2 +- internal/controller/plan/template_wrapper.go | 6 +-- test/integration/mysql_reconfigure_test.go | 2 +- 16 files changed, 72 insertions(+), 82 deletions(-) diff --git a/controllers/apps/configuration/reconfigure_policy.go b/controllers/apps/configuration/reconfigure_policy.go index 47ef8a358..a0d42bff7 100644 --- a/controllers/apps/configuration/reconfigure_policy.go +++ b/controllers/apps/configuration/reconfigure_policy.go @@ -130,12 +130,7 @@ func GetClientFactory() createReconfigureClient { } func (param *reconfigureParams) getConfigKey() string { - for _, configSpec := range param.Component.ConfigSpecs { - if configSpec.Name == param.ConfigSpecName { - return configSpec.VolumeName - } - } - return "" + return param.ConfigSpecName } func (param *reconfigureParams) getTargetVersionHash() string { diff --git a/controllers/apps/configuration/simple_policy.go b/controllers/apps/configuration/simple_policy.go index 3df11dbe9..0c42141ba 100644 --- a/controllers/apps/configuration/simple_policy.go +++ b/controllers/apps/configuration/simple_policy.go @@ -59,10 +59,6 @@ func rollingStatefulSets(param reconfigureParams) (ReturnedStatus, error) { progress = cfgcore.NotStarted ) - if configKey == "" { - return makeReturnedStatus(ESFailed), cfgcore.MakeError("failed to find config meta. configmap : %s", param.ConfigSpecName) - } - for _, sts := range units { if err := restartStsWithRolling(client, param.Ctx, sts, configKey, newVersion); err != nil { param.Ctx.Log.Error(err, "failed to restart statefulSet.", "stsName", sts.GetName()) diff --git a/controllers/apps/configuration/simple_policy_test.go b/controllers/apps/configuration/simple_policy_test.go index 229547d99..f13751975 100644 --- a/controllers/apps/configuration/simple_policy_test.go +++ b/controllers/apps/configuration/simple_policy_test.go @@ -171,23 +171,23 @@ var _ = Describe("Reconfigure simplePolicy", func() { }) }) - Context("simple reconfigure policy test without not configmap volume", func() { - It("Should failed", func() { - // mock not cc - mockParam := newMockReconfigureParams("simplePolicy", nil, - withMockStatefulSet(2, nil), - withConfigSpec("not_tpl_name", map[string]string{ - "key": "value", - }), - withCDComponent(appsv1alpha1.Consensus, []appsv1alpha1.ComponentConfigSpec{{ - ComponentTemplateSpec: appsv1alpha1.ComponentTemplateSpec{ - Name: "for_test", - VolumeName: "test_volume", - }}})) - status, err := simplePolicy.Upgrade(mockParam) - Expect(err).ShouldNot(Succeed()) - Expect(err.Error()).Should(ContainSubstring("failed to find config meta")) - Expect(status.Status).Should(BeEquivalentTo(ESFailed)) - }) - }) + // Context("simple reconfigure policy test without not configmap volume", func() { + // It("Should failed", func() { + // // mock not cc + // mockParam := newMockReconfigureParams("simplePolicy", nil, + // withMockStatefulSet(2, nil), + // withConfigSpec("not_tpl_name", map[string]string{ + // "key": "value", + // }), + // withCDComponent(appsv1alpha1.Consensus, []appsv1alpha1.ComponentConfigSpec{{ + // ComponentTemplateSpec: appsv1alpha1.ComponentTemplateSpec{ + // Name: "for_test", + // VolumeName: "test_volume", + // }}})) + // status, err := simplePolicy.Upgrade(mockParam) + // Expect(err).ShouldNot(Succeed()) + // Expect(err.Error()).Should(ContainSubstring("failed to find config meta")) + // Expect(status.Status).Should(BeEquivalentTo(ESFailed)) + // }) + // }) }) diff --git a/controllers/apps/operations/reconfigure.go b/controllers/apps/operations/reconfigure.go index 0f5e18d6d..8e2d9f92b 100644 --- a/controllers/apps/operations/reconfigure.go +++ b/controllers/apps/operations/reconfigure.go @@ -298,7 +298,7 @@ func (r *reconfigureAction) doMergeAndPersist(reqCtx intctrlutil.RequestCtx, cfgcore.MakeError("current configSpec not support reconfigure, configSpec: %v", configSpec.Name)) } result := updateCfgParams(config, *configSpec, client.ObjectKey{ - Name: cfgcore.GetComponentCfgName(clusterName, componentName, configSpec.VolumeName), + Name: cfgcore.GetComponentCfgName(clusterName, componentName, configSpec.Name), Namespace: resource.Cluster.Namespace, }, reqCtx.Ctx, cli, resource.OpsRequest.Name) if result.err != nil { diff --git a/controllers/apps/operations/reconfigure_test.go b/controllers/apps/operations/reconfigure_test.go index 1a9bcf698..03e828292 100644 --- a/controllers/apps/operations/reconfigure_test.go +++ b/controllers/apps/operations/reconfigure_test.go @@ -89,7 +89,7 @@ var _ = Describe("Reconfigure OpsRequest", func() { } var cmObj *corev1.ConfigMap for _, configSpec := range cdComponent.ConfigSpecs { - cmInsName := cfgcore.GetComponentCfgName(clusterName, componentName, configSpec.VolumeName) + cmInsName := cfgcore.GetComponentCfgName(clusterName, componentName, configSpec.Name) cfgCM := testapps.NewCustomizedObj("operations_config/config-template.yaml", &corev1.ConfigMap{}, testapps.WithNamespacedName(cmInsName, ns), diff --git a/docs/user_docs/cli/kbcli_cluster_configure.md b/docs/user_docs/cli/kbcli_cluster_configure.md index d84470b5e..aba8a4d95 100644 --- a/docs/user_docs/cli/kbcli_cluster_configure.md +++ b/docs/user_docs/cli/kbcli_cluster_configure.md @@ -12,21 +12,21 @@ kbcli cluster configure [flags] ``` # update component params - kbcli cluster configure --component= --template-name= --configure-file= --set max_connections=1000,general_log=OFF + kbcli cluster configure --component= --config-spec= --config-file= --set max_connections=1000,general_log=OFF # update mysql max_connections, cluster name is mycluster - kbcli cluster configure mycluster --component=mysql --template-name=mysql-3node-tpl --configure-file=my.cnf --set max_connections=2000 + kbcli cluster configure mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf --set max_connections=2000 ``` ### Options ``` --component strings Specify the name of Component to be updated. If the cluster has only one component, unset the parameter. - --configure-file string Specify the name of the configuration file to be updated (e.g. for mysql: --configure-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. + --config-file string Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. + --config-spec string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. -h, --help help for configure --name string OpsRequest name. if not specified, it will be randomly generated --set strings Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-configure'. - --template-name string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --template-name=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. --ttlSecondsAfterSucceed int Time to live after the OpsRequest succeed ``` diff --git a/docs/user_docs/cli/kbcli_cluster_describe-config.md b/docs/user_docs/cli/kbcli_cluster_describe-config.md index ff35c681a..eb5d4e978 100644 --- a/docs/user_docs/cli/kbcli_cluster_describe-config.md +++ b/docs/user_docs/cli/kbcli_cluster_describe-config.md @@ -21,17 +21,17 @@ kbcli cluster describe-config [flags] kbcli cluster describe-configure mycluster --component-name=mysql --show-detail # describe a content of configuration file. - kbcli cluster describe-configure mycluster --component-name=mysql --configure-file=my.cnf --show-detail + kbcli cluster describe-configure mycluster --component-name=mysql --config-file=my.cnf --show-detail ``` ### Options ``` - --component-name string Specify the name of Component to be describe (e.g. for apecloud-mysql: --component-name=mysql). If the cluster has only one component, unset the parameter." - --configure-file strings Specify the name of the configuration file to be describe (e.g. for mysql: --configure-file=my.cnf). If unset, all files. - -h, --help help for describe-config - --show-detail If true, the content of the files specified by configure-file will be printed. - --template-names strings Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --template-names=mysql-3node-tpl) + --component-name string Specify the name of Component to be describe (e.g. for apecloud-mysql: --component-name=mysql). If the cluster has only one component, unset the parameter." + --config-file strings Specify the name of the configuration file to be describe (e.g. for mysql: --config-file=my.cnf). If unset, all files. + --config-specs strings Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --config-specs=mysql-3node-tpl) + -h, --help help for describe-config + --show-detail If true, the content of the files specified by config-file will be printed. ``` ### Options inherited from parent commands diff --git a/docs/user_docs/cli/kbcli_cluster_explain-config.md b/docs/user_docs/cli/kbcli_cluster_explain-config.md index b3f3cb97d..c678b02a1 100644 --- a/docs/user_docs/cli/kbcli_cluster_explain-config.md +++ b/docs/user_docs/cli/kbcli_cluster_explain-config.md @@ -15,24 +15,24 @@ kbcli cluster explain-config [flags] kbcli cluster explain-configure mycluster # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl --trunc-document=false --trunc-enum=false + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false # describe a specified parameters, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl --param=sql_mode + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode ``` ### Options ``` - --component-name string Specify the name of Component to be describe (e.g. for apecloud-mysql: --component-name=mysql). If the cluster has only one component, unset the parameter." - -h, --help help for explain-config - --param string Specify the name of parameter to be query. It clearly display the details of the parameter. - --template-names strings Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --template-names=mysql-3node-tpl) - --trunc-document If the document length of the parameter is greater than 100, it will be truncated. - --trunc-enum If the value list length of the parameter is greater than 20, it will be truncated. (default true) + --component-name string Specify the name of Component to be describe (e.g. for apecloud-mysql: --component-name=mysql). If the cluster has only one component, unset the parameter." + --config-specs strings Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --config-specs=mysql-3node-tpl) + -h, --help help for explain-config + --param string Specify the name of parameter to be query. It clearly display the details of the parameter. + --trunc-document If the document length of the parameter is greater than 100, it will be truncated. + --trunc-enum If the value list length of the parameter is greater than 20, it will be truncated. (default true) ``` ### Options inherited from parent commands diff --git a/internal/cli/cmd/cluster/describe_configure.go b/internal/cli/cmd/cluster/describe_configure.go index 669507e8d..eaed3cc88 100644 --- a/internal/cli/cmd/cluster/describe_configure.go +++ b/internal/cli/cmd/cluster/describe_configure.go @@ -52,7 +52,7 @@ type reconfigureOptions struct { clusterName string componentName string - templateNames []string + configSpecs []string isExplain bool truncEnum bool @@ -98,19 +98,19 @@ var ( kbcli cluster describe-configure mycluster --component-name=mysql --show-detail # describe a content of configuration file. - kbcli cluster describe-configure mycluster --component-name=mysql --configure-file=my.cnf --show-detail`) + kbcli cluster describe-configure mycluster --component-name=mysql --config-file=my.cnf --show-detail`) explainReconfigureExample = templates.Examples(` # describe a cluster, e.g. cluster name is mycluster kbcli cluster explain-configure mycluster # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl --trunc-document=false --trunc-enum=false + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false # describe a specified parameters, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --template-names=mysql-3node-tpl --param=sql_mode`) + kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode`) diffConfigureExample = templates.Examples(` # compare config files kbcli cluster diff-configure opsrequest1 opsrequest2`) @@ -118,7 +118,7 @@ var ( func (r *reconfigureOptions) addCommonFlags(cmd *cobra.Command) { cmd.Flags().StringVar(&r.componentName, "component-name", "", "Specify the name of Component to be describe (e.g. for apecloud-mysql: --component-name=mysql). If the cluster has only one component, unset the parameter.\"") - cmd.Flags().StringSliceVar(&r.templateNames, "template-names", nil, "Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --template-names=mysql-3node-tpl)") + cmd.Flags().StringSliceVar(&r.configSpecs, "config-specs", nil, "Specify the name of the configuration template to be describe. (e.g. for apecloud-mysql: --config-specs=mysql-3node-tpl)") } func (r *reconfigureOptions) validate() error { @@ -132,11 +132,11 @@ func (r *reconfigureOptions) validate() error { return err } - if r.isExplain && len(r.templateNames) != 1 { + if r.isExplain && len(r.configSpecs) != 1 { return cfgcore.MakeError("explain require one template") } - for _, tplName := range r.templateNames { + for _, tplName := range r.configSpecs { tpl, err := r.findTemplateByName(tplName) if err != nil { return err @@ -171,7 +171,7 @@ func (r *reconfigureOptions) complete2(args []string) error { if err := r.syncClusterComponent(); err != nil { return err } - if len(r.templateNames) != 0 { + if len(r.configSpecs) != 0 { return nil } if err := r.syncComponentCfgTpl(); err != nil { @@ -186,7 +186,7 @@ func (r *reconfigureOptions) complete2(args []string) error { for _, tpl := range r.tpls { templateNames = append(templateNames, tpl.Name) } - r.templateNames = templateNames + r.configSpecs = templateNames return nil } @@ -196,7 +196,7 @@ func (r *reconfigureOptions) complete2(args []string) error { templateNames = append(templateNames, tpl.Name) } } - r.templateNames = templateNames + r.configSpecs = templateNames return nil } @@ -245,7 +245,7 @@ func (r *reconfigureOptions) printDescribeReconfigure() error { } func (r *reconfigureOptions) printAllExplainConfigure() error { - for _, templateName := range r.templateNames { + for _, templateName := range r.configSpecs { fmt.Println("template meta:") printer.PrintLineWithTabSeparator( printer.NewPair(" ConfigSpec", templateName), @@ -297,12 +297,12 @@ func (r *reconfigureOptions) printExplainConfigure(tplName string) error { func (r *reconfigureOptions) getReconfigureMeta() ([]types.ConfigTemplateInfo, error) { configs := make([]types.ConfigTemplateInfo, 0) - for _, tplName := range r.templateNames { + for _, tplName := range r.configSpecs { // checked by validate tpl, _ := r.findTemplateByName(tplName) // fetch config configmap cmObj := &corev1.ConfigMap{} - cmName := cfgcore.GetComponentCfgName(r.clusterName, r.componentName, tpl.VolumeName) + cmName := cfgcore.GetComponentCfgName(r.clusterName, r.componentName, tpl.Name) if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), client.ObjectKey{ Name: cmName, Namespace: r.namespace, @@ -319,7 +319,7 @@ func (r *reconfigureOptions) getReconfigureMeta() ([]types.ConfigTemplateInfo, e } func (r *reconfigureOptions) printConfigureContext(configs []types.ConfigTemplateInfo) { - printer.PrintTitle("Configures Context[${component-name}/${template-name}/${file-name}]") + printer.PrintTitle("Configures Context[${component-name}/${config-spec}/${file-name}]") keys := set.NewLinkedHashSetString(r.keys...) for _, info := range configs { @@ -828,8 +828,8 @@ func NewDescribeReconfigureCmd(f cmdutil.Factory, streams genericclioptions.IOSt }, } o.addCommonFlags(cmd) - cmd.Flags().BoolVar(&o.showDetail, "show-detail", o.showDetail, "If true, the content of the files specified by configure-file will be printed.") - cmd.Flags().StringSliceVar(&o.keys, "configure-file", nil, "Specify the name of the configuration file to be describe (e.g. for mysql: --configure-file=my.cnf). If unset, all files.") + cmd.Flags().BoolVar(&o.showDetail, "show-detail", o.showDetail, "If true, the content of the files specified by config-file will be printed.") + cmd.Flags().StringSliceVar(&o.keys, "config-file", nil, "Specify the name of the configuration file to be describe (e.g. for mysql: --config-file=my.cnf). If unset, all files.") return cmd } diff --git a/internal/cli/cmd/cluster/describe_ops.go b/internal/cli/cmd/cluster/describe_ops.go index f90ddb96f..4f03a3a64 100644 --- a/internal/cli/cmd/cluster/describe_ops.go +++ b/internal/cli/cmd/cluster/describe_ops.go @@ -61,8 +61,7 @@ type describeOpsOptions struct { } type opsObject interface { - appsv1alpha1.VerticalScaling | appsv1alpha1.HorizontalScaling | - appsv1alpha1.OpsRequestVolumeClaimTemplate | appsv1alpha1.VolumeExpansion + appsv1alpha1.VerticalScaling | appsv1alpha1.HorizontalScaling | appsv1alpha1.OpsRequestVolumeClaimTemplate | appsv1alpha1.VolumeExpansion } func newDescribeOpsOptions(f cmdutil.Factory, streams genericclioptions.IOStreams) *describeOpsOptions { @@ -343,7 +342,7 @@ func (o *describeOpsOptions) getReconfiguringCommand(spec appsv1alpha1.OpsReques commandArgs = append(commandArgs, "configure") commandArgs = append(commandArgs, spec.ClusterRef) commandArgs = append(commandArgs, fmt.Sprintf("--component-names=%s", componentName)) - commandArgs = append(commandArgs, fmt.Sprintf("--template-name=%s", configuration.Name)) + commandArgs = append(commandArgs, fmt.Sprintf("--config-spec=%s", configuration.Name)) config := configuration.Keys[0] commandArgs = append(commandArgs, fmt.Sprintf("--config-file=%s", config.Key)) diff --git a/internal/cli/cmd/cluster/errors.go b/internal/cli/cmd/cluster/errors.go index a5dcf079f..23fdf7466 100644 --- a/internal/cli/cmd/cluster/errors.go +++ b/internal/cli/cmd/cluster/errors.go @@ -25,7 +25,7 @@ var ( missingUpdatedParametersErrMessage = "missing updated parameters, using --help." multiComponentsErrorMessage = "when multi component exist, must specify which component to use. Please using --component-name" - multiConfigTemplateErrorMessage = "when multi config template exist, must specify which config template to use. Please using --template-name" + multiConfigTemplateErrorMessage = "when multi config template exist, must specify which config template to use. Please using --config-spec" notFoundValidConfigTemplateErrorMessage = "not find valid config template, component[name=%s] in the cluster[name=%s]" diff --git a/internal/cli/cmd/cluster/operations.go b/internal/cli/cmd/cluster/operations.go index 1f2dc2ae4..0c7b6a65c 100644 --- a/internal/cli/cmd/cluster/operations.go +++ b/internal/cli/cmd/cluster/operations.go @@ -100,10 +100,10 @@ func newBaseOperationsOptions(streams genericclioptions.IOStreams, opsType appsv var ( createReconfigureExample = templates.Examples(` # update component params - kbcli cluster configure --component= --template-name= --configure-file= --set max_connections=1000,general_log=OFF + kbcli cluster configure --component= --config-spec= --config-file= --set max_connections=1000,general_log=OFF # update mysql max_connections, cluster name is mycluster - kbcli cluster configure mycluster --component=mysql --template-name=mysql-3node-tpl --configure-file=my.cnf --set max_connections=2000 + kbcli cluster configure mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf --set max_connections=2000 `) ) @@ -284,7 +284,7 @@ func (o *OperationsOptions) validateTemplateParam(tpls []appsv1alpha1.ComponentC func (o *OperationsOptions) validateConfigMapKey(tpl *appsv1alpha1.ComponentConfigSpec, componentName string) error { var ( cmObj = corev1.ConfigMap{} - cmName = cfgcore.GetComponentCfgName(o.Name, componentName, tpl.VolumeName) + cmName = cfgcore.GetComponentCfgName(o.Name, componentName, tpl.Name) ) if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), client.ObjectKey{ @@ -692,8 +692,8 @@ func NewReconfigureCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) * o.buildCommonFlags(cmd) cmd.Flags().StringSliceVar(&o.Parameters, "set", nil, "Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-configure'.") cmd.Flags().StringSliceVar(&o.ComponentNames, "component", nil, "Specify the name of Component to be updated. If the cluster has only one component, unset the parameter.") - cmd.Flags().StringVar(&o.CfgTemplateName, "template-name", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --template-name=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") - cmd.Flags().StringVar(&o.CfgFile, "configure-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --configure-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") + cmd.Flags().StringVar(&o.CfgTemplateName, "config-spec", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") + cmd.Flags().StringVar(&o.CfgFile, "config-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") } inputs.Complete = o.fillTemplateArgForReconfiguring return create.BuildCommand(inputs) diff --git a/internal/cli/cmd/cluster/operations_test.go b/internal/cli/cmd/cluster/operations_test.go index f8634ccb6..2f489e2e1 100644 --- a/internal/cli/cmd/cluster/operations_test.go +++ b/internal/cli/cmd/cluster/operations_test.go @@ -165,7 +165,7 @@ var _ = Describe("operations", func() { configmap := testapps.NewCustomizedObj("resources/mysql-config-template.yaml", &corev1.ConfigMap{}, testapps.WithNamespace(ns)) constraint := testapps.NewCustomizedObj("resources/mysql-config-constraint.yaml", &appsv1alpha1.ConfigConstraint{}) - componentConfig := testapps.NewConfigMap(ns, cfgcore.GetComponentCfgName(clusterName, statefulCompName, configVolumeName), testapps.SetConfigMapData("my.cnf", "")) + componentConfig := testapps.NewConfigMap(ns, cfgcore.GetComponentCfgName(clusterName, statefulCompName, configSpecName), testapps.SetConfigMapData("my.cnf", "")) By("Create a clusterDefinition obj") clusterDefObj := testapps.NewClusterDefFactory(clusterDefName). AddComponent(testapps.StatefulMySQLComponent, statefulCompType). diff --git a/internal/configuration/type.go b/internal/configuration/type.go index cf048fb66..ec51c6762 100644 --- a/internal/configuration/type.go +++ b/internal/configuration/type.go @@ -155,7 +155,7 @@ func GenerateConstraintsUniqLabelKeyWithConfig(configKey string) string { // GetInstanceCMName {{statefull.Name}}-{{clusterVersion.Name}}-{{tpl.Name}}-"config" func GetInstanceCMName(obj client.Object, tpl *appsv1alpha1.ComponentTemplateSpec) string { - return getInstanceCfgCMName(obj.GetName(), tpl.VolumeName) + return getInstanceCfgCMName(obj.GetName(), tpl.Name) // return fmt.Sprintf("%s-%s-config", sts.GetName(), tpl.VolumeName) } diff --git a/internal/controller/plan/template_wrapper.go b/internal/controller/plan/template_wrapper.go index dba2c2684..65ff48a53 100644 --- a/internal/controller/plan/template_wrapper.go +++ b/internal/controller/plan/template_wrapper.go @@ -20,7 +20,6 @@ import ( "context" "strings" - "github.com/apecloud/kubeblocks/internal/generics" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/runtime" @@ -32,6 +31,7 @@ import ( "github.com/apecloud/kubeblocks/internal/constant" "github.com/apecloud/kubeblocks/internal/controller/builder" intctrltypes "github.com/apecloud/kubeblocks/internal/controller/types" + "github.com/apecloud/kubeblocks/internal/generics" ) type templateRenderValidator = func(map[string]string) error @@ -94,7 +94,7 @@ func (wrapper *renderWrapper) renderConfigTemplate(task *intctrltypes.ReconcileT scheme, _ := appsv1alpha1.SchemeBuilder.Build() for _, configSpec := range task.Component.ConfigTemplates { - cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, configSpec.VolumeName) + cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, configSpec.Name) if enableRerender, err = wrapper.enableRerenderTemplateSpec(cmName, task); err != nil { return err } @@ -122,7 +122,7 @@ func (wrapper *renderWrapper) renderConfigTemplate(task *intctrltypes.ReconcileT func (wrapper *renderWrapper) renderScriptTemplate(task *intctrltypes.ReconcileTask) error { scheme, _ := appsv1alpha1.SchemeBuilder.Build() for _, templateSpec := range task.Component.ScriptTemplates { - cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, templateSpec.VolumeName) + cmName := cfgcore.GetComponentCfgName(task.Cluster.Name, task.Component.Name, templateSpec.Name) if task.GetLocalResourceWithObjectKey(client.ObjectKey{ Name: cmName, Namespace: wrapper.cluster.Namespace, diff --git a/test/integration/mysql_reconfigure_test.go b/test/integration/mysql_reconfigure_test.go index f3fdec389..d0fc56576 100644 --- a/test/integration/mysql_reconfigure_test.go +++ b/test/integration/mysql_reconfigure_test.go @@ -130,7 +130,7 @@ var _ = Describe("MySQL Reconfigure function", func() { Expect(len(validTpls) > 0).Should(BeTrue()) cmObj = &corev1.ConfigMap{} - cmName := cfgcore.GetComponentCfgName(clusterObj.Name, componentName, tpls[0].VolumeName) + cmName := cfgcore.GetComponentCfgName(clusterObj.Name, componentName, tpls[0].Name) err = cliutil.GetResourceObjectFromGVR(clitypes.ConfigmapGVR(), client.ObjectKey{ Name: cmName, Namespace: testCtx.DefaultNamespace, From 9eef9a36a55c015081a3aeabf2382c7c6e6f9409 Mon Sep 17 00:00:00 2001 From: yuanyuan zhang <111744220+michelle-0808@users.noreply.github.com> Date: Fri, 7 Apr 2023 17:50:50 +0800 Subject: [PATCH 57/80] docs: fix yaml bugs (#2434) --- .../create-and-connect-a-mysql-cluster.md | 5 ++-- .../cluster-management/expand-volume.md | 10 ++++---- .../scale-for-apecloud-mysql.md | 23 +++++++++---------- .../start-stop-a-cluster.md | 18 ++++++++------- ...create-and-connect-a-postgresql-cluster.md | 5 ++-- .../cluster-management/expand-volume.md | 10 ++++---- .../scale-for-postgresql.md | 17 +++++++------- .../start-stop-a-cluster.md | 18 ++++++++------- 8 files changed, 55 insertions(+), 51 deletions(-) diff --git a/docs/user_docs/kubeblocks-for-mysql/cluster-management/create-and-connect-a-mysql-cluster.md b/docs/user_docs/kubeblocks-for-mysql/cluster-management/create-and-connect-a-mysql-cluster.md index 003fc753a..381e6973c 100644 --- a/docs/user_docs/kubeblocks-for-mysql/cluster-management/create-and-connect-a-mysql-cluster.md +++ b/docs/user_docs/kubeblocks-for-mysql/cluster-management/create-and-connect-a-mysql-cluster.md @@ -92,10 +92,10 @@ sidebar_label: Create and connect Change the corresponding parameters in the YAML file. ```bash - kbcli cluster create mysql-cluster --cluster-definition="apecloud-mysql" --set -<```kbcli cluster create --cluster-definition='apecloud-mysql' --node-labels='"topology.kubernetes.io/zone=us-east-1a","disktype=ssd,essd"'``` | | `--set` | It sets the cluster resource including CPU, memory, replicas, and storage, each set corresponds to a component. For example, `--set cpu=1000m,memory=1Gi,replicas=3,storage=10Gi`. | +| `--set-file` | It uses a yaml file, URL, or stdin to set the cluster resource. | | `--termination-policy` | It specifies the termination policy of the cluster. There are four available values, namely `DoNotTerminate`, `Halt`, `Delete`, and `WipeOut`. `Delete` is set as the default.
- `DoNotTerminate`: DoNotTerminate blocks the delete operation.
- `Halt`: Halt deletes workload resources such as statefulset, deployment workloads but keeps PVCs.
- `Delete`: Delete is based on Halt and deletes PVCs.
- `WipeOut`: WipeOut is based on Delete and wipes out all volume snapshots and snapshot data from backup storage location. | ## Connect to a MySQL Cluster diff --git a/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md b/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md index 337461593..d920a0e79 100644 --- a/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md +++ b/docs/user_docs/kubeblocks-for-mysql/cluster-management/expand-volume.md @@ -57,8 +57,8 @@ spec: volumeExpansion: - componentName: mysql volumeClaimTemplates: - - name: data - storage: "2Gi" + - name: data + storage: "2Gi" EOF ``` @@ -75,9 +75,9 @@ metadata: spec: clusterDefinitionRef: apecloud-mysql clusterVersionRef: ac-mysql-8.0.30 - components: + componentSpecs: - name: mysql - type: mysql + componentDefRef: mysql replicas: 1 volumeClaimTemplates: - name: data @@ -87,5 +87,5 @@ spec: resources: requests: storage: 1Gi # Change the volume storage size. -terminationPolicy: Halt + terminationPolicy: Halt ``` \ No newline at end of file diff --git a/docs/user_docs/kubeblocks-for-mysql/cluster-management/scale-for-apecloud-mysql.md b/docs/user_docs/kubeblocks-for-mysql/cluster-management/scale-for-apecloud-mysql.md index b088adb67..d0f765669 100644 --- a/docs/user_docs/kubeblocks-for-mysql/cluster-management/scale-for-apecloud-mysql.md +++ b/docs/user_docs/kubeblocks-for-mysql/cluster-management/scale-for-apecloud-mysql.md @@ -38,19 +38,18 @@ mysql-cluster default apecloud-mysql ac-mysql-8.0.30 **Option 1.** (**Recommended**) Use kbcli - Configure the parameters `component-names`, `requests`, and `limits` and run the command. + Configure the parameters `--component-names`, `--memory`, and `--cpu` and run the command. ***Example*** ```bash kbcli cluster vscale mysql-cluster \ --component-names="mysql" \ - --requests.memory="2Gi" --requests.cpu="1" \ - --limits.memory="4Gi" --limits.cpu="2" + --memory="4Gi" --cpu="2" \ ``` - - `component-names` describes the component name ready for vertical scaling. - - `requests` describes the minimum amount of computing resources required. If `requests` is omitted for a container, it uses the `limits` value if `limits` is explicitly specified, otherwise uses an implementation-defined value. For more details, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). - - `--limits` describes the maximum amount of computing resources allowed. For more details, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) + - `--component-names` describes the component name ready for vertical scaling. + - `--memory` describes the requested and limited size of the component memory. + - `--cpu` describes the requested and limited size of the component CPU. **Option 2.** Create an OpsRequest @@ -90,9 +89,9 @@ mysql-cluster default apecloud-mysql ac-mysql-8.0.30 spec: clusterDefinitionRef: apecloud-mysql clusterVersionRef: ac-mysql-8.0.30 - components: + componentSpecs: - name: mysql - type: mysql + componentDefRef: mysql replicas: 1 resources: # Change the values of resources. requests: @@ -158,7 +157,7 @@ Horizontal scaling changes the amount of pods. For example, you can apply horizo **Option 1.** (**Recommended**) Use kbcli - Configure the parameters `component-names` and `replicas`, and run the command. + Configure the parameters `--component-names` and `--replicas`, and run the command. ***Example*** @@ -167,7 +166,7 @@ Horizontal scaling changes the amount of pods. For example, you can apply horizo --component-names="mysql" --replicas=3 ``` - `--component-names` describes the component name ready for vertical scaling. - - `--replicas` describe the replicas with the specified components. + - `--replicas` describes the replicas with the specified components. **Option 2.** Create an OpsRequest @@ -206,9 +205,9 @@ Horizontal scaling changes the amount of pods. For example, you can apply horizo spec: clusterDefinitionRef: apecloud-mysql clusterVersionRef: ac-mysql-8.0.30 - components: + componentSpecs: - name: mysql - type: mysql + componentDefRef: mysql replicas: 1 # Change the pod amount. volumeClaimTemplates: - name: data diff --git a/docs/user_docs/kubeblocks-for-mysql/cluster-management/start-stop-a-cluster.md b/docs/user_docs/kubeblocks-for-mysql/cluster-management/start-stop-a-cluster.md index dc5dde5ec..d8e79e3bb 100644 --- a/docs/user_docs/kubeblocks-for-mysql/cluster-management/start-stop-a-cluster.md +++ b/docs/user_docs/kubeblocks-for-mysql/cluster-management/start-stop-a-cluster.md @@ -30,10 +30,11 @@ kbcli cluster stop mysql-cluster Run the command below to stop a cluster. ```bash kubectl apply -f - <```kbcli cluster create --cluster-definition='postgresql' --node-labels='"topology.kubernetes.io/zone=us-east-1a","disktype=ssd,essd"'``` | | `--set` | It sets the cluster resource including CPU, memory, and storage, each set corresponds to a component. For example, `--set cpu=1000m,memory=1Gi,storage=10Gi`. | +| `--set-file` | It uses a yaml file, URL, or stdin to set the cluster resource. | | `--termination-policy` | It specifies the termination policy of the cluster. There are four available values, namely `DoNotTerminate`, `Halt`, `Delete`, and `WipeOut`. `Delete` is set as the default.
- `DoNotTerminate`: DoNotTerminate blocks the delete operation.
- `Halt`: Halt deletes workload resources such as statefulset, deployment workloads but keeps PVCs.
- `Delete`: Delete is based on Halt and deletes PVCs.
- `WipeOut`: WipeOut is based on Delete and wipes out all volume snapshots and snapshot data from backup storage location. | ## Connect to a PostgreSQL Cluster diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md index 4948d4b66..cc895af01 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/expand-volume.md @@ -57,8 +57,8 @@ spec: volumeExpansion: - componentName: pg-replication volumeClaimTemplates: - - name: data - storage: "2Gi" + - name: data + storage: "2Gi" EOF ``` @@ -75,9 +75,9 @@ metadata: spec: clusterDefinitionRef: postgresql clusterVersionRef: postgresql-14.7.0 - components: + componentSpecs: - name: pg-replication - type: postgresql + componentDefRef: postgresql replicas: 1 volumeClaimTemplates: - name: data @@ -87,5 +87,5 @@ spec: resources: requests: storage: 1Gi # Change the volume storage size. -terminationPolicy: Halt + terminationPolicy: Halt ``` \ No newline at end of file diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md index 20f59e62e..0f1c18cbe 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/scale-for-postgresql.md @@ -1,6 +1,6 @@ --- title: Scale for PostgreSQL -description: How to scale a MPostgreSQL cluster, horizontal scaling, vertical scaling +description: How to scale a PostgreSQL cluster, horizontal scaling, vertical scaling sidebar_position: 2 sidebar_label: Scale --- @@ -39,19 +39,18 @@ pg-cluster default postgresql-cluster postgresql-14.7.0 Delete **Option 1.** (**Recommended**) Use kbcli - Configure the parameters `component-names`, `requests`, and `limits` and run the command. + Configure the parameters `--component-names`, `--memory`, and `--cpu` and run the command. ***Example*** ```bash kbcli cluster vscale pg-cluster \ --component-names="pg-replication" \ - --requests.memory="2Gi" --requests.cpu="1" \ - --limits.memory="4Gi" --limits.cpu="2" + --memory="4Gi" --cpu="2" \ ``` - - `component-names` describes the component name ready for vertical scaling. - - `requests` describes the minimum amount of computing resources required. If `requests` is omitted for a container, it uses the `limits` value if `limits` is explicitly specified, otherwise uses an implementation-defined value. For more details, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). - - `--limits` describes the maximum amount of computing resources allowed. For more details, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) + - `--component-names` describes the component name ready for vertical scaling. + - `--memory` describes the requested and limited size of the component memory. + - `--cpu` describes the requested and limited size of the component CPU. **Option 2.** Create an OpsRequest @@ -91,9 +90,9 @@ pg-cluster default postgresql-cluster postgresql-14.7.0 Delete spec: clusterDefinitionRef: postgresql-cluster clusterVersionRef: postgre-14.7.0 - components: + componentSpecs: - name: pg-replication - type: postgresql + componentDefRef: postgresql replicas: 1 resources: # Change the values of resources. requests: diff --git a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md index ed231a092..e631f79ac 100644 --- a/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md +++ b/docs/user_docs/kubeblocks-for-postgresql/cluster-management/start-stop-a-cluster.md @@ -28,10 +28,11 @@ kbcli cluster stop pg-cluster Run the command below to stop a cluster. ```bash kubectl apply -f - < Date: Sat, 8 Apr 2023 15:12:21 +0800 Subject: [PATCH 58/80] fix: invalid class check and typo (#2443) --- docs/user_docs/cli/kbcli_class_create.md | 2 +- internal/cli/cmd/class/create.go | 2 +- internal/controller/lifecycle/transformer_fill_class.go | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user_docs/cli/kbcli_class_create.md b/docs/user_docs/cli/kbcli_class_create.md index 966cd13d7..46dcf7849 100644 --- a/docs/user_docs/cli/kbcli_class_create.md +++ b/docs/user_docs/cli/kbcli_class_create.md @@ -22,7 +22,7 @@ kbcli class create [NAME] [flags] ``` --class-family string Specify class family - --cluster-definition string Specify cluster definition, run "kbcli cluster-definition list" to show all available cluster definition + --cluster-definition string Specify cluster definition, run "kbcli clusterdefinition list" to show all available cluster definition --cpu string Specify component cpu cores --file string Specify file path which contains YAML definition of class -h, --help help for create diff --git a/internal/cli/cmd/class/create.go b/internal/cli/cmd/class/create.go index 1e809d7a0..ac82d22fa 100644 --- a/internal/cli/cmd/class/create.go +++ b/internal/cli/cmd/class/create.go @@ -76,7 +76,7 @@ func NewCreateCommand(f cmdutil.Factory, streams genericclioptions.IOStreams) *c util.CheckErr(o.run()) }, } - cmd.Flags().StringVar(&o.ClusterDefRef, "cluster-definition", "", "Specify cluster definition, run \"kbcli cluster-definition list\" to show all available cluster definition") + cmd.Flags().StringVar(&o.ClusterDefRef, "cluster-definition", "", "Specify cluster definition, run \"kbcli clusterdefinition list\" to show all available cluster definition") util.CheckErr(cmd.MarkFlagRequired("cluster-definition")) cmd.Flags().StringVar(&o.ComponentType, "type", "", "Specify component type") util.CheckErr(cmd.MarkFlagRequired("type")) diff --git a/internal/controller/lifecycle/transformer_fill_class.go b/internal/controller/lifecycle/transformer_fill_class.go index 60616cf42..55ee0c104 100644 --- a/internal/controller/lifecycle/transformer_fill_class.go +++ b/internal/controller/lifecycle/transformer_fill_class.go @@ -86,10 +86,10 @@ func (r *fillClass) fillClass(reqCtx intctrlutil.RequestCtx, cluster *appsv1alph matchComponentClass := func(comp appsv1alpha1.ClusterComponentSpec, classes map[string]*class.ComponentClass) *class.ComponentClass { filters := class.Filters(make(map[string]resource.Quantity)) - if comp.Resources.Requests.Cpu() != nil { + if !comp.Resources.Requests.Cpu().IsZero() { filters[corev1.ResourceCPU.String()] = *comp.Resources.Requests.Cpu() } - if comp.Resources.Requests.Memory() != nil { + if !comp.Resources.Requests.Memory().IsZero() { filters[corev1.ResourceMemory.String()] = *comp.Resources.Requests.Memory() } return class.ChooseComponentClasses(classes, filters) From 2261ad21b1890e1eafd0764a6f47719fa38ddceb Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Sun, 9 Apr 2023 23:28:44 +0800 Subject: [PATCH 59/80] chore: tidy up apis/... packages, receiver functions, uses struct type for read-only public functions for small struct (#2445) --- Makefile | 6 +- apis/apps/v1alpha1/clusterversion_types.go | 6 +- apis/apps/v1alpha1/opsrequest_types.go | 149 +++++++++--------- apis/apps/v1alpha1/opsrequest_types_test.go | 32 ++-- apis/apps/v1alpha1/type.go | 2 + apis/extensions/v1alpha1/addon_types.go | 19 +-- apis/extensions/v1alpha1/addon_types_test.go | 8 +- controllers/apps/operations/expose.go | 13 +- .../apps/operations/horizontal_scaling.go | 6 +- controllers/apps/operations/ops_util.go | 2 +- controllers/apps/operations/reconfigure.go | 2 +- controllers/apps/operations/restart.go | 4 +- controllers/apps/operations/upgrade.go | 2 +- .../apps/operations/vertical_scaling.go | 8 +- .../apps/operations/volume_expansion.go | 8 +- githooks/scripts/run-lint.sh | 2 +- .../lifecycle/transformer_cluster.go | 2 +- 17 files changed, 130 insertions(+), 141 deletions(-) diff --git a/Makefile b/Makefile index 2dc5e3f82..7e8572173 100644 --- a/Makefile +++ b/Makefile @@ -157,13 +157,13 @@ cue-fmt: cuetool ## Run cue fmt against code. git ls-files --exclude-standard | grep "\.cue$$" | xargs $(CUE) fmt git ls-files --exclude-standard | grep "\.cue$$" | xargs $(CUE) fix -.PHONY: fast-lint -fast-lint: golangci staticcheck vet # [INTERNAL] fast lint +.PHONY: lint-fast +lint-fast: golangci staticcheck vet # [INTERNAL] fast lint $(GOLANGCILINT) run ./... .PHONY: lint lint: test-go-generate generate ## Run golangci-lint against code. - $(MAKE) fast-lint + $(MAKE) lint-fast .PHONY: staticcheck staticcheck: staticchecktool ## Run staticcheck against code. diff --git a/apis/apps/v1alpha1/clusterversion_types.go b/apis/apps/v1alpha1/clusterversion_types.go index d01f3d04a..5e233c5d9 100644 --- a/apis/apps/v1alpha1/clusterversion_types.go +++ b/apis/apps/v1alpha1/clusterversion_types.go @@ -136,10 +136,10 @@ func init() { } // GetDefNameMappingComponents returns ComponentDefRef name mapping ClusterComponentVersion. -func (r *ClusterVersion) GetDefNameMappingComponents() map[string]*ClusterComponentVersion { +func (r ClusterVersionSpec) GetDefNameMappingComponents() map[string]*ClusterComponentVersion { m := map[string]*ClusterComponentVersion{} - for i, c := range r.Spec.ComponentVersions { - m[c.ComponentDefRef] = &r.Spec.ComponentVersions[i] + for i, c := range r.ComponentVersions { + m[c.ComponentDefRef] = &r.ComponentVersions[i] } return m } diff --git a/apis/apps/v1alpha1/opsrequest_types.go b/apis/apps/v1alpha1/opsrequest_types.go index 527955ad2..4e540b2ac 100644 --- a/apis/apps/v1alpha1/opsrequest_types.go +++ b/apis/apps/v1alpha1/opsrequest_types.go @@ -432,127 +432,128 @@ func init() { SchemeBuilder.Register(&OpsRequest{}, &OpsRequestList{}) } -// GetComponentNameMap if the operations are within the scope of component, this function should be implemented -func (r *OpsRequest) GetComponentNameMap() map[string]struct{} { - switch r.Spec.Type { - case RestartType: - return r.GetRestartComponentNameMap() - case VerticalScalingType: - return r.GetVerticalScalingComponentNameMap() - case HorizontalScalingType: - return r.GetHorizontalScalingComponentNameMap() - case VolumeExpansionType: - return r.GetVolumeExpansionComponentNameMap() - case UpgradeType: - return r.GetUpgradeComponentNameMap() - case ReconfiguringType: - return r.GetReconfiguringComponentNameMap() - case ExposeType: - return r.GetExposeComponentNameMap() - default: - return map[string]struct{}{} +// GetRestartComponentNameSet gets the component name map with restart operation. +func (r OpsRequestSpec) GetRestartComponentNameSet() ComponentNameSet { + set := make(ComponentNameSet) + for _, v := range r.RestartList { + set[v.ComponentName] = struct{}{} } + return set } -// GetRestartComponentNameMap gets the component name map with restart operation. -func (r *OpsRequest) GetRestartComponentNameMap() map[string]struct{} { - componentNameMap := make(map[string]struct{}) - for _, v := range r.Spec.RestartList { - componentNameMap[v.ComponentName] = struct{}{} +// GetVerticalScalingComponentNameSet gets the component name map with vertical scaling operation. +func (r OpsRequestSpec) GetVerticalScalingComponentNameSet() ComponentNameSet { + set := make(ComponentNameSet) + for _, v := range r.VerticalScalingList { + set[v.ComponentName] = struct{}{} } - return componentNameMap + return set } -// GetVerticalScalingComponentNameMap gets the component name map with vertical scaling operation. -func (r *OpsRequest) GetVerticalScalingComponentNameMap() map[string]struct{} { - componentNameMap := make(map[string]struct{}) - for _, v := range r.Spec.VerticalScalingList { - componentNameMap[v.ComponentName] = struct{}{} - } - return componentNameMap -} - -// ConvertVerticalScalingListToMap converts OpsRequest.spec.verticalScaling list to map -func (r *OpsRequest) ConvertVerticalScalingListToMap() map[string]VerticalScaling { +// ToVerticalScalingListToMap converts OpsRequest.spec.verticalScaling list to map +func (r OpsRequestSpec) ToVerticalScalingListToMap() map[string]VerticalScaling { verticalScalingMap := make(map[string]VerticalScaling) - for _, v := range r.Spec.VerticalScalingList { + for _, v := range r.VerticalScalingList { verticalScalingMap[v.ComponentName] = v } return verticalScalingMap } -// GetHorizontalScalingComponentNameMap gets the component name map with horizontal scaling operation. -func (r *OpsRequest) GetHorizontalScalingComponentNameMap() map[string]struct{} { - componentNameMap := make(map[string]struct{}) - for _, v := range r.Spec.HorizontalScalingList { - componentNameMap[v.ComponentName] = struct{}{} +// GetHorizontalScalingComponentNameSet gets the component name map with horizontal scaling operation. +func (r OpsRequestSpec) GetHorizontalScalingComponentNameSet() ComponentNameSet { + set := make(ComponentNameSet) + for _, v := range r.HorizontalScalingList { + set[v.ComponentName] = struct{}{} } - return componentNameMap + return set } -// ConvertHorizontalScalingListToMap converts OpsRequest.spec.horizontalScaling list to map -func (r *OpsRequest) ConvertHorizontalScalingListToMap() map[string]HorizontalScaling { +// ToHorizontalScalingListToMap converts OpsRequest.spec.horizontalScaling list to map +func (r OpsRequestSpec) ToHorizontalScalingListToMap() map[string]HorizontalScaling { verticalScalingMap := make(map[string]HorizontalScaling) - for _, v := range r.Spec.HorizontalScalingList { + for _, v := range r.HorizontalScalingList { verticalScalingMap[v.ComponentName] = v } return verticalScalingMap } -// GetVolumeExpansionComponentNameMap gets the component name map with volume expansion operation. -func (r *OpsRequest) GetVolumeExpansionComponentNameMap() map[string]struct{} { - componentNameMap := make(map[string]struct{}) - for _, v := range r.Spec.VolumeExpansionList { - componentNameMap[v.ComponentName] = struct{}{} +// GetVolumeExpansionComponentNameSet gets the component name map with volume expansion operation. +func (r OpsRequestSpec) GetVolumeExpansionComponentNameSet() ComponentNameSet { + set := make(ComponentNameSet) + for _, v := range r.VolumeExpansionList { + set[v.ComponentName] = struct{}{} } - return componentNameMap + return set } -// ConvertVolumeExpansionListToMap converts volumeExpansionList to map -func (r *OpsRequest) ConvertVolumeExpansionListToMap() map[string]VolumeExpansion { +// ToVolumeExpansionListToMap converts volumeExpansionList to map +func (r OpsRequestSpec) ToVolumeExpansionListToMap() map[string]VolumeExpansion { volumeExpansionMap := make(map[string]VolumeExpansion) - for _, v := range r.Spec.VolumeExpansionList { + for _, v := range r.VolumeExpansionList { volumeExpansionMap[v.ComponentName] = v } return volumeExpansionMap } -func (r *OpsRequest) ConvertExposeListToMap() map[string]Expose { +// ToExposeListToMap build expose map +func (r OpsRequestSpec) ToExposeListToMap() map[string]Expose { exposeMap := make(map[string]Expose) - for _, v := range r.Spec.ExposeList { + for _, v := range r.ExposeList { exposeMap[v.ComponentName] = v } return exposeMap } -// GetUpgradeComponentNameMap gets the component name map with upgrade operation. -func (r *OpsRequest) GetUpgradeComponentNameMap() map[string]struct{} { - if r.Spec.Upgrade == nil { +// GetReconfiguringComponentNameSet gets the component name map with reconfiguring operation. +func (r OpsRequestSpec) GetReconfiguringComponentNameSet() ComponentNameSet { + if r.Reconfigure == nil { return nil } - componentNameMap := make(map[string]struct{}) - for k := range r.Status.Components { - componentNameMap[k] = struct{}{} + return ComponentNameSet{ + r.Reconfigure.ComponentName: {}, } - return componentNameMap } -// GetReconfiguringComponentNameMap gets the component name map with reconfiguring operation. -func (r *OpsRequest) GetReconfiguringComponentNameMap() map[string]struct{} { - if r.Spec.Reconfigure == nil { +func (r OpsRequestSpec) GetExposeComponentNameSet() ComponentNameSet { + set := make(ComponentNameSet) + for _, v := range r.ExposeList { + set[v.ComponentName] = struct{}{} + } + return set +} + +// GetUpgradeComponentNameSet gets the component name map with upgrade operation. +func (r *OpsRequest) GetUpgradeComponentNameSet() ComponentNameSet { + if r == nil || r.Spec.Upgrade == nil { return nil } - return map[string]struct{}{ - r.Spec.Reconfigure.ComponentName: {}, + set := make(ComponentNameSet) + for k := range r.Status.Components { + set[k] = struct{}{} } + return set } -func (r *OpsRequest) GetExposeComponentNameMap() map[string]struct{} { - componentNameMap := make(map[string]struct{}) - for _, v := range r.Spec.ExposeList { - componentNameMap[v.ComponentName] = struct{}{} +// GetComponentNameSet if the operations are within the scope of component, this function should be implemented +func (r *OpsRequest) GetComponentNameSet() ComponentNameSet { + switch r.Spec.Type { + case RestartType: + return r.Spec.GetRestartComponentNameSet() + case VerticalScalingType: + return r.Spec.GetVerticalScalingComponentNameSet() + case HorizontalScalingType: + return r.Spec.GetHorizontalScalingComponentNameSet() + case VolumeExpansionType: + return r.Spec.GetVolumeExpansionComponentNameSet() + case UpgradeType: + return r.GetUpgradeComponentNameSet() + case ReconfiguringType: + return r.Spec.GetReconfiguringComponentNameSet() + case ExposeType: + return r.Spec.GetExposeComponentNameSet() + default: + return nil } - return componentNameMap } func (p *ProgressStatusDetail) SetStatusAndMessage(status ProgressStatus, message string) { diff --git a/apis/apps/v1alpha1/opsrequest_types_test.go b/apis/apps/v1alpha1/opsrequest_types_test.go index 090e569cb..d041caa7f 100644 --- a/apis/apps/v1alpha1/opsrequest_types_test.go +++ b/apis/apps/v1alpha1/opsrequest_types_test.go @@ -35,9 +35,9 @@ func mockRestartOps() *OpsRequest { func TestGetRestartComponentNameMap(t *testing.T) { ops := mockRestartOps() - componentNameMap := ops.GetRestartComponentNameMap() + componentNameMap := ops.Spec.GetRestartComponentNameSet() checkComponentMap(t, componentNameMap, len(ops.Spec.RestartList), componentName) - componentNameMap1 := ops.GetComponentNameMap() + componentNameMap1 := ops.GetComponentNameSet() checkComponentMap(t, componentNameMap1, len(ops.Spec.RestartList), componentName) } @@ -56,9 +56,9 @@ func mockVerticalScalingOps() *OpsRequest { func TestVerticalScalingComponentNameMap(t *testing.T) { ops := mockVerticalScalingOps() - componentNameMap := ops.GetVerticalScalingComponentNameMap() + componentNameMap := ops.Spec.GetVerticalScalingComponentNameSet() checkComponentMap(t, componentNameMap, len(ops.Spec.VerticalScalingList), componentName) - componentNameMap1 := ops.GetComponentNameMap() + componentNameMap1 := ops.GetComponentNameSet() checkComponentMap(t, componentNameMap1, len(ops.Spec.VerticalScalingList), componentName) } @@ -77,9 +77,9 @@ func mockHorizontalScalingOps() *OpsRequest { func TestHorizontalScalingComponentNameMap(t *testing.T) { ops := mockHorizontalScalingOps() - componentNameMap := ops.GetHorizontalScalingComponentNameMap() + componentNameMap := ops.Spec.GetHorizontalScalingComponentNameSet() checkComponentMap(t, componentNameMap, len(ops.Spec.HorizontalScalingList), componentName) - componentNameMap1 := ops.GetComponentNameMap() + componentNameMap1 := ops.GetComponentNameSet() checkComponentMap(t, componentNameMap1, len(ops.Spec.HorizontalScalingList), componentName) } @@ -98,9 +98,9 @@ func mockVolumeExpansionOps() *OpsRequest { func TestVolumeExpansionComponentNameMap(t *testing.T) { ops := mockVolumeExpansionOps() - componentNameMap := ops.GetVolumeExpansionComponentNameMap() + componentNameMap := ops.Spec.GetVolumeExpansionComponentNameSet() checkComponentMap(t, componentNameMap, len(ops.Spec.VolumeExpansionList), componentName) - componentNameMap1 := ops.GetComponentNameMap() + componentNameMap1 := ops.GetComponentNameSet() checkComponentMap(t, componentNameMap1, len(ops.Spec.VolumeExpansionList), componentName) } @@ -113,9 +113,9 @@ func checkComponentMap(t *testing.T, componentNameMap map[string]struct{}, expec } } -func TestConvertVerticalScalingListToMap(t *testing.T) { +func TestToVerticalScalingListToMap(t *testing.T) { ops := mockVerticalScalingOps() - verticalScalingMap := ops.ConvertVerticalScalingListToMap() + verticalScalingMap := ops.Spec.ToVerticalScalingListToMap() if len(verticalScalingMap) != len(ops.Spec.VerticalScalingList) { t.Error(`Expected vertical scaling map length equals list length`) } @@ -126,7 +126,7 @@ func TestConvertVerticalScalingListToMap(t *testing.T) { func TestConvertVolumeExpansionListToMap(t *testing.T) { ops := mockVolumeExpansionOps() - volumeExpansionMap := ops.ConvertVolumeExpansionListToMap() + volumeExpansionMap := ops.Spec.ToVolumeExpansionListToMap() if len(volumeExpansionMap) != len(ops.Spec.VolumeExpansionList) { t.Error(`Expected volume expansion map length equals list length`) } @@ -135,9 +135,9 @@ func TestConvertVolumeExpansionListToMap(t *testing.T) { } } -func TestConvertHorizontalScalingListToMap(t *testing.T) { +func TestToHorizontalScalingListToMap(t *testing.T) { ops := mockHorizontalScalingOps() - horizontalScalingMap := ops.ConvertHorizontalScalingListToMap() + horizontalScalingMap := ops.Spec.ToHorizontalScalingListToMap() if len(horizontalScalingMap) != len(ops.Spec.HorizontalScalingList) { t.Error(`Expected horizontal scaling map length equals list length`) } @@ -149,7 +149,7 @@ func TestConvertHorizontalScalingListToMap(t *testing.T) { func TestGetUpgradeComponentNameMap(t *testing.T) { ops := &OpsRequest{} ops.Spec.Type = UpgradeType - componentNameMap := ops.GetUpgradeComponentNameMap() + componentNameMap := ops.GetUpgradeComponentNameSet() if componentNameMap != nil { t.Error(`Expected component name map of upgrade ops is nil`) } @@ -160,9 +160,9 @@ func TestGetUpgradeComponentNameMap(t *testing.T) { componentName: {}, } - componentNameMap = ops.GetUpgradeComponentNameMap() + componentNameMap = ops.GetUpgradeComponentNameSet() checkComponentMap(t, componentNameMap, len(ops.Status.Components), componentName) - componentNameMap1 := ops.GetComponentNameMap() + componentNameMap1 := ops.GetComponentNameSet() checkComponentMap(t, componentNameMap1, len(ops.Status.Components), componentName) } diff --git a/apis/apps/v1alpha1/type.go b/apis/apps/v1alpha1/type.go index 07bf3fdb2..8ff8b48c2 100644 --- a/apis/apps/v1alpha1/type.go +++ b/apis/apps/v1alpha1/type.go @@ -445,3 +445,5 @@ const ( func RegisterWebhookManager(mgr manager.Manager) { webhookMgr = &webhookManager{mgr.GetClient()} } + +type ComponentNameSet map[string]struct{} diff --git a/apis/extensions/v1alpha1/addon_types.go b/apis/extensions/v1alpha1/addon_types.go index 151744a4b..0d981158b 100644 --- a/apis/extensions/v1alpha1/addon_types.go +++ b/apis/extensions/v1alpha1/addon_types.go @@ -382,10 +382,7 @@ func buildSelectorStrings(selectors []SelectorRequirement) []string { } // GetSelectorsStrings extract selectors to string representations. -func (r *AddonDefaultInstallSpecItem) GetSelectorsStrings() []string { - if r == nil { - return nil - } +func (r AddonDefaultInstallSpecItem) GetSelectorsStrings() []string { return buildSelectorStrings(r.Selectors) } @@ -397,16 +394,13 @@ func (r *InstallableSpec) GetSelectorsStrings() []string { return buildSelectorStrings(r.Selectors) } -func (r *SelectorRequirement) String() string { +func (r SelectorRequirement) String() string { return fmt.Sprintf("{key=%s,op=%s,values=%v}", r.Key, r.Operator, r.Values) } // MatchesFromConfig matches selector requirement value. -func (r *SelectorRequirement) MatchesFromConfig() bool { - if r == nil { - return false - } +func (r SelectorRequirement) MatchesFromConfig() bool { verIf := viper.Get(constant.CfgKeyServerInfo) ver, ok := verIf.(version.Info) if !ok { @@ -422,7 +416,7 @@ func (r *SelectorRequirement) MatchesFromConfig() bool { return r.matchesLine(l) } -func (r *SelectorRequirement) matchesLine(line string) bool { +func (r SelectorRequirement) matchesLine(line string) bool { processor := func(op bool, predicate func(string) bool) bool { if len(r.Values) == 0 { return !op @@ -569,10 +563,7 @@ func (r *HelmTypeInstallSpec) BuildMergedValues(installSpec *AddonInstallSpec) H // GetSortedDefaultInstallValues return DefaultInstallValues items with items that has // provided selector first. -func (r *AddonSpec) GetSortedDefaultInstallValues() []AddonDefaultInstallSpecItem { - if r == nil { - return nil - } +func (r AddonSpec) GetSortedDefaultInstallValues() []AddonDefaultInstallSpecItem { values := make([]AddonDefaultInstallSpecItem, 0, len(r.DefaultInstallValues)) nvalues := make([]AddonDefaultInstallSpecItem, 0, len(r.DefaultInstallValues)) for _, i := range r.DefaultInstallValues { diff --git a/apis/extensions/v1alpha1/addon_types_test.go b/apis/extensions/v1alpha1/addon_types_test.go index c988610d4..28c91e78e 100644 --- a/apis/extensions/v1alpha1/addon_types_test.go +++ b/apis/extensions/v1alpha1/addon_types_test.go @@ -41,9 +41,7 @@ func TestSelectorRequirementString(t *testing.T) { func TestSelectorRequirementNoOperator(t *testing.T) { g := NewGomegaWithT(t) - var r *SelectorRequirement - g.Expect(r.MatchesFromConfig()).Should(BeFalse()) - r = &SelectorRequirement{ + r := SelectorRequirement{ Key: KubeGitVersion, } g.Expect(r.MatchesFromConfig()).Should(BeFalse()) @@ -346,9 +344,7 @@ func TestHelmInstallSpecBuildMergedValues(t *testing.T) { func TestAddonSpecMisc(t *testing.T) { g := NewGomegaWithT(t) - var addonSpec *AddonSpec - g.Expect(addonSpec.GetSortedDefaultInstallValues()).Should(BeNil()) - addonSpec = &AddonSpec{} + addonSpec := AddonSpec{} g.Expect(addonSpec.InstallSpec.GetEnabled()).Should(BeFalse()) g.Expect(addonSpec.Helm.BuildMergedValues(nil)).Should(BeEquivalentTo(HelmInstallValues{})) addonSpec.InstallSpec = &AddonInstallSpec{ diff --git a/controllers/apps/operations/expose.go b/controllers/apps/operations/expose.go index 7430f7f85..c6cee17c6 100644 --- a/controllers/apps/operations/expose.go +++ b/controllers/apps/operations/expose.go @@ -34,6 +34,8 @@ import ( type ExposeOpsHandler struct { } +var _ OpsHandler = ExposeOpsHandler{} + func init() { // ToClusterPhase is not defined, because expose not affect the cluster status. exposeBehavior := OpsBehaviour{ @@ -48,7 +50,7 @@ func init() { func (e ExposeOpsHandler) Action(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { var ( - exposeMap = opsRes.OpsRequest.ConvertExposeListToMap() + exposeMap = opsRes.OpsRequest.Spec.ToExposeListToMap() ) for index, component := range opsRes.Cluster.Spec.ComponentSpecs { @@ -170,13 +172,12 @@ func (e ExposeOpsHandler) ActionStartedCondition(opsRequest *appsv1alpha1.OpsReq } func (e ExposeOpsHandler) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, cli client.Client, opsResource *OpsResource) error { - componentNameMap := opsResource.OpsRequest.GetComponentNameMap() + componentNameSet := opsResource.OpsRequest.GetComponentNameSet() lastComponentInfo := map[string]appsv1alpha1.LastComponentConfiguration{} for _, v := range opsResource.Cluster.Spec.ComponentSpecs { - if _, ok := componentNameMap[v.Name]; !ok { + if _, ok := componentNameSet[v.Name]; !ok { continue } - lastComponentInfo[v.Name] = appsv1alpha1.LastComponentConfiguration{ Services: v.Services, } @@ -186,7 +187,5 @@ func (e ExposeOpsHandler) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, c } func (e ExposeOpsHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { - return opsRequest.GetExposeComponentNameMap() + return realAffectedComponentMap(opsRequest.Spec.GetExposeComponentNameSet()) } - -var _ OpsHandler = ExposeOpsHandler{} diff --git a/controllers/apps/operations/horizontal_scaling.go b/controllers/apps/operations/horizontal_scaling.go index 3f56d20d5..7720c3778 100644 --- a/controllers/apps/operations/horizontal_scaling.go +++ b/controllers/apps/operations/horizontal_scaling.go @@ -52,7 +52,7 @@ func (hs horizontalScalingOpsHandler) ActionStartedCondition(opsRequest *appsv1a // Action modifies Cluster.spec.components[*].replicas from the opsRequest func (hs horizontalScalingOpsHandler) Action(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { var ( - horizontalScalingMap = opsRes.OpsRequest.ConvertHorizontalScalingListToMap() + horizontalScalingMap = opsRes.OpsRequest.Spec.ToHorizontalScalingListToMap() horizontalScaling appsv1alpha1.HorizontalScaling ok bool ) @@ -85,7 +85,7 @@ func (hs horizontalScalingOpsHandler) ReconcileAction(reqCtx intctrlutil.Request // GetRealAffectedComponentMap gets the real affected component map for the operation func (hs horizontalScalingOpsHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { realChangedMap := realAffectedComponentMap{} - hsMap := opsRequest.ConvertHorizontalScalingListToMap() + hsMap := opsRequest.Spec.ToHorizontalScalingListToMap() for k, v := range opsRequest.Status.LastConfiguration.Components { currHs, ok := hsMap[k] if !ok { @@ -102,7 +102,7 @@ func (hs horizontalScalingOpsHandler) GetRealAffectedComponentMap(opsRequest *ap func (hs horizontalScalingOpsHandler) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { opsRequest := opsRes.OpsRequest lastComponentInfo := map[string]appsv1alpha1.LastComponentConfiguration{} - componentNameMap := opsRequest.ConvertHorizontalScalingListToMap() + componentNameMap := opsRequest.Spec.ToHorizontalScalingListToMap() for _, v := range opsRes.Cluster.Spec.ComponentSpecs { hsInfo, ok := componentNameMap[v.Name] if !ok { diff --git a/controllers/apps/operations/ops_util.go b/controllers/apps/operations/ops_util.go index 0d7f8343d..b097ee6f5 100644 --- a/controllers/apps/operations/ops_util.go +++ b/controllers/apps/operations/ops_util.go @@ -68,7 +68,7 @@ func ReconcileActionWithComponentOps(reqCtx intctrlutil.RequestCtx, completedProgressCount int32 checkAllClusterComponent bool ) - componentNameMap := opsRequest.GetComponentNameMap() + componentNameMap := opsRequest.GetComponentNameSet() // if no specified components, we should check the all components phase of cluster. if len(componentNameMap) == 0 { checkAllClusterComponent = true diff --git a/controllers/apps/operations/reconfigure.go b/controllers/apps/operations/reconfigure.go index 8e2d9f92b..56b39928e 100644 --- a/controllers/apps/operations/reconfigure.go +++ b/controllers/apps/operations/reconfigure.go @@ -58,7 +58,7 @@ func (r *reconfigureAction) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, // GetRealAffectedComponentMap gets the real affected component map for the operation func (r *reconfigureAction) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { - return opsRequest.GetReconfiguringComponentNameMap() + return realAffectedComponentMap(opsRequest.Spec.GetReconfiguringComponentNameSet()) } func (r *reconfigureAction) Handle(eventContext cfgcore.ConfigEventContext, lastOpsRequest string, phase appsv1alpha1.OpsPhase, cfgError error) error { diff --git a/controllers/apps/operations/restart.go b/controllers/apps/operations/restart.go index d7a49f463..342ab10f6 100644 --- a/controllers/apps/operations/restart.go +++ b/controllers/apps/operations/restart.go @@ -59,7 +59,7 @@ func (r restartOpsHandler) Action(reqCtx intctrlutil.RequestCtx, cli client.Clie if opsRes.OpsRequest.Status.StartTimestamp.IsZero() { return fmt.Errorf("status.startTimestamp can not be null") } - componentNameMap := opsRes.OpsRequest.GetRestartComponentNameMap() + componentNameMap := opsRes.OpsRequest.Spec.GetRestartComponentNameSet() if err := restartDeployment(reqCtx, cli, opsRes, componentNameMap); err != nil { return err } @@ -74,7 +74,7 @@ func (r restartOpsHandler) ReconcileAction(reqCtx intctrlutil.RequestCtx, cli cl // GetRealAffectedComponentMap gets the real affected component map for the operation func (r restartOpsHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { - return opsRequest.GetRestartComponentNameMap() + return realAffectedComponentMap(opsRequest.Spec.GetRestartComponentNameSet()) } // SaveLastConfiguration this operation only restart the pods of the component, no changes in Cluster.spec. diff --git a/controllers/apps/operations/upgrade.go b/controllers/apps/operations/upgrade.go index 3625df5e6..e04fc914c 100644 --- a/controllers/apps/operations/upgrade.go +++ b/controllers/apps/operations/upgrade.go @@ -65,7 +65,7 @@ func (u upgradeOpsHandler) ReconcileAction(reqCtx intctrlutil.RequestCtx, cli cl // GetRealAffectedComponentMap gets the real affected component map for the operation func (u upgradeOpsHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { - return opsRequest.GetUpgradeComponentNameMap() + return realAffectedComponentMap(opsRequest.GetUpgradeComponentNameSet()) } // SaveLastConfiguration records last configuration to the OpsRequest.status.lastConfiguration diff --git a/controllers/apps/operations/vertical_scaling.go b/controllers/apps/operations/vertical_scaling.go index dc286bcf3..1913d98b5 100644 --- a/controllers/apps/operations/vertical_scaling.go +++ b/controllers/apps/operations/vertical_scaling.go @@ -53,7 +53,7 @@ func (vs verticalScalingHandler) ActionStartedCondition(opsRequest *appsv1alpha1 // Action modifies cluster component resources according to // the definition of opsRequest with spec.componentNames and spec.componentOps.verticalScaling func (vs verticalScalingHandler) Action(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { - verticalScalingMap := opsRes.OpsRequest.ConvertVerticalScalingListToMap() + verticalScalingMap := opsRes.OpsRequest.Spec.ToVerticalScalingListToMap() for index, component := range opsRes.Cluster.Spec.ComponentSpecs { if verticalScaling, ok := verticalScalingMap[component.Name]; ok { component.Resources = verticalScaling.ResourceRequirements @@ -71,10 +71,10 @@ func (vs verticalScalingHandler) ReconcileAction(reqCtx intctrlutil.RequestCtx, // SaveLastConfiguration records last configuration to the OpsRequest.status.lastConfiguration func (vs verticalScalingHandler) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { - componentNameMap := opsRes.OpsRequest.GetComponentNameMap() + componentNameSet := opsRes.OpsRequest.GetComponentNameSet() lastComponentInfo := map[string]appsv1alpha1.LastComponentConfiguration{} for _, v := range opsRes.Cluster.Spec.ComponentSpecs { - if _, ok := componentNameMap[v.Name]; !ok { + if _, ok := componentNameSet[v.Name]; !ok { continue } lastComponentInfo[v.Name] = appsv1alpha1.LastComponentConfiguration{ @@ -88,7 +88,7 @@ func (vs verticalScalingHandler) SaveLastConfiguration(reqCtx intctrlutil.Reques // GetRealAffectedComponentMap gets the real affected component map for the operation func (vs verticalScalingHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { realChangedMap := realAffectedComponentMap{} - vsMap := opsRequest.ConvertVerticalScalingListToMap() + vsMap := opsRequest.Spec.ToVerticalScalingListToMap() for k, v := range opsRequest.Status.LastConfiguration.Components { currVs, ok := vsMap[k] if !ok { diff --git a/controllers/apps/operations/volume_expansion.go b/controllers/apps/operations/volume_expansion.go index 79bd1a00d..4e7f079ac 100644 --- a/controllers/apps/operations/volume_expansion.go +++ b/controllers/apps/operations/volume_expansion.go @@ -64,7 +64,7 @@ func (ve volumeExpansionOpsHandler) ActionStartedCondition(opsRequest *appsv1alp // Action modifies Cluster.spec.components[*].VolumeClaimTemplates[*].spec.resources func (ve volumeExpansionOpsHandler) Action(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { var ( - volumeExpansionMap = opsRes.OpsRequest.ConvertVolumeExpansionListToMap() + volumeExpansionMap = opsRes.OpsRequest.Spec.ToVolumeExpansionListToMap() volumeExpansionOps appsv1alpha1.VolumeExpansion ok bool ) @@ -166,17 +166,17 @@ func (ve volumeExpansionOpsHandler) ReconcileAction(reqCtx intctrlutil.RequestCt // GetRealAffectedComponentMap gets the real affected component map for the operation func (ve volumeExpansionOpsHandler) GetRealAffectedComponentMap(opsRequest *appsv1alpha1.OpsRequest) realAffectedComponentMap { - return opsRequest.GetVolumeExpansionComponentNameMap() + return realAffectedComponentMap(opsRequest.Spec.GetVolumeExpansionComponentNameSet()) } // SaveLastConfiguration records last configuration to the OpsRequest.status.lastConfiguration func (ve volumeExpansionOpsHandler) SaveLastConfiguration(reqCtx intctrlutil.RequestCtx, cli client.Client, opsRes *OpsResource) error { opsRequest := opsRes.OpsRequest - componentNameMap := opsRequest.GetComponentNameMap() + componentNameSet := opsRequest.GetComponentNameSet() storageMap := ve.getRequestStorageMap(opsRequest) lastComponentInfo := map[string]appsv1alpha1.LastComponentConfiguration{} for _, v := range opsRes.Cluster.Spec.ComponentSpecs { - if _, ok := componentNameMap[v.Name]; !ok { + if _, ok := componentNameSet[v.Name]; !ok { continue } lastVCTs := make([]appsv1alpha1.OpsRequestVolumeClaimTemplate, 0) diff --git a/githooks/scripts/run-lint.sh b/githooks/scripts/run-lint.sh index 03d59e99a..6eb4fdd2e 100755 --- a/githooks/scripts/run-lint.sh +++ b/githooks/scripts/run-lint.sh @@ -1,2 +1,2 @@ #!/bin/sh -make fast-lint +make lint-fast diff --git a/internal/controller/lifecycle/transformer_cluster.go b/internal/controller/lifecycle/transformer_cluster.go index 856216447..2a40432a9 100644 --- a/internal/controller/lifecycle/transformer_cluster.go +++ b/internal/controller/lifecycle/transformer_cluster.go @@ -70,7 +70,7 @@ func (c *clusterTransformer) Transform(dag *graph.DAG) error { } clusterCompSpecMap := cluster.Spec.GetDefNameMappingComponents() - clusterCompVerMap := c.cc.cv.GetDefNameMappingComponents() + clusterCompVerMap := c.cc.cv.Spec.GetDefNameMappingComponents() process1stComp := true // TODO: should move credential secrets creation from system_account_controller & here into credential_transformer, From 3bac86e97718c3e5342ffe93a77f6b781294a5e3 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Mon, 10 Apr 2023 11:33:07 +0800 Subject: [PATCH 60/80] fix: fixed Makefile err (#2467) --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 7e8572173..b25afb41f 100644 --- a/Makefile +++ b/Makefile @@ -170,7 +170,7 @@ staticcheck: staticchecktool ## Run staticcheck against code. $(STATICCHECK) ./... .PHONY: build-checks -build-checks: generate fmt vet goimports fast-lint ## Run build checks. +build-checks: generate fmt vet goimports lint-fast ## Run build checks. .PHONY: mod-download mod-download: ## Run go mod download against go modules. From a690c1d33da137151c03aa29da94d27415953b67 Mon Sep 17 00:00:00 2001 From: wangyelei Date: Mon, 10 Apr 2023 14:34:49 +0800 Subject: [PATCH 61/80] fix: restart apecloud-mysql failed (#2461) --- deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml | 3 +++ deploy/apecloud-mysql/templates/clusterdefinition.yaml | 3 +++ 2 files changed, 6 insertions(+) diff --git a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml index 3c9cbb8db..8e67682c6 100644 --- a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml @@ -54,6 +54,9 @@ spec: followers: - name: follower accessMode: Readonly + learner: + name: learner + accessMode: Readonly service: ports: - name: mysql diff --git a/deploy/apecloud-mysql/templates/clusterdefinition.yaml b/deploy/apecloud-mysql/templates/clusterdefinition.yaml index 087f8b3b8..a7fca264e 100644 --- a/deploy/apecloud-mysql/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql/templates/clusterdefinition.yaml @@ -50,6 +50,9 @@ spec: followers: - name: follower accessMode: Readonly + learner: + name: learner + accessMode: Readonly service: ports: - name: mysql From 629488e3bb238f09a937705c3f87cc798a8cf130 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Mon, 10 Apr 2023 15:14:55 +0800 Subject: [PATCH 62/80] chore: prepare v0.5 release notes (#2479) --- docs/release_notes/v0.5.0/_category_.yml | 4 +++ docs/release_notes/v0.5.0/v0.5.0.md | 37 ++++++++++++++++++++++++ 2 files changed, 41 insertions(+) create mode 100644 docs/release_notes/v0.5.0/_category_.yml create mode 100644 docs/release_notes/v0.5.0/v0.5.0.md diff --git a/docs/release_notes/v0.5.0/_category_.yml b/docs/release_notes/v0.5.0/_category_.yml new file mode 100644 index 000000000..ab9391a44 --- /dev/null +++ b/docs/release_notes/v0.5.0/_category_.yml @@ -0,0 +1,4 @@ +position: 1 +label: v0.5.0 +collapsible: true +collapsed: true \ No newline at end of file diff --git a/docs/release_notes/v0.5.0/v0.5.0.md b/docs/release_notes/v0.5.0/v0.5.0.md new file mode 100644 index 000000000..e62720bb7 --- /dev/null +++ b/docs/release_notes/v0.5.0/v0.5.0.md @@ -0,0 +1,37 @@ +# KubeBlocks 0.5.0 (TBD) + +We are happy to announce the release of KubeBlocks 0.5.0 with some exciting new features and improvements. + +## Highlights + +## Acknowledgements + +Thanks to everyone who made this release possible! + + +## What's Changed + +### New Features + +#### PostgreSQL + +#### Redis + +#### Compatibility +- Pass the AWS EKS v1.22 / v1.23 / v1.24 / v1.25 compatibility test. + +#### Maintainability + + +#### Easy of Use + +#### Resource Isolation + + +#### Observability + + +### Bug Fixes + + +## Breaking changes From 3abe10cc7bbd3fffb506b19821ef9a88bb589735 Mon Sep 17 00:00:00 2001 From: xingran Date: Mon, 10 Apr 2023 16:41:08 +0800 Subject: [PATCH 63/80] chore: update pg patroni clusterVersion to 15.2.0 (#2485) --- deploy/postgresql-patroni-ha-cluster/Chart.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy/postgresql-patroni-ha-cluster/Chart.yaml b/deploy/postgresql-patroni-ha-cluster/Chart.yaml index 5985d60c8..56c51fcc3 100644 --- a/deploy/postgresql-patroni-ha-cluster/Chart.yaml +++ b/deploy/postgresql-patroni-ha-cluster/Chart.yaml @@ -6,4 +6,4 @@ type: application version: 0.5.0-alpha.3 -appVersion: "14.5.0" +appVersion: "15.2.0" From 380027dafe07231817011566d668d2310e8a4bbb Mon Sep 17 00:00:00 2001 From: xingran Date: Mon, 10 Apr 2023 16:41:54 +0800 Subject: [PATCH 64/80] chore: fix error when creates redis sentinel cluster with no primaryIndex by kbcli (#2484) --- deploy/redis/scripts/redis-sentinel-setup.sh.tpl | 6 ++++-- deploy/redis/templates/clusterdefinition.yaml | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/deploy/redis/scripts/redis-sentinel-setup.sh.tpl b/deploy/redis/scripts/redis-sentinel-setup.sh.tpl index af020399f..440d5cec9 100644 --- a/deploy/redis/scripts/redis-sentinel-setup.sh.tpl +++ b/deploy/redis/scripts/redis-sentinel-setup.sh.tpl @@ -12,8 +12,10 @@ set -ex {{- $sentinel_component = $e }} {{- else if eq $e.componentDefRef "redis" }} {{- $redis_component = $e }} - {{- if ne ($e.primaryIndex | int) 0 }} - {{- $primary_index = ($e.primaryIndex | int) }} + {{- if index $e "primaryIndex" }} + {{- if ne ($e.primaryIndex | int) 0 }} + {{- $primary_index = ($e.primaryIndex | int) }} + {{- end }} {{- end }} {{- end }} {{- end }} diff --git a/deploy/redis/templates/clusterdefinition.yaml b/deploy/redis/templates/clusterdefinition.yaml index b84c3ce45..a9bda7e01 100644 --- a/deploy/redis/templates/clusterdefinition.yaml +++ b/deploy/redis/templates/clusterdefinition.yaml @@ -61,6 +61,7 @@ spec: port: 6379 targetPort: redis - name: metrics + port: 9121 targetPort: metrics configSpecs: - name: redis-replication-config From 3170b3351adb02b0186cdb334bb513cc919cde3a Mon Sep 17 00:00:00 2001 From: shaojiang Date: Mon, 10 Apr 2023 18:04:38 +0800 Subject: [PATCH 65/80] feat: add manifests spec for backup api (#2187) --- apis/dataprotection/v1alpha1/backup_types.go | 83 +++++++++-- .../v1alpha1/backuppolicy_types.go | 35 +++++ .../v1alpha1/backuppolicytemplate_types.go | 4 + ...otection.kubeblocks.io_backuppolicies.yaml | 27 ++++ ...n.kubeblocks.io_backuppolicytemplates.yaml | 27 ++++ .../dataprotection.kubeblocks.io_backups.yaml | 77 ++++++++-- .../dataprotection/backup_controller.go | 134 ++++++++++++++++-- .../dataprotection/backup_controller_test.go | 51 ++++--- .../dataprotection/backuppolicy_controller.go | 3 + ...otection.kubeblocks.io_backuppolicies.yaml | 27 ++++ ...n.kubeblocks.io_backuppolicytemplates.yaml | 27 ++++ .../dataprotection.kubeblocks.io_backups.yaml | 77 ++++++++-- internal/controller/builder/builder.go | 17 +++ .../builder/cue/backup_manifests_template.cue | 47 ++++++ 14 files changed, 574 insertions(+), 62 deletions(-) create mode 100644 internal/controller/builder/cue/backup_manifests_template.cue diff --git a/apis/dataprotection/v1alpha1/backup_types.go b/apis/dataprotection/v1alpha1/backup_types.go index 62be77bc4..7fe18d89d 100644 --- a/apis/dataprotection/v1alpha1/backup_types.go +++ b/apis/dataprotection/v1alpha1/backup_types.go @@ -36,7 +36,7 @@ type BackupSpec struct { // +optional ParentBackupName string `json:"parentBackupName,omitempty"` - // TTL is a time.Duration-parsable string describing how long + // ttl is a time.Duration-parsable string describing how long // the Backup should be retained for. // +optional TTL *metav1.Duration `json:"ttl,omitempty"` @@ -74,30 +74,89 @@ type BackupStatus struct { // +optional TotalSize string `json:"totalSize,omitempty"` - // backup total size - // string with capacity units in the form of "1Gi", "1Mi", "1Ki". + // the reason if backup failed. // +optional - UploadTotalSize string `json:"uploadTotalSize,omitempty"` + FailureReason string `json:"failureReason,omitempty"` - // checksum of backup file, generated by md5 or sha1 or sha256 + // remoteVolume saves the backup data. // +optional - CheckSum string `json:"checkSum,omitempty"` + RemoteVolume *corev1.Volume `json:"remoteVolume,omitempty"` - // backup check point, for incremental backup. + // backupToolName referenced backup tool name. // +optional - CheckPoint string `json:"CheckPoint,omitempty"` + BackupToolName string `json:"backupToolName,omitempty"` - // the reason if backup failed. + // manifests determines the backup metadata info // +optional - FailureReason string `json:"failureReason,omitempty"` + Manifests *ManifestsStatus `json:"manifests,omitempty"` +} - // remoteVolume saves the backup data. +type ManifestsStatus struct { + // backupLog records startTime and stopTime of data logging // +optional - RemoteVolume *corev1.Volume `json:"remoteVolume,omitempty"` + BackupLog *BackupLogStatus `json:"backupLog,omitempty"` + + // target records the target cluster metadata string, which are in JSON format. + // +optional + Target string `json:"target,omitempty"` + // snapshot records the volume snapshot metadata + // +optional + Snapshot *BackupSnapshotStatus `json:"backupSnapshot,omitempty"` + + // backupTool records information about backup files generated by the backup tool. + // +optional + BackupTool *BackupToolManifestsStatus `json:"backupTool,omitempty"` + + // userContext stores some loosely structured and extensible information. + // +optional + UserContext map[string]string `json:"userContext,omitempty"` +} + +type BackupLogStatus struct { + // startTime record start time of data logging + // +optional + StartTime *metav1.Time `json:"startTime,omitempty"` + + // stopTime record start time of data logging + // +optional + StopTime *metav1.Time `json:"stopTime,omitempty"` +} + +type BackupSnapshotStatus struct { + // volumeSnapshotName record the volumeSnapshot name + // +optional + VolumeSnapshotName string `json:"volumeSnapshotName,omitempty"` + + // volumeSnapshotContentName specifies the name of a pre-existing VolumeSnapshotContent + // object representing an existing volume snapshot. + // This field should be set if the snapshot already exists and only needs a representation in Kubernetes. + // This field is immutable. + // +optional + VolumeSnapshotContentName string `json:"volumeSnapshotContentName,omitempty"` +} + +type BackupToolManifestsStatus struct { // backupToolName referenced backup tool name. // +optional BackupToolName string `json:"backupToolName,omitempty"` + + // filePath records the file path of backup. + // +optional + FilePath string `json:"filePath,omitempty"` + + // backup upload total size + // string with capacity units in the form of "1Gi", "1Mi", "1Ki". + // +optional + UploadTotalSize string `json:"uploadTotalSize,omitempty"` + + // checksum of backup file, generated by md5 or sha1 or sha256 + // +optional + CheckSum string `json:"checkSum,omitempty"` + + // backup check point, for incremental backup. + // +optional + CheckPoint string `json:"CheckPoint,omitempty"` } // +kubebuilder:object:root=true diff --git a/apis/dataprotection/v1alpha1/backuppolicy_types.go b/apis/dataprotection/v1alpha1/backuppolicy_types.go index af90dc587..e3d2df55b 100644 --- a/apis/dataprotection/v1alpha1/backuppolicy_types.go +++ b/apis/dataprotection/v1alpha1/backuppolicy_types.go @@ -69,6 +69,10 @@ type BackupPolicySpec struct { // count of backup stop retries on fail. // +optional OnFailAttempted int32 `json:"onFailAttempted,omitempty"` + + // define how to update metadata for backup status. + // +optional + BackupStatusUpdates []BackupStatusUpdate `json:"backupStatusUpdates,omitempty"` } // TargetCluster TODO (dsj): target cluster need redefined from Cluster API @@ -122,6 +126,37 @@ type BackupPolicyHook struct { ContainerName string `json:"containerName,omitempty"` } +// BackupStatusUpdateStage defines the stage of backup status update. +// +enum +// +kubebuilder:validation:Enum={pre,post} +type BackupStatusUpdateStage string + +const ( + PRE BackupStatusUpdateStage = "pre" + POST BackupStatusUpdateStage = "post" +) + +type BackupStatusUpdate struct { + // specify the json path of backup object for patch. + // example: manifests.backupLog -- means patch the backup json path of status.manifests.backupLog. + // +optional + Path string `json:"path,omitempty"` + + // which container name that kubectl can execute. + // +optional + ContainerName string `json:"containerName,omitempty"` + + // the shell Script commands to collect backup status metadata. + // The script must exist in the container of ContainerName and the output format must be set to JSON. + // Note that outputting to stderr may cause the result format to not be in JSON. + // +optional + Script string `json:"script,omitempty"` + + // when to update the backup status, pre: before backup, post: after backup + // +optional + UpdateStage BackupStatusUpdateStage `json:"updateStage,omitempty"` +} + // BackupPolicyStatus defines the observed state of BackupPolicy type BackupPolicyStatus struct { // backup policy phase valid value: available, failed, new. diff --git a/apis/dataprotection/v1alpha1/backuppolicytemplate_types.go b/apis/dataprotection/v1alpha1/backuppolicytemplate_types.go index d0cff8178..988a34ac2 100644 --- a/apis/dataprotection/v1alpha1/backuppolicytemplate_types.go +++ b/apis/dataprotection/v1alpha1/backuppolicytemplate_types.go @@ -49,6 +49,10 @@ type BackupPolicyTemplateSpec struct { // the backupTool gets the credentials according to the user and password keyword defined by secret // +optional CredentialKeyword *BackupPolicyCredentialKeyword `json:"credentialKeyword,omitempty"` + + // define how to update metadata for backup status. + // +optional + BackupStatusUpdates []BackupStatusUpdate `json:"backupStatusUpdates,omitempty"` } // BackupPolicyCredentialKeyword defined for the target database secret that backup tool can connect. diff --git a/config/crd/bases/dataprotection.kubeblocks.io_backuppolicies.yaml b/config/crd/bases/dataprotection.kubeblocks.io_backuppolicies.yaml index 26d0bd03a..6e359243d 100644 --- a/config/crd/bases/dataprotection.kubeblocks.io_backuppolicies.yaml +++ b/config/crd/bases/dataprotection.kubeblocks.io_backuppolicies.yaml @@ -56,6 +56,33 @@ spec: fields. pattern: ^[a-z0-9]([a-z0-9\.\-]*[a-z0-9])?$ type: string + backupStatusUpdates: + description: define how to update metadata for backup status. + items: + properties: + containerName: + description: which container name that kubectl can execute. + type: string + path: + description: 'specify the json path of backup object for patch. + example: manifests.backupLog -- means patch the backup json + path of status.manifests.backupLog.' + type: string + script: + description: the shell Script commands to collect backup status + metadata. The script must exist in the container of ContainerName + and the output format must be set to JSON. Note that outputting + to stderr may cause the result format to not be in JSON. + type: string + updateStage: + description: 'when to update the backup status, pre: before + backup, post: after backup' + enum: + - pre + - post + type: string + type: object + type: array backupToolName: description: which backup tool to perform database backup, only support one tool. diff --git a/config/crd/bases/dataprotection.kubeblocks.io_backuppolicytemplates.yaml b/config/crd/bases/dataprotection.kubeblocks.io_backuppolicytemplates.yaml index ae9fde8d6..ea15b74c6 100644 --- a/config/crd/bases/dataprotection.kubeblocks.io_backuppolicytemplates.yaml +++ b/config/crd/bases/dataprotection.kubeblocks.io_backuppolicytemplates.yaml @@ -38,6 +38,33 @@ spec: spec: description: BackupPolicyTemplateSpec defines the desired state of BackupPolicyTemplate properties: + backupStatusUpdates: + description: define how to update metadata for backup status. + items: + properties: + containerName: + description: which container name that kubectl can execute. + type: string + path: + description: 'specify the json path of backup object for patch. + example: manifests.backupLog -- means patch the backup json + path of status.manifests.backupLog.' + type: string + script: + description: the shell Script commands to collect backup status + metadata. The script must exist in the container of ContainerName + and the output format must be set to JSON. Note that outputting + to stderr may cause the result format to not be in JSON. + type: string + updateStage: + description: 'when to update the backup status, pre: before + backup, post: after backup' + enum: + - pre + - post + type: string + type: object + type: array backupToolName: description: which backup tool to perform database backup, only support one tool. diff --git a/config/crd/bases/dataprotection.kubeblocks.io_backups.yaml b/config/crd/bases/dataprotection.kubeblocks.io_backups.yaml index b15187d8b..188fb3fb3 100644 --- a/config/crd/bases/dataprotection.kubeblocks.io_backups.yaml +++ b/config/crd/bases/dataprotection.kubeblocks.io_backups.yaml @@ -73,7 +73,7 @@ spec: description: if backupType is incremental, parentBackupName is required. type: string ttl: - description: TTL is a time.Duration-parsable string describing how + description: ttl is a time.Duration-parsable string describing how long the Backup should be retained for. type: string required: @@ -83,16 +83,9 @@ spec: status: description: BackupStatus defines the observed state of Backup properties: - CheckPoint: - description: backup check point, for incremental backup. - type: string backupToolName: description: backupToolName referenced backup tool name. type: string - checkSum: - description: checksum of backup file, generated by md5 or sha1 or - sha256 - type: string completionTimestamp: description: Date/time when the backup finished being processed. format: date-time @@ -110,6 +103,70 @@ spec: failureReason: description: the reason if backup failed. type: string + manifests: + description: manifests determines the backup metadata info + properties: + backupLog: + description: backupLog records startTime and stopTime of data + logging + properties: + startTime: + description: startTime record start time of data logging + format: date-time + type: string + stopTime: + description: stopTime record start time of data logging + format: date-time + type: string + type: object + backupSnapshot: + description: snapshot records the volume snapshot metadata + properties: + volumeSnapshotContentName: + description: volumeSnapshotContentName specifies the name + of a pre-existing VolumeSnapshotContent object representing + an existing volume snapshot. This field should be set if + the snapshot already exists and only needs a representation + in Kubernetes. This field is immutable. + type: string + volumeSnapshotName: + description: volumeSnapshotName record the volumeSnapshot + name + type: string + type: object + backupTool: + description: backupTool records information about backup files + generated by the backup tool. + properties: + CheckPoint: + description: backup check point, for incremental backup. + type: string + backupToolName: + description: backupToolName referenced backup tool name. + type: string + checkSum: + description: checksum of backup file, generated by md5 or + sha1 or sha256 + type: string + filePath: + description: filePath records the file path of backup. + type: string + uploadTotalSize: + description: backup upload total size string with capacity + units in the form of "1Gi", "1Mi", "1Ki". + type: string + type: object + target: + description: target records the target cluster metadata string, + which are in JSON format. + type: string + userContext: + additionalProperties: + type: string + description: userContext stores some loosely structured and extensible + information. + type: object + type: object parentBackupName: description: record parentBackupName if backupType is incremental. type: string @@ -1659,10 +1716,6 @@ spec: description: backup total size string with capacity units in the form of "1Gi", "1Mi", "1Ki". type: string - uploadTotalSize: - description: backup total size string with capacity units in the form - of "1Gi", "1Mi", "1Ki". - type: string type: object type: object served: true diff --git a/controllers/dataprotection/backup_controller.go b/controllers/dataprotection/backup_controller.go index 3e3ab387a..d6810e785 100644 --- a/controllers/dataprotection/backup_controller.go +++ b/controllers/dataprotection/backup_controller.go @@ -47,6 +47,7 @@ import ( appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" dataprotectionv1alpha1 "github.com/apecloud/kubeblocks/apis/dataprotection/v1alpha1" "github.com/apecloud/kubeblocks/internal/constant" + ctrlbuilder "github.com/apecloud/kubeblocks/internal/controller/builder" intctrlutil "github.com/apecloud/kubeblocks/internal/controllerutil" ) @@ -209,6 +210,9 @@ func (r *BackupReconciler) doInProgressPhaseAction( if !isOK { return intctrlutil.RequeueAfter(reconcileInterval, reqCtx.Log, "") } + if err = r.createUpdatesJobs(reqCtx, backup, dataprotectionv1alpha1.PRE); err != nil { + r.Recorder.Event(backup, corev1.EventTypeNormal, "CreatedPreUpdatesJob", err.Error()) + } if err = r.createVolumeSnapshot(reqCtx, backup); err != nil { return r.updateStatusIfFailed(reqCtx, backup, err) } @@ -231,6 +235,11 @@ func (r *BackupReconciler) doInProgressPhaseAction( return intctrlutil.RequeueAfter(reconcileInterval, reqCtx.Log, "") } + // Failure MetadataCollectionJob does not affect the backup status. + if err = r.createUpdatesJobs(reqCtx, backup, dataprotectionv1alpha1.POST); err != nil { + r.Recorder.Event(backup, corev1.EventTypeNormal, "CreatedPostUpdatesJob", err.Error()) + } + backup.Status.Phase = dataprotectionv1alpha1.BackupCompleted backup.Status.CompletionTimestamp = &metav1.Time{Time: r.clock.Now().UTC()} snap := &snapshotv1.VolumeSnapshot{} @@ -452,7 +461,7 @@ func (r *BackupReconciler) createVolumeSnapshot( } reqCtx.Log.V(1).Info("create a volumeSnapshot from backup", "snapshot", snap.Name) - if err := r.Client.Create(reqCtx.Ctx, snap); err != nil { + if err = r.Client.Create(reqCtx.Ctx, snap); err != nil && !apierrors.IsAlreadyExists(err) { return err } } @@ -483,6 +492,54 @@ func (r *BackupReconciler) ensureVolumeSnapshotReady(reqCtx intctrlutil.RequestC return ready, nil } +func (r *BackupReconciler) createUpdatesJobs(reqCtx intctrlutil.RequestCtx, + backup *dataprotectionv1alpha1.Backup, stage dataprotectionv1alpha1.BackupStatusUpdateStage) error { + // get backup policy + backupPolicy := &dataprotectionv1alpha1.BackupPolicy{} + backupPolicyNameSpaceName := types.NamespacedName{ + Namespace: reqCtx.Req.Namespace, + Name: backup.Spec.BackupPolicyName, + } + if err := r.Get(reqCtx.Ctx, backupPolicyNameSpaceName, backupPolicy); err != nil { + reqCtx.Log.V(1).Error(err, "Unable to get backupPolicy for backup.", "backupPolicy", backupPolicyNameSpaceName) + return err + } + for _, update := range backupPolicy.Spec.BackupStatusUpdates { + if update.UpdateStage != stage { + continue + } + if err := r.createMetadataCollectionJob(reqCtx, backup, update); err != nil { + return err + } + } + return nil +} + +func (r *BackupReconciler) createMetadataCollectionJob(reqCtx intctrlutil.RequestCtx, + backup *dataprotectionv1alpha1.Backup, updateInfo dataprotectionv1alpha1.BackupStatusUpdate) error { + mgrNS := viper.GetString(constant.CfgKeyCtrlrMgrNS) + key := types.NamespacedName{Namespace: mgrNS, Name: backup.Name + "-" + strings.ToLower(updateInfo.Path)} + job := &batchv1.Job{} + // check if job is created + if exists, err := intctrlutil.CheckResourceExists(reqCtx.Ctx, r.Client, key, job); err != nil { + return err + } else if exists { + return nil + } + + // build job and create + jobPodSpec, err := r.buildMetadataCollectionPodSpec(reqCtx, backup, updateInfo) + if err != nil { + return err + } + if job, err = ctrlbuilder.BuildBackupManifestsJob(key, backup, &jobPodSpec); err != nil { + return err + } + msg := fmt.Sprintf("creating job %s", key.Name) + r.Recorder.Event(backup, corev1.EventTypeNormal, "CreatingJob-"+key.Name, msg) + return client.IgnoreAlreadyExists(r.Client.Create(reqCtx.Ctx, job)) +} + func (r *BackupReconciler) createBackupToolJob( reqCtx intctrlutil.RequestCtx, backup *dataprotectionv1alpha1.Backup) error { @@ -594,6 +651,7 @@ func (r *BackupReconciler) createBatchV1Job( backup *dataprotectionv1alpha1.Backup, templatePodSpec corev1.PodSpec) error { + backOffLimit := int32(3) job := &batchv1.Job{ // TypeMeta: metav1.TypeMeta{Kind: "Job", APIVersion: "batch/v1"}, ObjectMeta: metav1.ObjectMeta{ @@ -608,15 +666,13 @@ func (r *BackupReconciler) createBatchV1Job( Name: key.Name}, Spec: templatePodSpec, }, + BackoffLimit: &backOffLimit, }, } controllerutil.AddFinalizer(job, dataProtectionFinalizerName) reqCtx.Log.V(1).Info("create a built-in job from backup", "job", job) - if err := r.Client.Create(reqCtx.Ctx, job); err != nil { - return err - } - return nil + return client.IgnoreAlreadyExists(r.Client.Create(reqCtx.Ctx, job)) } func (r *BackupReconciler) getBatchV1Job(reqCtx intctrlutil.RequestCtx, backup *dataprotectionv1alpha1.Backup) (*batchv1.Job, error) { @@ -937,21 +993,83 @@ func (r *BackupReconciler) buildSnapshotPodSpec( podSpec.RestartPolicy = corev1.RestartPolicyNever podSpec.ServiceAccountName = viper.GetString("KUBEBLOCKS_SERVICEACCOUNT_NAME") + if err = addTolerations(&podSpec); err != nil { + return podSpec, err + } + + return podSpec, nil +} + +func generateJSON(path string, value string) string { + segments := strings.Split(path, ".") + jsonString := value + for i := len(segments) - 1; i >= 0; i-- { + jsonString = fmt.Sprintf(`{\"%s\":%s}`, segments[i], jsonString) + } + return jsonString +} + +func addTolerations(podSpec *corev1.PodSpec) (err error) { if cmTolerations := viper.GetString(constant.CfgKeyCtrlrMgrTolerations); cmTolerations != "" { if err = json.Unmarshal([]byte(cmTolerations), &podSpec.Tolerations); err != nil { - return podSpec, err + return err } } if cmAffinity := viper.GetString(constant.CfgKeyCtrlrMgrAffinity); cmAffinity != "" { if err = json.Unmarshal([]byte(cmAffinity), &podSpec.Affinity); err != nil { - return podSpec, err + return err } } if cmNodeSelector := viper.GetString(constant.CfgKeyCtrlrMgrNodeSelector); cmNodeSelector != "" { if err = json.Unmarshal([]byte(cmNodeSelector), &podSpec.NodeSelector); err != nil { - return podSpec, err + return err } } + return nil +} + +func (r *BackupReconciler) buildMetadataCollectionPodSpec( + reqCtx intctrlutil.RequestCtx, + backup *dataprotectionv1alpha1.Backup, + updateInfo dataprotectionv1alpha1.BackupStatusUpdate) (corev1.PodSpec, error) { + podSpec := corev1.PodSpec{} + logger := reqCtx.Log + + // get backup policy + backupPolicy := &dataprotectionv1alpha1.BackupPolicy{} + backupPolicyNameSpaceName := types.NamespacedName{ + Namespace: reqCtx.Req.Namespace, + Name: backup.Spec.BackupPolicyName, + } + + if err := r.Get(reqCtx.Ctx, backupPolicyNameSpaceName, backupPolicy); err != nil { + logger.Error(err, "Unable to get backupPolicy for backup.", "backupPolicy", backupPolicyNameSpaceName) + return podSpec, err + } + targetPod, err := r.getTargetPod(reqCtx, backup, backupPolicy.Spec.Target.LabelsSelector.MatchLabels) + if err != nil { + return podSpec, err + } + + container := corev1.Container{} + container.Name = backup.Name + container.Command = []string{"sh", "-c"} + args := "set -o errexit; set -o nounset;" + + "OUTPUT=$(kubectl -n %s exec -it pod/%s -c %s -- %s);" + + "kubectl -n %s patch backup %s --subresource=status --type=merge --patch \"%s\";" + patchJSON := generateJSON("status."+updateInfo.Path, "$OUTPUT") + args = fmt.Sprintf(args, targetPod.Namespace, targetPod.Name, updateInfo.ContainerName, + updateInfo.Script, backup.Namespace, backup.Name, patchJSON) + container.Args = []string{args} + container.Image = viper.GetString(constant.KBToolsImage) + container.ImagePullPolicy = corev1.PullPolicy(viper.GetString(constant.KBImagePullPolicy)) + podSpec.Containers = []corev1.Container{container} + podSpec.RestartPolicy = corev1.RestartPolicyNever + podSpec.ServiceAccountName = viper.GetString("KUBEBLOCKS_SERVICEACCOUNT_NAME") + + if err = addTolerations(&podSpec); err != nil { + return podSpec, err + } return podSpec, nil } diff --git a/controllers/dataprotection/backup_controller_test.go b/controllers/dataprotection/backup_controller_test.go index d86273ed5..6d640e5c7 100644 --- a/controllers/dataprotection/backup_controller_test.go +++ b/controllers/dataprotection/backup_controller_test.go @@ -17,6 +17,8 @@ limitations under the License. package dataprotection import ( + "strings" + . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" @@ -24,7 +26,6 @@ import ( "github.com/spf13/viper" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" @@ -34,11 +35,10 @@ import ( testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" ) -var _ = Describe("Backup for a StatefulSet", func() { +var _ = Describe("Backup Controller test", func() { const clusterName = "wesql-cluster" const componentName = "replicasets-primary" const containerName = "mysql" - const defaultPVCSize = "1Gi" const backupPolicyName = "test-backup-policy" const backupRemoteVolumeName = "backup-remote-volume" const backupRemotePVCName = "backup-remote-pvc" @@ -60,7 +60,6 @@ var _ = Describe("Backup for a StatefulSet", func() { ml := client.HasLabels{testCtx.TestObjLabelKey} // namespaced testapps.ClearResources(&testCtx, intctrlutil.ClusterSignature, inNS, ml) - testapps.ClearResources(&testCtx, intctrlutil.StatefulSetSignature, inNS, ml) testapps.ClearResources(&testCtx, intctrlutil.PodSignature, inNS, ml) testapps.ClearResources(&testCtx, intctrlutil.BackupSignature, inNS, ml) testapps.ClearResources(&testCtx, intctrlutil.BackupPolicySignature, inNS, ml) @@ -80,33 +79,24 @@ var _ = Describe("Backup for a StatefulSet", func() { By("mock a cluster") testapps.NewClusterFactory(testCtx.DefaultNamespace, clusterName, "test-cd", "test-cv").Create(&testCtx) - - By("By mocking a statefulset") - sts := testapps.NewStatefulSetFactory(testCtx.DefaultNamespace, clusterName+"-"+componentName, clusterName, componentName). - AddAppInstanceLabel(clusterName). - AddContainer(corev1.Container{Name: containerName, Image: testapps.ApeCloudMySQLImage}). - AddVolumeClaimTemplate(corev1.PersistentVolumeClaim{ - ObjectMeta: metav1.ObjectMeta{Name: testapps.DataVolumeName}, - Spec: testapps.NewPVC(defaultPVCSize), - }).Create(&testCtx).GetObject() - + podGenerateName := clusterName + "-" + componentName By("By mocking a pvc belonging to the pod") pvc := testapps.NewPersistentVolumeClaimFactory( - testCtx.DefaultNamespace, "data-"+sts.Name+"-0", clusterName, componentName, "data"). + testCtx.DefaultNamespace, "data-"+podGenerateName+"-0", clusterName, componentName, "data"). SetStorage("1Gi"). Create(&testCtx).GetObject() pvcName = pvc.Name By("By mocking a pvc belonging to the pod2") pvc2 := testapps.NewPersistentVolumeClaimFactory( - testCtx.DefaultNamespace, "data-"+sts.Name+"-1", clusterName, componentName, "data"). + testCtx.DefaultNamespace, "data-"+podGenerateName+"-1", clusterName, componentName, "data"). SetStorage("1Gi"). Create(&testCtx).GetObject() By("By mocking a pod belonging to the statefulset") volume := corev1.Volume{Name: pvc.Name, VolumeSource: corev1.VolumeSource{ PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ClaimName: pvc.Name}}} - pod := testapps.NewPodFactory(testCtx.DefaultNamespace, sts.Name+"-0"). + pod := testapps.NewPodFactory(testCtx.DefaultNamespace, podGenerateName+"-0"). AddAppInstanceLabel(clusterName). AddRoleLabel("leader"). AddAppComponentLabel(componentName). @@ -118,7 +108,7 @@ var _ = Describe("Backup for a StatefulSet", func() { By("By mocking a pod 2 belonging to the statefulset") volume2 := corev1.Volume{Name: pvc2.Name, VolumeSource: corev1.VolumeSource{ PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ClaimName: pvc2.Name}}} - _ = testapps.NewPodFactory(testCtx.DefaultNamespace, sts.Name+"-1"). + _ = testapps.NewPodFactory(testCtx.DefaultNamespace, podGenerateName+"-1"). AddAppInstanceLabel(clusterName). AddAppComponentLabel(componentName). AddContainer(corev1.Container{Name: containerName, Image: testapps.ApeCloudMySQLImage}). @@ -223,6 +213,9 @@ var _ = Describe("Backup for a StatefulSet", func() { }) It("should success after all jobs complete", func() { + backupPolicyKey := types.NamespacedName{Name: backupPolicyName, Namespace: backupKey.Namespace} + patchBackupPolicySpecBackupStatusUpdates(backupPolicyKey) + preJobKey := types.NamespacedName{Name: backupKey.Name + "-pre", Namespace: backupKey.Namespace} postJobKey := types.NamespacedName{Name: backupKey.Name + "-post", Namespace: backupKey.Namespace} patchK8sJobStatus(preJobKey, batchv1.JobComplete) @@ -237,6 +230,9 @@ var _ = Describe("Backup for a StatefulSet", func() { patchVolumeSnapshotStatus(backupKey, true) patchK8sJobStatus(postJobKey, batchv1.JobComplete) + logJobKey := types.NamespacedName{Name: backupKey.Name + "-" + strings.ToLower("manifests.backupLog"), Namespace: backupKey.Namespace} + patchK8sJobStatus(logJobKey, batchv1.JobComplete) + By("Check backup job completed") Eventually(testapps.CheckObj(&testCtx, backupKey, func(g Gomega, fetched *dataprotectionv1alpha1.Backup) { g.Expect(fetched.Status.Phase).To(Equal(dataprotectionv1alpha1.BackupCompleted)) @@ -411,3 +407,22 @@ func patchVolumeSnapshotStatus(key types.NamespacedName, readyToUse bool) { fetched.Status = &snapStatus })).Should(Succeed()) } + +func patchBackupPolicySpecBackupStatusUpdates(key types.NamespacedName) { + Eventually(testapps.GetAndChangeObj(&testCtx, key, func(fetched *dataprotectionv1alpha1.BackupPolicy) { + fetched.Spec.BackupStatusUpdates = []dataprotectionv1alpha1.BackupStatusUpdate{ + { + Path: "manifests.backupLog", + ContainerName: "postgresql", + Script: "echo {\"startTime\": \"2023-03-01T00:00:00Z\", \"stopTime\": \"2023-03-01T00:00:00Z\"}", + UpdateStage: dataprotectionv1alpha1.PRE, + }, + { + Path: "manifests.backupTool", + ContainerName: "postgresql", + Script: "echo {\"FilePath\": \"/backup/test.file\"}", + UpdateStage: dataprotectionv1alpha1.POST, + }, + } + })).Should(Succeed()) +} diff --git a/controllers/dataprotection/backuppolicy_controller.go b/controllers/dataprotection/backuppolicy_controller.go index 80501e672..3698baab9 100644 --- a/controllers/dataprotection/backuppolicy_controller.go +++ b/controllers/dataprotection/backuppolicy_controller.go @@ -268,6 +268,9 @@ func (r *BackupPolicyReconciler) mergeBackupPolicyTemplate( if backupPolicy.Spec.OnFailAttempted == 0 { backupPolicy.Spec.OnFailAttempted = template.Spec.OnFailAttempted } + if len(backupPolicy.Spec.BackupStatusUpdates) == 0 { + backupPolicy.Spec.BackupStatusUpdates = template.Spec.BackupStatusUpdates + } return nil } diff --git a/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicies.yaml b/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicies.yaml index 26d0bd03a..6e359243d 100644 --- a/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicies.yaml +++ b/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicies.yaml @@ -56,6 +56,33 @@ spec: fields. pattern: ^[a-z0-9]([a-z0-9\.\-]*[a-z0-9])?$ type: string + backupStatusUpdates: + description: define how to update metadata for backup status. + items: + properties: + containerName: + description: which container name that kubectl can execute. + type: string + path: + description: 'specify the json path of backup object for patch. + example: manifests.backupLog -- means patch the backup json + path of status.manifests.backupLog.' + type: string + script: + description: the shell Script commands to collect backup status + metadata. The script must exist in the container of ContainerName + and the output format must be set to JSON. Note that outputting + to stderr may cause the result format to not be in JSON. + type: string + updateStage: + description: 'when to update the backup status, pre: before + backup, post: after backup' + enum: + - pre + - post + type: string + type: object + type: array backupToolName: description: which backup tool to perform database backup, only support one tool. diff --git a/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicytemplates.yaml b/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicytemplates.yaml index ae9fde8d6..ea15b74c6 100644 --- a/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicytemplates.yaml +++ b/deploy/helm/crds/dataprotection.kubeblocks.io_backuppolicytemplates.yaml @@ -38,6 +38,33 @@ spec: spec: description: BackupPolicyTemplateSpec defines the desired state of BackupPolicyTemplate properties: + backupStatusUpdates: + description: define how to update metadata for backup status. + items: + properties: + containerName: + description: which container name that kubectl can execute. + type: string + path: + description: 'specify the json path of backup object for patch. + example: manifests.backupLog -- means patch the backup json + path of status.manifests.backupLog.' + type: string + script: + description: the shell Script commands to collect backup status + metadata. The script must exist in the container of ContainerName + and the output format must be set to JSON. Note that outputting + to stderr may cause the result format to not be in JSON. + type: string + updateStage: + description: 'when to update the backup status, pre: before + backup, post: after backup' + enum: + - pre + - post + type: string + type: object + type: array backupToolName: description: which backup tool to perform database backup, only support one tool. diff --git a/deploy/helm/crds/dataprotection.kubeblocks.io_backups.yaml b/deploy/helm/crds/dataprotection.kubeblocks.io_backups.yaml index b15187d8b..188fb3fb3 100644 --- a/deploy/helm/crds/dataprotection.kubeblocks.io_backups.yaml +++ b/deploy/helm/crds/dataprotection.kubeblocks.io_backups.yaml @@ -73,7 +73,7 @@ spec: description: if backupType is incremental, parentBackupName is required. type: string ttl: - description: TTL is a time.Duration-parsable string describing how + description: ttl is a time.Duration-parsable string describing how long the Backup should be retained for. type: string required: @@ -83,16 +83,9 @@ spec: status: description: BackupStatus defines the observed state of Backup properties: - CheckPoint: - description: backup check point, for incremental backup. - type: string backupToolName: description: backupToolName referenced backup tool name. type: string - checkSum: - description: checksum of backup file, generated by md5 or sha1 or - sha256 - type: string completionTimestamp: description: Date/time when the backup finished being processed. format: date-time @@ -110,6 +103,70 @@ spec: failureReason: description: the reason if backup failed. type: string + manifests: + description: manifests determines the backup metadata info + properties: + backupLog: + description: backupLog records startTime and stopTime of data + logging + properties: + startTime: + description: startTime record start time of data logging + format: date-time + type: string + stopTime: + description: stopTime record start time of data logging + format: date-time + type: string + type: object + backupSnapshot: + description: snapshot records the volume snapshot metadata + properties: + volumeSnapshotContentName: + description: volumeSnapshotContentName specifies the name + of a pre-existing VolumeSnapshotContent object representing + an existing volume snapshot. This field should be set if + the snapshot already exists and only needs a representation + in Kubernetes. This field is immutable. + type: string + volumeSnapshotName: + description: volumeSnapshotName record the volumeSnapshot + name + type: string + type: object + backupTool: + description: backupTool records information about backup files + generated by the backup tool. + properties: + CheckPoint: + description: backup check point, for incremental backup. + type: string + backupToolName: + description: backupToolName referenced backup tool name. + type: string + checkSum: + description: checksum of backup file, generated by md5 or + sha1 or sha256 + type: string + filePath: + description: filePath records the file path of backup. + type: string + uploadTotalSize: + description: backup upload total size string with capacity + units in the form of "1Gi", "1Mi", "1Ki". + type: string + type: object + target: + description: target records the target cluster metadata string, + which are in JSON format. + type: string + userContext: + additionalProperties: + type: string + description: userContext stores some loosely structured and extensible + information. + type: object + type: object parentBackupName: description: record parentBackupName if backupType is incremental. type: string @@ -1659,10 +1716,6 @@ spec: description: backup total size string with capacity units in the form of "1Gi", "1Mi", "1Ki". type: string - uploadTotalSize: - description: backup total size string with capacity units in the form - of "1Gi", "1Mi", "1Ki". - type: string type: object type: object served: true diff --git a/internal/controller/builder/builder.go b/internal/controller/builder/builder.go index cf2fdde97..a5bd768f7 100644 --- a/internal/controller/builder/builder.go +++ b/internal/controller/builder/builder.go @@ -705,3 +705,20 @@ func BuildTLSSecret(namespace, clusterName, componentName string) (*corev1.Secre } return secret, nil } + +func BuildBackupManifestsJob(key types.NamespacedName, backup *dataprotectionv1alpha1.Backup, podSpec *corev1.PodSpec) (*batchv1.Job, error) { + const tplFile = "backup_manifests_template.cue" + + job := &batchv1.Job{} + if err := buildFromCUE(tplFile, + map[string]any{ + "job.metadata.name": key.Name, + "job.metadata.namespace": key.Namespace, + "backup": backup, + "podSpec": podSpec, + }, + "job", job); err != nil { + return nil, err + } + return job, nil +} diff --git a/internal/controller/builder/cue/backup_manifests_template.cue b/internal/controller/builder/cue/backup_manifests_template.cue new file mode 100644 index 000000000..852bed36f --- /dev/null +++ b/internal/controller/builder/cue/backup_manifests_template.cue @@ -0,0 +1,47 @@ +// Copyright ApeCloud, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +backup: { + metadata: { + name: string + namespace: string + } +} + +podSpec: { + containers: [...] + volumes: [...] + env: [...] + restartPolicy: "Never" + securityContext: + runAsUser: 0 +} + +job: { + apiVersion: "batch/v1" + kind: "Job" + metadata: { + name: string + namespace: string + labels: + "app.kubernetes.io/managed-by": "kubeblocks" + } + spec: { + template: { + spec: podSpec + } + backOffLimit: 3 + ttlSecondsAfterFinished: 10 + } +} From d1b11417b6930762009eca9b6c589928e090c297 Mon Sep 17 00:00:00 2001 From: shaojiang Date: Mon, 10 Apr 2023 19:21:02 +0800 Subject: [PATCH 66/80] feat: add postgresql scripts for backup manifests collection (#2215) --- deploy/postgresql/templates/backuppolicytemplate.yaml | 10 ++++++++++ deploy/postgresql/templates/scripts.yaml | 11 ++++++++++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/deploy/postgresql/templates/backuppolicytemplate.yaml b/deploy/postgresql/templates/backuppolicytemplate.yaml index 42643e0f5..48a8be568 100644 --- a/deploy/postgresql/templates/backuppolicytemplate.yaml +++ b/deploy/postgresql/templates/backuppolicytemplate.yaml @@ -13,3 +13,13 @@ spec: credentialKeyword: userKeyword: username passwordKeyword: password + + hooks: + containerName: postgresql + preCommands: + - psql -c "CHECKPOINT" + backupStatusUpdates: + - path: manifests.backupLog + containerName: postgresql + script: /scripts/backup-log-collector.sh + updateStage: pre diff --git a/deploy/postgresql/templates/scripts.yaml b/deploy/postgresql/templates/scripts.yaml index d10c632e1..81d4f7ac9 100644 --- a/deploy/postgresql/templates/scripts.yaml +++ b/deploy/postgresql/templates/scripts.yaml @@ -70,4 +70,13 @@ data: # add recovery.signal/standby.signal to trigger recovery cp ${PGDATA}/../init-scripts/kb_restore.sh /docker-entrypoint-preinitdb.d fi - /opt/bitnami/scripts/postgresql/entrypoint.sh /opt/bitnami/scripts/postgresql/run.sh \ No newline at end of file + /opt/bitnami/scripts/postgresql/entrypoint.sh /opt/bitnami/scripts/postgresql/run.sh + backup-log-collector.sh: | + #!/bin/bash + set -o errexit + set -o nounset + LOG_START_TIME=$(pg_waldump $(ls -tr /postgresql/data/pg_wal/ | grep '[[:digit:]]$'|head -n 1) --rmgr=Transaction 2>/dev/null |head -n 1|awk -F ' COMMIT ' '{print $2}'|awk -F ';' '{print $1}') + LOG_STOP_TIME=$(pg_waldump $(ls -t /postgresql/data/pg_wal/ | grep '[[:digit:]]$'|head -n 1) --rmgr=Transaction 2>/dev/null |tail -n 1|awk -F ' COMMIT ' '{print $2}'|awk -F ';' '{print $1}') + LOG_START_TIME=$(date -d "$LOG_START_TIME" -u '+%Y-%m-%dT%H:%M:%SZ') + LOG_STOP_TIME=$(date -d "$LOG_STOP_TIME" -u '+%Y-%m-%dT%H:%M:%SZ') + printf "{\"startTime\": \"$LOG_START_TIME\" ,\"stopTime\": \"$LOG_STOP_TIME\"}" \ No newline at end of file From 8f6f9371833ac36b626286cf7f96480f87c2b09a Mon Sep 17 00:00:00 2001 From: a le <101848970+1aal@users.noreply.github.com> Date: Mon, 10 Apr 2023 20:38:29 +0800 Subject: [PATCH 67/80] fix: github action uploads kbcli asset for windows and add powershell script to install on windows (#2449) --- .github/workflows/release-publish.yml | 59 ++++--- hack/install_cli.ps1 | 218 ++++++++++++++++++++++++++ 2 files changed, 252 insertions(+), 25 deletions(-) create mode 100644 hack/install_cli.ps1 diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index 3420c27ba..6ad92b85e 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -67,35 +67,46 @@ jobs: uses: bruceadams/get-release@v1.3.2 - name: make build - env: - CLI_BINARY: ${{ env.CLI_NAME }}-${{ matrix.os }}-${{ env.TAG_NAME }}.tar.gz - run: | - mkdir -p ${{ matrix.os }} - + run: | CLI_OS_ARCH=`bash ${{ github.workspace }}/.github/utils/utils.sh \ --tag-name ${{ matrix.os }} \ --type 2` RELEASE_VERSION=`bash ${{ github.workspace }}/.github/utils/utils.sh \ --tag-name ${{ env.TAG_NAME }} \ - --type 1` - + --type 1` VERSION=$RELEASE_VERSION make bin/${{ env.CLI_NAME }}.$CLI_OS_ARCH - - mv bin/${{ env.CLI_NAME }}.$CLI_OS_ARCH ${{ matrix.os }}/${{ env.CLI_NAME }} - - tar -zcvf ${{ env.CLI_BINARY }} ${{ matrix.os }} - - mv ${{ env.CLI_BINARY }} bin/ + mkdir -p ${{ matrix.os }} + echo "CLI_OS_ARCH=${CLI_OS_ARCH}" >> $GITHUB_ENV + echo "CLI_FILENAME=${{ env.CLI_NAME }}-${{ matrix.os }}-${{ env.TAG_NAME }}" >> $GITHUB_ENV + echo "CLI_DIR=${{ matrix.os }}" >> $GITHUB_ENV + + - name: make zip + if: matrix.os == 'windows-amd64' + run: | + mv bin/${{ env.CLI_NAME }}.${{ env.CLI_OS_ARCH }} ${{ matrix.os }}/${{ env.CLI_NAME }}.exe + zip -r -o ${{ env.CLI_FILENAME }}.zip ${{ env.CLI_DIR }} + file ${{ env.CLI_FILENAME }}.zip # for debug + mv ${{ env.CLI_FILENAME }}.zip bin/ + echo "ASSET_NAME=${{ env.CLI_FILENAME }}.zip" >> $GITHUB_ENV + echo "ASSET_CONTENT_TYPE=application/zip" >> $GITHUB_ENV + + - name: make tar + if: matrix.os != 'windows-amd64' + run: | + mv bin/${{ env.CLI_NAME }}.${{ env.CLI_OS_ARCH }} ${{ matrix.os }}/${{ env.CLI_NAME }} + tar -zcvf ${{ env.CLI_FILENAME }}.tar.gz ${{ env.CLI_DIR }} + file ${{ env.CLI_FILENAME }}.tar.gz # for debug + mv ${{ env.CLI_FILENAME }}.tar.gz bin/ + echo "ASSET_NAME=${{ env.CLI_FILENAME }}.tar.gz" >> $GITHUB_ENV + echo "ASSET_CONTENT_TYPE=application/gzip" >> $GITHUB_ENV - name: upload release asset ${{ matrix.os }} uses: actions/upload-release-asset@main - env: - CLI_BINARY: ${{ env.CLI_NAME }}-${{ matrix.os }}-${{ env.TAG_NAME }}.tar.gz with: upload_url: ${{ steps.get_release.outputs.upload_url }} - asset_path: ./bin/${{ env.CLI_BINARY }} - asset_name: ${{ env.CLI_BINARY }} - asset_content_type: application/zip + asset_path: ./bin/${{ env.ASSET_NAME }} + asset_name: ${{ env.ASSET_NAME }} + asset_content_type: ${{ env.ASSET_CONTENT_TYPE }} - name: get release kbcli upload url run: | @@ -107,13 +118,11 @@ jobs: - name: upload release kbcli asset ${{ matrix.os }} uses: actions/upload-release-asset@main - env: - CLI_BINARY: ${{ env.CLI_NAME }}-${{ matrix.os }}-${{ env.TAG_NAME }}.tar.gz with: upload_url: ${{ env.UPLOAD_URL }} - asset_path: ./bin/${{ env.CLI_BINARY }} - asset_name: ${{ env.CLI_BINARY }} - asset_content_type: application/zip + asset_path: ./bin/${{ env.ASSET_NAME }} + asset_name: ${{ env.ASSET_NAME }} + asset_content_type: ${{ env.ASSET_CONTENT_TYPE }} - name: upload gitlab kbcli asset ${{ matrix.os }} env: @@ -123,6 +132,6 @@ jobs: --type 2 \ --project-id ${{ env.GITLAB_KBCLI_PROJECT_ID }} \ --tag-name ${{ env.TAG_NAME }} \ - --asset-path ./bin/${{ env.CLI_BINARY }} \ - --asset-name ${{ env.CLI_BINARY }} \ + --asset-path ./bin/${{ env.ASSET_NAME }} \ + --asset-name ${{ env.ASSET_NAME }} \ --access-token ${{ secrets.GITLAB_ACCESS_TOKEN }} diff --git a/hack/install_cli.ps1 b/hack/install_cli.ps1 new file mode 100644 index 000000000..b742c426c --- /dev/null +++ b/hack/install_cli.ps1 @@ -0,0 +1,218 @@ +# kbcli filename +$CLI_FILENAME = "kbcli" + +$CLI_TMP_ROOT = "" +$ARTIFACT_TMP_FILE = "" + +$REPO = "apecloud/kbcli" +$GITHUB = "https://api.github.com" +$GITLAB_REPO = "85948" +$GITLAB = "https://jihulab.com/api/v4/projects" +$COUNTRY_CODE = "" + +Import-Module Microsoft.PowerShell.Utility + +function getCountryCode() { + return (Invoke-WebRequest -Uri "https://ifconfig.io/country_code" -UseBasicParsing | Select-Object -ExpandProperty Content).Trim() +} + +# verify whether x86 or AMD64 ; not supported x86 +function verifySupported { + $arch = (Get-Process -Id $PID).StartInfo.EnvironmentVariables["PROCESSOR_ARCHITECTURE"] + if ($arch -eq "AMD64") { + Write-Output "AMD64 is supported..." + return + } + Write-Output "No support to your pc $arch ARCH" + exit 1 +} + +function checkExistingCLI { + if (Get-Command "kbcli" -ErrorAction SilentlyContinue) { + # kbcli has installed + $exist_cli = (Get-Command "kbcli" -CommandType Application).Source + Write-Output "kbcli is detected: $exist_cli. Please uninstall first." + exit 1 + } + else { + Write-Output "Installing kbcli..." + } +} + +function getLatestRelease { + $latest_release = "" + + if ($COUNTRY_CODE -eq "CN") { + $releaseURL = "$GITLAB/$GITLAB_REPO/repository/tags/latest" + $latest_release = (Invoke-WebRequest -Uri $releaseURL | Select-String -Pattern "message" | Select-Object -First 1).Line + $latest_release = ($latest_release -split ",")[1].Trim() + $latest_release = ($latest_release -split ":")[1].Trim() + $latest_release = $latest_release.Trim('"') + + } + else { + $releaseURL = "$GITHUB/repos/$REPO/releases/latest" + $response = Invoke-WebRequest -Uri $releaseURL -ContentType "application/json" | Select-String -Pattern "tag_name" + $json = $response | ConvertFrom-Json + $latest_release = $json.tag_name + + } + return $latest_release +} + +$webClient = New-Object System.Net.WebClient +$isDownLoaded = $False +$Data = +$timeout = New-TimeSpan -Seconds 60 +function downloadFile { + param ( + $LATEST_RELEASE_TAG + ) + $CLI_ARTIFACT="${CLI_FILENAME}-windows-amd64-${LATEST_RELEASE_TAG}.zip" + $DOWNLOAD_BASE = "https://github.com/$REPO/releases/download" + if ($COUNTRY_CODE -eq "CN") { + $DOWNLOAD_BASE = "$GITLAB/$GITLAB_REPO/packages/generic/kubeblocks" + } + $DOWNLOAD_URL = "${DOWNLOAD_BASE}/${LATEST_RELEASE_TAG}/${CLI_ARTIFACT}" + # Check the Resource + # Write-Host DOWNLOAD_URL = $DOWNLOAD_URL + + $webRequest = [System.Net.HttpWebRequest]::Create($DOWNLOAD_URL) + $webRequest.Method = "HEAD" + try { + $webResponse = $webRequest.GetResponse() + Write-Host "Resource has been found" + } catch { + Write-Host "Resource not found." + exit 1 + } + # Create the temp directory + $CLI_TMP_ROOT = New-Item -ItemType Directory -Path (Join-Path $env:TEMP "kbcli-install-$(Get-Date -Format 'yyyyMMddHHmmss')") + $Global:ARTIFACT_TMP_FILE = Join-Path $CLI_TMP_ROOT $CLI_ARTIFACT + + Register-ObjectEvent -InputObject $webClient -EventName DownloadFileCompleted ` + -Action { + $Global:isDownLoaded = $True + $timer.Stop() + $webClient.Dispose() + } -SourceIdentifier "DownloadFileCompleted" | Out-Null + + Register-ObjectEvent -InputObject $webClient -EventName DownloadProgressChanged ` + -SourceIdentifier "DownloadProgressChanged" -Action { + $Global:Data = $event + } | Out-Null + + $Global:isDownLoaded = $False + $timer = New-Object System.Timers.Timer + $timer.Interval = 500 + + Register-ObjectEvent -InputObject $timer -EventName Elapsed -SourceIdentifier "TimerElapsed" -Action { + $precent = $Global:Data.SourceArgs.ProgressPercentage + $totalBytes = $Global:Data.SourceArgs.TotalBytesToReceive + $receivedBytes = $Global:Data.SourceArgs.BytesReceived + if ($precent -ne $null) { + $downloadProgress = [Math]::Round(($receivedBytes / $totalBytes) * 100, 2) + $status = "Downloaded {0} of {1} bytes" -f $receivedBytes, $totalBytes + Write-Progress -Activity "Downloading kbcli..." -Status $status -PercentComplete $downloadProgress + } + } | Out-Null + + try { + $webClient.DownloadFileAsync($DOWNLOAD_URL, $Global:ARTIFACT_TMP_FILE) + $timer.Start() + } + catch { + Write-Host "Download Failed" + exit 1; + } + + while (-not $Global:isDownLoaded) { + + } + Unregister-Event -SourceIdentifier "DownloadFileCompleted" + Unregister-Event -SourceIdentifier "DownloadProgressChanged" + Unregister-Event -SourceIdentifier "TimerElapsed" + Write-Host "Download Completed" + return $CLI_TMP_ROOT +} + +function installFile { + $DIR_NAME = "kbcli-windows-amd64" + $kbcliexe = "kbcli.exe" + $installPath = Join-Path "C:\Program Files" $DIR_NAME + + if (!(Test-Path -Path $installPath -PathType Container)) { + New-Item -ItemType Directory -Path $installPath | Out-Null + } + + $tmp_root_kbcli = Join-Path $CLI_TMP_ROOT "windows-amd64" #Must match the folder name with workflow + $tmp_root_kbcli = Join-Path $tmp_root_kbcli $kbcliexe + + Expand-Archive -Path "$Global:ARTIFACT_TMP_FILE" -DestinationPath $CLI_TMP_ROOT + + if ($? -ne $True -or !(Test-Path $tmp_root_kbcli -PathType Leaf) ) { + throw "Failed to unpack kbcli executable." + } + + $envPath = [Environment]::GetEnvironmentVariable("Path", "User") # add to PATH + if ($envPath -notlike "*$installPath*") { + [Environment]::SetEnvironmentVariable("Path", "$envPath;$installPath", "User") + Set-Item -Path Env:Path -Value $env:Path + } + + Copy-Item -Path $tmp_root_kbcli -Destination $installPath + if ( $? -eq $True -and (Test-Path (Join-Path $installPath $kbcliexe) -PathType Leaf) ) { + Write-Host "kbcli installed successfully." + Write-Host "" + Write-Host "Make sure your docker service is running and begin your journey with kbcli:`n" + Write-Host "`t$CLI_FILENAME playground init`n" + } else { + throw "Failed to install $CLI_FILENAME" + } +} + +function cleanup { + if (Test-Path $CLI_TMP_ROOT) { + Remove-Item $CLI_TMP_ROOT -Recurse -Force + } +} + +function installCompleted { + Write-Host "`nFor more information on how to get started, please visit:" + Write-Host "https://kubeblocks.io" + +} +# --------------------------------------- +# main +# --------------------------------------- + +verifySupported +checkExistingCLI +$COUNTRY_CODE = getCountryCode +$ret_val + +if (-not $args) { + Write-Host "Getting the latest kbcli ..." + $ret_val = getLatestRelease +} +elseif ($args[0] -match "^v.*$") { + $ret_val = $args[0] +} +else { + $ret_val = "v" + $args[0] +} + +$CLI_TMP_ROOT = downloadFile $ret_val +try { + installFile +} catch { + Write-Host "An error occurred: $($_.Exception.Message)" + Write-Host "Please try again in administrator mode!" + cleanup + exit 1 +} + +cleanup +installCompleted + + From d6064f8d5a71bd01f31b6c814f001096987ba499 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Tue, 11 Apr 2023 08:44:59 +0800 Subject: [PATCH 68/80] chore: adjust kbcli example-commands prompt (#2454) (#2463) --- docs/user_docs/cli/kbcli_cluster_configure.md | 6 +++--- .../cli/kbcli_cluster_describe-config.md | 8 ++++---- .../cli/kbcli_cluster_diff-config.md | 2 +- .../cli/kbcli_cluster_explain-config.md | 8 ++++---- .../cli/cmd/cluster/describe_configure.go | 20 +++++++++---------- internal/cli/cmd/cluster/operations.go | 6 +++--- 6 files changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/user_docs/cli/kbcli_cluster_configure.md b/docs/user_docs/cli/kbcli_cluster_configure.md index aba8a4d95..d9610c949 100644 --- a/docs/user_docs/cli/kbcli_cluster_configure.md +++ b/docs/user_docs/cli/kbcli_cluster_configure.md @@ -22,11 +22,11 @@ kbcli cluster configure [flags] ``` --component strings Specify the name of Component to be updated. If the cluster has only one component, unset the parameter. - --config-file string Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. - --config-spec string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'. + --config-file string Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. + --config-spec string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. -h, --help help for configure --name string OpsRequest name. if not specified, it will be randomly generated - --set strings Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-configure'. + --set strings Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-config'. --ttlSecondsAfterSucceed int Time to live after the OpsRequest succeed ``` diff --git a/docs/user_docs/cli/kbcli_cluster_describe-config.md b/docs/user_docs/cli/kbcli_cluster_describe-config.md index eb5d4e978..f7127fd60 100644 --- a/docs/user_docs/cli/kbcli_cluster_describe-config.md +++ b/docs/user_docs/cli/kbcli_cluster_describe-config.md @@ -12,16 +12,16 @@ kbcli cluster describe-config [flags] ``` # describe a cluster, e.g. cluster name is mycluster - kbcli cluster describe-configure mycluster + kbcli cluster describe-config mycluster # describe a component, e.g. cluster name is mycluster, component name is mysql - kbcli cluster describe-configure mycluster --component-name=mysql + kbcli cluster describe-config mycluster --component-name=mysql # describe all configuration files. - kbcli cluster describe-configure mycluster --component-name=mysql --show-detail + kbcli cluster describe-config mycluster --component-name=mysql --show-detail # describe a content of configuration file. - kbcli cluster describe-configure mycluster --component-name=mysql --config-file=my.cnf --show-detail + kbcli cluster describe-config mycluster --component-name=mysql --config-file=my.cnf --show-detail ``` ### Options diff --git a/docs/user_docs/cli/kbcli_cluster_diff-config.md b/docs/user_docs/cli/kbcli_cluster_diff-config.md index d77233562..efd0217fa 100644 --- a/docs/user_docs/cli/kbcli_cluster_diff-config.md +++ b/docs/user_docs/cli/kbcli_cluster_diff-config.md @@ -12,7 +12,7 @@ kbcli cluster diff-config [flags] ``` # compare config files - kbcli cluster diff-configure opsrequest1 opsrequest2 + kbcli cluster diff-config opsrequest1 opsrequest2 ``` ### Options diff --git a/docs/user_docs/cli/kbcli_cluster_explain-config.md b/docs/user_docs/cli/kbcli_cluster_explain-config.md index c678b02a1..569b4d2d8 100644 --- a/docs/user_docs/cli/kbcli_cluster_explain-config.md +++ b/docs/user_docs/cli/kbcli_cluster_explain-config.md @@ -12,16 +12,16 @@ kbcli cluster explain-config [flags] ``` # describe a cluster, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster + kbcli cluster explain-config mycluster # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false # describe a specified parameters, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode ``` ### Options diff --git a/internal/cli/cmd/cluster/describe_configure.go b/internal/cli/cmd/cluster/describe_configure.go index eaed3cc88..6fdd3647d 100644 --- a/internal/cli/cmd/cluster/describe_configure.go +++ b/internal/cli/cmd/cluster/describe_configure.go @@ -89,31 +89,31 @@ type parameterTemplate struct { var ( describeReconfigureExample = templates.Examples(` # describe a cluster, e.g. cluster name is mycluster - kbcli cluster describe-configure mycluster + kbcli cluster describe-config mycluster # describe a component, e.g. cluster name is mycluster, component name is mysql - kbcli cluster describe-configure mycluster --component-name=mysql + kbcli cluster describe-config mycluster --component-name=mysql # describe all configuration files. - kbcli cluster describe-configure mycluster --component-name=mysql --show-detail + kbcli cluster describe-config mycluster --component-name=mysql --show-detail # describe a content of configuration file. - kbcli cluster describe-configure mycluster --component-name=mysql --config-file=my.cnf --show-detail`) + kbcli cluster describe-config mycluster --component-name=mysql --config-file=my.cnf --show-detail`) explainReconfigureExample = templates.Examples(` # describe a cluster, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster + kbcli cluster explain-config mycluster # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl # describe a specified configure template, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl --trunc-document=false --trunc-enum=false # describe a specified parameters, e.g. cluster name is mycluster - kbcli cluster explain-configure mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode`) + kbcli cluster explain-config mycluster --component-name=mysql --config-specs=mysql-3node-tpl --param=sql_mode`) diffConfigureExample = templates.Examples(` # compare config files - kbcli cluster diff-configure opsrequest1 opsrequest2`) + kbcli cluster diff-config opsrequest1 opsrequest2`) ) func (r *reconfigureOptions) addCommonFlags(cmd *cobra.Command) { @@ -565,7 +565,7 @@ func (o *opsRequestDiffOptions) run() error { baseConfigs[tplName] = baseObj } - printer.PrintTitle("DIFF-CONFIGURE RESULT") + printer.PrintTitle("DIFF-CONFIG RESULT") for tplName, diff := range configDiffs { configObjects := baseConfigs[tplName] for _, params := range diff { diff --git a/internal/cli/cmd/cluster/operations.go b/internal/cli/cmd/cluster/operations.go index 0c7b6a65c..8e163ce20 100644 --- a/internal/cli/cmd/cluster/operations.go +++ b/internal/cli/cmd/cluster/operations.go @@ -690,10 +690,10 @@ func NewReconfigureCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) * inputs.Example = createReconfigureExample inputs.BuildFlags = func(cmd *cobra.Command) { o.buildCommonFlags(cmd) - cmd.Flags().StringSliceVar(&o.Parameters, "set", nil, "Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-configure'.") + cmd.Flags().StringSliceVar(&o.Parameters, "set", nil, "Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-config'.") cmd.Flags().StringSliceVar(&o.ComponentNames, "component", nil, "Specify the name of Component to be updated. If the cluster has only one component, unset the parameter.") - cmd.Flags().StringVar(&o.CfgTemplateName, "config-spec", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") - cmd.Flags().StringVar(&o.CfgFile, "config-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-configure'.") + cmd.Flags().StringVar(&o.CfgTemplateName, "config-spec", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") + cmd.Flags().StringVar(&o.CfgFile, "config-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") } inputs.Complete = o.fillTemplateArgForReconfiguring return create.BuildCommand(inputs) From 3105d52bbc2cc76753eb8372eb45338b331295a1 Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Tue, 11 Apr 2023 10:02:49 +0800 Subject: [PATCH 69/80] feat: add edit-config sub command for kbcli (#2412) (#2473) --- docs/user_docs/cli/cli.md | 1 + docs/user_docs/cli/kbcli_cluster.md | 1 + docs/user_docs/cli/kbcli_cluster_configure.md | 2 +- .../cli/kbcli_cluster_edit-config.md | 63 ++++ go.mod | 2 +- internal/cli/cmd/cluster/cluster.go | 1 + .../{describe_configure.go => config.go} | 2 +- internal/cli/cmd/cluster/config_edit.go | 187 +++++++++++ internal/cli/cmd/cluster/config_ops.go | 213 +++++++++++++ internal/cli/cmd/cluster/config_ops_test.go | 131 ++++++++ internal/cli/cmd/cluster/config_util.go | 135 ++++++++ .../{fake_ops_util.go => config_util_test.go} | 18 +- internal/cli/cmd/cluster/config_wrapper.go | 253 +++++++++++++++ internal/cli/cmd/cluster/errors.go | 26 +- internal/cli/cmd/cluster/operations.go | 298 +----------------- internal/cli/cmd/cluster/operations_test.go | 56 ---- internal/cli/util/util.go | 64 ++-- 17 files changed, 1060 insertions(+), 393 deletions(-) create mode 100644 docs/user_docs/cli/kbcli_cluster_edit-config.md rename internal/cli/cmd/cluster/{describe_configure.go => config.go} (99%) create mode 100644 internal/cli/cmd/cluster/config_edit.go create mode 100644 internal/cli/cmd/cluster/config_ops.go create mode 100644 internal/cli/cmd/cluster/config_ops_test.go create mode 100644 internal/cli/cmd/cluster/config_util.go rename internal/cli/cmd/cluster/{fake_ops_util.go => config_util_test.go} (84%) create mode 100644 internal/cli/cmd/cluster/config_wrapper.go diff --git a/docs/user_docs/cli/cli.md b/docs/user_docs/cli/cli.md index 9ee6411f1..cbbf95f9e 100644 --- a/docs/user_docs/cli/cli.md +++ b/docs/user_docs/cli/cli.md @@ -64,6 +64,7 @@ Cluster command. * [kbcli cluster describe-config](kbcli_cluster_describe-config.md) - Show details of a specific reconfiguring. * [kbcli cluster describe-ops](kbcli_cluster_describe-ops.md) - Show details of a specific OpsRequest. * [kbcli cluster diff-config](kbcli_cluster_diff-config.md) - Show the difference in parameters between the two submitted OpsRequest. +* [kbcli cluster edit-config](kbcli_cluster_edit-config.md) - Edit the config file of the component. * [kbcli cluster explain-config](kbcli_cluster_explain-config.md) - List the constraint for supported configuration params. * [kbcli cluster expose](kbcli_cluster_expose.md) - Expose a cluster. * [kbcli cluster grant-role](kbcli_cluster_grant-role.md) - Grant role to account diff --git a/docs/user_docs/cli/kbcli_cluster.md b/docs/user_docs/cli/kbcli_cluster.md index 06a885001..359bc280f 100644 --- a/docs/user_docs/cli/kbcli_cluster.md +++ b/docs/user_docs/cli/kbcli_cluster.md @@ -52,6 +52,7 @@ Cluster command. * [kbcli cluster describe-config](kbcli_cluster_describe-config.md) - Show details of a specific reconfiguring. * [kbcli cluster describe-ops](kbcli_cluster_describe-ops.md) - Show details of a specific OpsRequest. * [kbcli cluster diff-config](kbcli_cluster_diff-config.md) - Show the difference in parameters between the two submitted OpsRequest. +* [kbcli cluster edit-config](kbcli_cluster_edit-config.md) - Edit the config file of the component. * [kbcli cluster explain-config](kbcli_cluster_explain-config.md) - List the constraint for supported configuration params. * [kbcli cluster expose](kbcli_cluster_expose.md) - Expose a cluster. * [kbcli cluster grant-role](kbcli_cluster_grant-role.md) - Grant role to account diff --git a/docs/user_docs/cli/kbcli_cluster_configure.md b/docs/user_docs/cli/kbcli_cluster_configure.md index d9610c949..009e132d1 100644 --- a/docs/user_docs/cli/kbcli_cluster_configure.md +++ b/docs/user_docs/cli/kbcli_cluster_configure.md @@ -21,7 +21,7 @@ kbcli cluster configure [flags] ### Options ``` - --component strings Specify the name of Component to be updated. If the cluster has only one component, unset the parameter. + --component string Specify the name of Component to be updated. If the cluster has only one component, unset the parameter. --config-file string Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. --config-spec string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. -h, --help help for configure diff --git a/docs/user_docs/cli/kbcli_cluster_edit-config.md b/docs/user_docs/cli/kbcli_cluster_edit-config.md new file mode 100644 index 000000000..201cc04d6 --- /dev/null +++ b/docs/user_docs/cli/kbcli_cluster_edit-config.md @@ -0,0 +1,63 @@ +--- +title: kbcli cluster edit-config +--- + +Edit the config file of the component. + +``` +kbcli cluster edit-config [flags] +``` + +### Examples + +``` + # edit config for component + kbcli cluster edit-config [--component=] [--config-spec=] [--config-file=] + + # update mysql max_connections, cluster name is mycluster + kbcli cluster edit-config mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf +``` + +### Options + +``` + --component string Specify the name of Component to be updated. If the cluster has only one component, unset the parameter. + --config-file string Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. + --config-spec string Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'. + -h, --help help for edit-config + --name string OpsRequest name. if not specified, it will be randomly generated + --replace Specify whether to replace the config file. Default to false. + --set strings Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-config'. + --ttlSecondsAfterSucceed int Time to live after the OpsRequest succeed +``` + +### Options inherited from parent commands + +``` + --as string Username to impersonate for the operation. User could be a regular user or a service account in a namespace. + --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. + --as-uid string UID to impersonate for the operation. + --cache-dir string Default cache directory (default "$HOME/.kube/cache") + --certificate-authority string Path to a cert file for the certificate authority + --client-certificate string Path to a client certificate file for TLS + --client-key string Path to a client key file for TLS + --cluster string The name of the kubeconfig cluster to use + --context string The name of the kubeconfig context to use + --disable-compression If true, opt-out of response compression for all requests to the server + --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure + --kubeconfig string Path to the kubeconfig file to use for CLI requests. + --match-server-version Require server version to match client version + -n, --namespace string If present, the namespace scope for this CLI request + --request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0") + -s, --server string The address and port of the Kubernetes API server + --tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used + --token string Bearer token for authentication to the API server + --user string The name of the kubeconfig user to use +``` + +### SEE ALSO + +* [kbcli cluster](kbcli_cluster.md) - Cluster command. + +#### Go Back to [CLI Overview](cli.md) Homepage. + diff --git a/go.mod b/go.mod index d66c5b558..e5774b932 100644 --- a/go.mod +++ b/go.mod @@ -49,6 +49,7 @@ require ( github.com/opencontainers/image-spec v1.1.0-rc2 github.com/pingcap/go-tpc v1.0.9 github.com/pkg/errors v0.9.1 + github.com/pmezard/go-difflib v1.0.0 github.com/redis/go-redis/v9 v9.0.1 github.com/replicatedhq/termui/v3 v3.1.1-0.20200811145416-f40076d26851 github.com/replicatedhq/troubleshoot v0.57.0 @@ -275,7 +276,6 @@ require ( github.com/pelletier/go-toml/v2 v2.0.6 // indirect github.com/peterbourgon/diskv v2.0.1+incompatible // indirect github.com/pjbgf/sha1cd v0.2.3 // indirect - github.com/pmezard/go-difflib v1.0.0 // indirect github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect github.com/prometheus/client_golang v1.14.0 // indirect github.com/prometheus/client_model v0.3.0 // indirect diff --git a/internal/cli/cmd/cluster/cluster.go b/internal/cli/cmd/cluster/cluster.go index 25aa6e030..857fdc4eb 100644 --- a/internal/cli/cmd/cluster/cluster.go +++ b/internal/cli/cmd/cluster/cluster.go @@ -68,6 +68,7 @@ func NewClusterCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobr NewListOpsCmd(f, streams), NewDeleteOpsCmd(f, streams), NewReconfigureCmd(f, streams), + NewEditConfigureCmd(f, streams), NewExposeCmd(f, streams), NewDescribeReconfigureCmd(f, streams), NewExplainReconfigureCmd(f, streams), diff --git a/internal/cli/cmd/cluster/describe_configure.go b/internal/cli/cmd/cluster/config.go similarity index 99% rename from internal/cli/cmd/cluster/describe_configure.go rename to internal/cli/cmd/cluster/config.go index 6fdd3647d..a919a12d8 100644 --- a/internal/cli/cmd/cluster/describe_configure.go +++ b/internal/cli/cmd/cluster/config.go @@ -217,7 +217,7 @@ func (r *reconfigureOptions) syncClusterComponent() error { return nil } - componentNames, err := util.GetComponentsFromClusterCR(client.ObjectKey{ + componentNames, err := util.GetComponentsFromClusterName(client.ObjectKey{ Namespace: r.namespace, Name: r.clusterName, }, r.dynamic) diff --git a/internal/cli/cmd/cluster/config_edit.go b/internal/cli/cmd/cluster/config_edit.go new file mode 100644 index 000000000..073aea666 --- /dev/null +++ b/internal/cli/cmd/cluster/config_edit.go @@ -0,0 +1,187 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "fmt" + "strings" + + "github.com/spf13/cobra" + "golang.org/x/exp/slices" + "k8s.io/cli-runtime/pkg/genericclioptions" + cmdutil "k8s.io/kubectl/pkg/cmd/util" + "k8s.io/kubectl/pkg/cmd/util/editor" + "k8s.io/kubectl/pkg/util/templates" + "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/printer" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/cli/util" + "github.com/apecloud/kubeblocks/internal/cli/util/prompt" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +) + +type editConfigOptions struct { + configOpsOptions + + // config file replace + replaceFile bool +} + +var editConfigExample = templates.Examples(` + # edit config for component + kbcli cluster edit-config [--component=] [--config-spec=] [--config-file=] + + # update mysql max_connections, cluster name is mycluster + kbcli cluster edit-config mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf + `) + +func (o *editConfigOptions) Run(fn func(info *cfgcore.ConfigPatchInfo, cc *appsv1alpha1.ConfigConstraintSpec) error) error { + wrapper := o.wrapper + cfgEditContext := newConfigContext(o.BaseOptions, o.Name, wrapper.ComponentName(), wrapper.ConfigSpecName(), wrapper.ConfigFile()) + if err := cfgEditContext.prepare(); err != nil { + return err + } + + editor := editor.NewDefaultEditor([]string{ + "KUBE_EDITOR", + "EDITOR", + }) + if err := cfgEditContext.editConfig(editor); err != nil { + return err + } + + diff, err := cfgEditContext.getUnifiedDiffString() + if err != nil { + return err + } + if diff == "" { + fmt.Println("Edit cancelled, no changes made.") + return nil + } + + displayDiffWithColor(o.IOStreams.Out, diff) + + oldVersion := map[string]string{ + o.CfgFile: cfgEditContext.getOriginal(), + } + newVersion := map[string]string{ + o.CfgFile: cfgEditContext.getEdited(), + } + + configSpec := wrapper.ConfigSpec() + configConstraintKey := client.ObjectKey{ + Namespace: "", + Name: configSpec.ConfigConstraintRef, + } + configConstraint := appsv1alpha1.ConfigConstraint{} + if err := util.GetResourceObjectFromGVR(types.ConfigConstraintGVR(), configConstraintKey, o.Dynamic, &configConstraint); err != nil { + return err + } + formatterConfig := configConstraint.Spec.FormatterConfig + if formatterConfig == nil { + return cfgcore.MakeError("config spec[%s] not support reconfigure!", wrapper.ConfigSpecName()) + } + configPatch, _, err := cfgcore.CreateConfigPatch(oldVersion, newVersion, formatterConfig.Format, configSpec.Keys, false) + if err != nil { + return err + } + if !configPatch.IsModify { + fmt.Println("No parameters changes made.") + return nil + } + + fmt.Fprintf(o.Out, "Config patch(updated parameters): \n%s\n\n", string(configPatch.UpdateConfig[o.CfgFile])) + + dynamicUpdated, err := cfgcore.IsUpdateDynamicParameters(&configConstraint.Spec, configPatch) + if err != nil { + return nil + } + + confirmPrompt := confirmApplyReconfigurePrompt + if !dynamicUpdated { + confirmPrompt = restartConfirmPrompt + } + yes, err := o.confirmReconfigure(confirmPrompt) + if err != nil { + return err + } + if !yes { + return nil + } + return fn(configPatch, &configConstraint.Spec) +} + +func (o *editConfigOptions) confirmReconfigure(promptStr string) (bool, error) { + const yesStr = "yes" + const noStr = "no" + + confirmStr := []string{yesStr, noStr} + printer.Warning(o.Out, promptStr) + input, err := prompt.NewPrompt("Please type [yes/No] to confirm:", + func(input string) error { + if !slices.Contains(confirmStr, strings.ToLower(input)) { + return fmt.Errorf("typed \"%s\" does not match \"%s\"", input, confirmStr) + } + return nil + }, o.In).Run() + if err != nil { + return false, err + } + return strings.ToLower(input) == yesStr, nil +} + +// NewEditConfigureCmd shows the difference between two configuration version. +func NewEditConfigureCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { + editOptions := &editConfigOptions{ + configOpsOptions: configOpsOptions{ + editMode: true, + OperationsOptions: newBaseOperationsOptions(streams, appsv1alpha1.ReconfiguringType, false), + }} + inputs := buildOperationsInputs(f, editOptions.OperationsOptions) + inputs.Use = "edit-config" + inputs.Short = "Edit the config file of the component." + inputs.Example = editConfigExample + inputs.BuildFlags = func(cmd *cobra.Command) { + editOptions.buildReconfigureCommonFlags(cmd) + cmd.Flags().BoolVar(&editOptions.replaceFile, "replace", false, "Specify whether to replace the config file. Default to false.") + } + inputs.Complete = editOptions.Complete + inputs.Validate = editOptions.Validate + + cmd := &cobra.Command{ + Use: inputs.Use, + Short: inputs.Short, + Example: inputs.Example, + Run: func(cmd *cobra.Command, args []string) { + util.CheckErr(inputs.BaseOptionsObj.Complete(inputs, args)) + util.CheckErr(inputs.BaseOptionsObj.Validate(inputs)) + util.CheckErr(editOptions.Run(func(info *cfgcore.ConfigPatchInfo, cc *appsv1alpha1.ConfigConstraintSpec) error { + // generate patch for config + formatterConfig := cc.FormatterConfig + params := cfgcore.GenerateVisualizedParamsList(info, formatterConfig, nil) + editOptions.KeyValues = fromKeyValuesToMap(params, editOptions.CfgFile) + return inputs.BaseOptionsObj.Run(inputs) + })) + }, + } + if inputs.BuildFlags != nil { + inputs.BuildFlags(cmd) + } + return cmd +} diff --git a/internal/cli/cmd/cluster/config_ops.go b/internal/cli/cmd/cluster/config_ops.go new file mode 100644 index 000000000..ff48df16c --- /dev/null +++ b/internal/cli/cmd/cluster/config_ops.go @@ -0,0 +1,213 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "fmt" + "strings" + + "github.com/spf13/cobra" + "k8s.io/cli-runtime/pkg/genericclioptions" + cmdutil "k8s.io/kubectl/pkg/cmd/util" + "k8s.io/kubectl/pkg/util/templates" + "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/create" + "github.com/apecloud/kubeblocks/internal/cli/printer" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/cli/util" + "github.com/apecloud/kubeblocks/internal/cli/util/prompt" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +) + +type configOpsOptions struct { + *OperationsOptions + + editMode bool + wrapper *configWrapper + + // Reconfiguring options + ComponentName string + URLPath string `json:"urlPath"` + Parameters []string `json:"parameters"` +} + +var ( + createReconfigureExample = templates.Examples(` + # update component params + kbcli cluster configure --component= --config-spec= --config-file= --set max_connections=1000,general_log=OFF + + # update mysql max_connections, cluster name is mycluster + kbcli cluster configure mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf --set max_connections=2000 + `) +) + +func (o *configOpsOptions) Complete() error { + if o.Name == "" { + return makeMissingClusterNameErr() + } + + if !o.editMode { + kvs, err := o.parseUpdatedParams() + if err != nil { + return err + } + o.KeyValues = kvs + } + + wrapper, err := newConfigWrapper(o.BaseOptions, o.Name, o.ComponentName, o.CfgTemplateName, o.CfgFile, o.KeyValues) + if err != nil { + return err + } + + o.wrapper = wrapper + return wrapper.AutoFillRequiredParam() +} + +// Validate command flags or args is legal +func (o *configOpsOptions) Validate() error { + if err := o.wrapper.ValidateRequiredParam(); err != nil { + return err + } + + o.CfgFile = o.wrapper.ConfigFile() + o.CfgTemplateName = o.wrapper.ConfigSpecName() + o.ComponentNames = []string{o.wrapper.ComponentName()} + + if o.editMode { + return nil + } + if err := o.validateConfigParams(o.wrapper.ConfigSpec()); err != nil { + return err + } + o.printConfigureTips() + return nil +} + +func (o *configOpsOptions) validateConfigParams(tpl *appsv1alpha1.ComponentConfigSpec) error { + configConstraintKey := client.ObjectKey{ + Namespace: "", + Name: tpl.ConfigConstraintRef, + } + configConstraint := appsv1alpha1.ConfigConstraint{} + if err := util.GetResourceObjectFromGVR(types.ConfigConstraintGVR(), configConstraintKey, o.Dynamic, &configConstraint); err != nil { + return err + } + + newConfigData, err := cfgcore.MergeAndValidateConfigs(configConstraint.Spec, map[string]string{o.CfgFile: ""}, tpl.Keys, []cfgcore.ParamPairs{{ + Key: o.CfgFile, + UpdatedParams: cfgcore.FromStringMap(o.KeyValues), + }}) + if err != nil { + return err + } + return o.checkChangedParamsAndDoubleConfirm(&configConstraint.Spec, newConfigData, tpl) +} + +func (o *configOpsOptions) checkChangedParamsAndDoubleConfirm(cc *appsv1alpha1.ConfigConstraintSpec, data map[string]string, tpl *appsv1alpha1.ComponentConfigSpec) error { + mockEmptyData := func(m map[string]string) map[string]string { + r := make(map[string]string, len(data)) + for key := range m { + r[key] = "" + } + return r + } + + configPatch, _, err := cfgcore.CreateConfigPatch(mockEmptyData(data), data, cc.FormatterConfig.Format, tpl.Keys, false) + if err != nil { + return err + } + + dynamicUpdated, err := cfgcore.IsUpdateDynamicParameters(cc, configPatch) + if err != nil { + return nil + } + if dynamicUpdated { + return nil + } + return o.confirmReconfigureWithRestart() +} + +func (o *configOpsOptions) confirmReconfigureWithRestart() error { + const confirmStr = "yes" + printer.Warning(o.Out, restartConfirmPrompt) + _, err := prompt.NewPrompt(fmt.Sprintf("Please type \"%s\" to confirm:", confirmStr), + func(input string) error { + if input != confirmStr { + return fmt.Errorf("typed \"%s\" does not match \"%s\"", input, confirmStr) + } + return nil + }, o.In).Run() + return err +} + +func (o *configOpsOptions) parseUpdatedParams() (map[string]string, error) { + if len(o.Parameters) == 0 && len(o.URLPath) == 0 { + return nil, cfgcore.MakeError(missingUpdatedParametersErrMessage) + } + + keyValues := make(map[string]string) + for _, param := range o.Parameters { + pp := strings.Split(param, ",") + for _, p := range pp { + fields := strings.SplitN(p, "=", 2) + if len(fields) != 2 { + return nil, cfgcore.MakeError("updated parameter format: key=value") + } + keyValues[fields[0]] = fields[1] + } + } + return keyValues, nil +} + +func (o *configOpsOptions) printConfigureTips() { + fmt.Println("Will updated configure file meta:") + printer.PrintLineWithTabSeparator( + printer.NewPair(" ConfigSpec", printer.BoldYellow(o.CfgTemplateName)), + printer.NewPair(" ConfigFile", printer.BoldYellow(o.CfgFile)), + printer.NewPair("ComponentName", o.ComponentName), + printer.NewPair("ClusterName", o.Name)) +} + +// buildCommonFlags build common flags for operations command +func (o *configOpsOptions) buildReconfigureCommonFlags(cmd *cobra.Command) { + o.buildCommonFlags(cmd) + cmd.Flags().StringSliceVar(&o.Parameters, "set", nil, "Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-config'.") + cmd.Flags().StringVar(&o.ComponentName, "component", "", "Specify the name of Component to be updated. If the cluster has only one component, unset the parameter.") + cmd.Flags().StringVar(&o.CfgTemplateName, "config-spec", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") + cmd.Flags().StringVar(&o.CfgFile, "config-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") +} + +// NewReconfigureCmd creates a Reconfiguring command +func NewReconfigureCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { + o := &configOpsOptions{ + editMode: false, + OperationsOptions: newBaseOperationsOptions(streams, appsv1alpha1.ReconfiguringType, false), + } + inputs := buildOperationsInputs(f, o.OperationsOptions) + inputs.Use = "configure" + inputs.Short = "Reconfigure parameters with the specified components in the cluster." + inputs.Example = createReconfigureExample + inputs.BuildFlags = func(cmd *cobra.Command) { + o.buildReconfigureCommonFlags(cmd) + } + + inputs.Complete = o.Complete + inputs.Validate = o.Validate + return create.BuildCommand(inputs) +} diff --git a/internal/cli/cmd/cluster/config_ops_test.go b/internal/cli/cmd/cluster/config_ops_test.go new file mode 100644 index 000000000..236fa865e --- /dev/null +++ b/internal/cli/cmd/cluster/config_ops_test.go @@ -0,0 +1,131 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "bytes" + "io" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/cli-runtime/pkg/genericclioptions" + clientfake "k8s.io/client-go/rest/fake" + cmdtesting "k8s.io/kubectl/pkg/cmd/testing" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/testing" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" + testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" +) + +var _ = Describe("reconfigure test", func() { + const ( + clusterName = "cluster-ops" + clusterName1 = "cluster-ops1" + ) + var ( + streams genericclioptions.IOStreams + tf *cmdtesting.TestFactory + in *bytes.Buffer + ) + + BeforeEach(func() { + streams, in, _, _ = genericclioptions.NewTestIOStreams() + tf = cmdtesting.NewTestFactory().WithNamespace(testing.Namespace) + clusterWithTwoComps := testing.FakeCluster(clusterName, testing.Namespace) + clusterWithOneComp := clusterWithTwoComps.DeepCopy() + clusterWithOneComp.Name = clusterName1 + clusterWithOneComp.Spec.ComponentSpecs = []appsv1alpha1.ClusterComponentSpec{ + clusterWithOneComp.Spec.ComponentSpecs[0], + } + tf.FakeDynamicClient = testing.FakeDynamicClient(testing.FakeClusterDef(), + testing.FakeClusterVersion(), clusterWithTwoComps, clusterWithOneComp) + tf.Client = &clientfake.RESTClient{} + }) + + AfterEach(func() { + tf.Cleanup() + }) + + It("check params for reconfiguring operations", func() { + const ( + ns = "default" + clusterDefName = "test-clusterdef" + clusterVersionName = "test-clusterversion" + clusterName = "test-cluster" + statefulCompType = "replicasets" + statefulCompName = "mysql" + configSpecName = "mysql-config-tpl" + configVolumeName = "mysql-config" + ) + + By("Create configmap and config constraint obj") + configmap := testapps.NewCustomizedObj("resources/mysql-config-template.yaml", &corev1.ConfigMap{}, testapps.WithNamespace(ns)) + constraint := testapps.NewCustomizedObj("resources/mysql-config-constraint.yaml", + &appsv1alpha1.ConfigConstraint{}) + componentConfig := testapps.NewConfigMap(ns, cfgcore.GetComponentCfgName(clusterName, statefulCompName, configSpecName), testapps.SetConfigMapData("my.cnf", "")) + By("Create a clusterDefinition obj") + clusterDefObj := testapps.NewClusterDefFactory(clusterDefName). + AddComponent(testapps.StatefulMySQLComponent, statefulCompType). + AddConfigTemplate(configSpecName, configmap.Name, constraint.Name, ns, configVolumeName). + GetObject() + By("Create a clusterVersion obj") + clusterVersionObj := testapps.NewClusterVersionFactory(clusterVersionName, clusterDefObj.GetName()). + AddComponent(statefulCompType). + GetObject() + By("creating a cluster") + clusterObj := testapps.NewClusterFactory(ns, clusterName, + clusterDefObj.Name, ""). + AddComponent(statefulCompName, statefulCompType).GetObject() + + objs := []runtime.Object{configmap, constraint, clusterDefObj, clusterVersionObj, clusterObj, componentConfig} + ttf, ops := NewFakeOperationsOptions(ns, clusterObj.Name, appsv1alpha1.ReconfiguringType, objs...) + o := &configOpsOptions{ + // nil cannot be set to a map struct in CueLang, so init the map of KeyValues. + OperationsOptions: &OperationsOptions{ + BaseOptions: *ops, + }, + } + o.KeyValues = make(map[string]string) + defer ttf.Cleanup() + + By("validate reconfiguring parameter") + o.ComponentNames = []string{statefulCompName} + _, err := o.parseUpdatedParams() + Expect(err.Error()).To(ContainSubstring(missingUpdatedParametersErrMessage)) + o.Parameters = []string{"abcd"} + + _, err = o.parseUpdatedParams() + Expect(err.Error()).To(ContainSubstring("updated parameter format")) + o.Parameters = []string{"abcd=test"} + o.CfgTemplateName = configSpecName + o.IOStreams = streams + in.Write([]byte(o.Name + "\n")) + + Expect(o.Complete()).Should(Succeed()) + + in := &bytes.Buffer{} + in.Write([]byte("yes\n")) + + o.BaseOptions.In = io.NopCloser(in) + Expect(o.Validate()).Should(Succeed()) + }) + +}) diff --git a/internal/cli/cmd/cluster/config_util.go b/internal/cli/cmd/cluster/config_util.go new file mode 100644 index 000000000..8d03756f7 --- /dev/null +++ b/internal/cli/cmd/cluster/config_util.go @@ -0,0 +1,135 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + "bytes" + "fmt" + "io" + "path/filepath" + "strings" + + "github.com/fatih/color" + "github.com/pmezard/go-difflib/difflib" + corev1 "k8s.io/api/core/v1" + "k8s.io/kubectl/pkg/cmd/util/editor" + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/apecloud/kubeblocks/internal/cli/create" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/cli/util" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +) + +type configEditContext struct { + create.BaseOptions + + clusterName string + componentName string + configSpecName string + configKey string + + original string + edited string +} + +func (c *configEditContext) getOriginal() string { + return c.original +} + +func (c *configEditContext) getEdited() string { + return c.edited +} + +func (c *configEditContext) prepare() error { + cmObj := corev1.ConfigMap{} + cmKey := client.ObjectKey{ + Name: cfgcore.GetComponentCfgName(c.clusterName, c.componentName, c.configSpecName), + Namespace: c.Namespace, + } + if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), cmKey, c.Dynamic, &cmObj); err != nil { + return err + } + + val, ok := cmObj.Data[c.configKey] + if !ok { + return makeNotFoundConfigFileErr(c.configKey, c.configSpecName, cfgcore.ToSet(cmObj.Data).AsSlice()) + } + + c.original = val + return nil +} + +func (c *configEditContext) editConfig(editor editor.Editor) error { + edited, _, err := editor.LaunchTempFile(fmt.Sprintf("%s-edit-", filepath.Base(c.configKey)), "", bytes.NewBufferString(c.original)) + if err != nil { + return err + } + + c.edited = string(edited) + return nil +} + +func (c *configEditContext) getUnifiedDiffString() (string, error) { + diff := difflib.UnifiedDiff{ + A: difflib.SplitLines(c.original), + B: difflib.SplitLines(c.edited), + FromFile: "Original", + ToFile: "Current", + Context: 3, + } + return difflib.GetUnifiedDiffString(diff) +} + +func newConfigContext(baseOptions create.BaseOptions, clusterName, componentName, configSpec, file string) *configEditContext { + return &configEditContext{ + BaseOptions: baseOptions, + clusterName: clusterName, + componentName: componentName, + configSpecName: configSpec, + configKey: file, + } +} + +func displayDiffWithColor(out io.Writer, diffText string) { + for _, line := range difflib.SplitLines(diffText) { + switch { + case strings.HasPrefix(line, "---"), strings.HasPrefix(line, "+++"): + line = color.HiYellowString(line) + case strings.HasPrefix(line, "@@"): + line = color.HiBlueString(line) + case strings.HasPrefix(line, "-"): + line = color.RedString(line) + case strings.HasPrefix(line, "+"): + line = color.GreenString(line) + } + fmt.Fprint(out, line) + } +} + +func fromKeyValuesToMap(params []cfgcore.VisualizedParam, file string) map[string]string { + result := make(map[string]string) + for _, param := range params { + if param.Key != file { + continue + } + for _, kv := range param.Parameters { + result[kv.Key] = kv.Value + } + } + return result +} diff --git a/internal/cli/cmd/cluster/fake_ops_util.go b/internal/cli/cmd/cluster/config_util_test.go similarity index 84% rename from internal/cli/cmd/cluster/fake_ops_util.go rename to internal/cli/cmd/cluster/config_util_test.go index f52c9cc69..0dfe2d71a 100644 --- a/internal/cli/cmd/cluster/fake_ops_util.go +++ b/internal/cli/cmd/cluster/config_util_test.go @@ -29,17 +29,13 @@ import ( "github.com/apecloud/kubeblocks/internal/cli/types" ) -func NewFakeOperationsOptions(ns, cName string, opsType appsv1alpha1.OpsType, objs ...runtime.Object) (*cmdtesting.TestFactory, *OperationsOptions) { +func NewFakeOperationsOptions(ns, cName string, opsType appsv1alpha1.OpsType, objs ...runtime.Object) (*cmdtesting.TestFactory, *create.BaseOptions) { streams, _, _, _ := genericclioptions.NewTestIOStreams() tf := cmdtesting.NewTestFactory().WithNamespace(ns) - o := &OperationsOptions{ - BaseOptions: create.BaseOptions{ - IOStreams: streams, - Name: cName, - Namespace: ns, - }, - TTLSecondsAfterSucceed: 30, - OpsType: opsType, + baseOptions := &create.BaseOptions{ + IOStreams: streams, + Name: cName, + Namespace: ns, } err := appsv1alpha1.AddToScheme(scheme.Scheme) @@ -57,6 +53,6 @@ func NewFakeOperationsOptions(ns, cName string, opsType appsv1alpha1.OpsType, ob types.RestoreJobGVR(): types.KindRestoreJob + "List", types.OpsGVR(): types.KindOps + "List", } - o.Dynamic = dynamicfakeclient.NewSimpleDynamicClientWithCustomListKinds(scheme.Scheme, listMapping, objs...) - return tf, o + baseOptions.Dynamic = dynamicfakeclient.NewSimpleDynamicClientWithCustomListKinds(scheme.Scheme, listMapping, objs...) + return tf, baseOptions } diff --git a/internal/cli/cmd/cluster/config_wrapper.go b/internal/cli/cmd/cluster/config_wrapper.go new file mode 100644 index 000000000..9543683c8 --- /dev/null +++ b/internal/cli/cmd/cluster/config_wrapper.go @@ -0,0 +1,253 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cluster + +import ( + corev1 "k8s.io/api/core/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + "github.com/apecloud/kubeblocks/internal/cli/cluster" + "github.com/apecloud/kubeblocks/internal/cli/create" + "github.com/apecloud/kubeblocks/internal/cli/types" + "github.com/apecloud/kubeblocks/internal/cli/util" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +) + +type configWrapper struct { + create.BaseOptions + + clusterName string + updatedParams map[string]string + + // auto fill field + componentName string + configSpecName string + configKey string + + configSpec appsv1alpha1.ComponentConfigSpec + + clusterObj *appsv1alpha1.Cluster + clusterDefObj *appsv1alpha1.ClusterDefinition + clusterVerObj *appsv1alpha1.ClusterVersion +} + +func (w *configWrapper) ConfigSpec() *appsv1alpha1.ComponentConfigSpec { + return &w.configSpec +} + +func (w *configWrapper) ConfigSpecName() string { + return w.configSpecName +} + +func (w *configWrapper) ComponentName() string { + return w.componentName +} + +func (w *configWrapper) ConfigFile() string { + return w.configKey +} + +// AutoFillRequiredParam auto fill required param. +func (w *configWrapper) AutoFillRequiredParam() error { + if err := w.fillComponent(); err != nil { + return err + } + if err := w.fillConfigSpec(); err != nil { + return err + } + return w.fillConfigFile() +} + +// ValidateRequiredParam validate required param. +func (w *configWrapper) ValidateRequiredParam() error { + // step1: validate component exist. + if w.clusterObj.Spec.GetComponentByName(w.componentName) == nil { + return makeComponentNotExistErr(w.clusterName, w.componentName) + } + + // step2: validate configmap exist. + cmObj := corev1.ConfigMap{} + cmKey := client.ObjectKey{ + Name: cfgcore.GetComponentCfgName(w.clusterName, w.componentName, w.configSpecName), + Namespace: w.Namespace, + } + if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), cmKey, w.Dynamic, &cmObj); err != nil { + return err + } + + // step3: validate fileKey exist. + if _, ok := cmObj.Data[w.configKey]; !ok { + return makeNotFoundConfigFileErr(w.configKey, w.configSpecName, cfgcore.ToSet(cmObj.Data).AsSlice()) + } + + // TODO support all config file update. + if !cfgcore.CheckConfigTemplateReconfigureKey(w.configSpec, w.configKey) { + return makeNotSupportConfigFileUpdateErr(w.configKey, w.configSpec) + } + return nil +} + +func (w *configWrapper) fillComponent() error { + if w.componentName != "" { + return nil + } + componentNames, err := util.GetComponentsFromResource(w.clusterObj.Spec.ComponentSpecs, w.clusterDefObj) + if err != nil { + return err + } + if len(componentNames) != 1 { + return cfgcore.MakeError(multiComponentsErrorMessage) + } + w.componentName = componentNames[0] + return nil +} + +func (w *configWrapper) fillConfigSpec() error { + foundConfigSpec := func(configSpecs []appsv1alpha1.ComponentConfigSpec, name string) *appsv1alpha1.ComponentConfigSpec { + for _, configSpec := range configSpecs { + if configSpec.Name == name { + w.configSpec = configSpec + return &configSpec + } + } + return nil + } + + var vComponents []appsv1alpha1.ClusterComponentVersion + var cComponents = w.clusterObj.Spec.ComponentSpecs + var dComponents = w.clusterDefObj.Spec.ComponentDefs + + if w.clusterVerObj != nil { + vComponents = w.clusterVerObj.Spec.ComponentVersions + } + + configSpecs, err := util.GetConfigTemplateListWithResource(cComponents, dComponents, vComponents, w.componentName, true) + if err != nil { + return err + } + if len(configSpecs) == 0 { + return makeNotFoundTemplateErr(w.clusterName, w.componentName) + } + + if w.configSpecName != "" { + if foundConfigSpec(configSpecs, w.configSpecName) == nil { + return makeConfigSpecNotExistErr(w.clusterName, w.componentName, w.configSpecName) + } + return nil + } + + w.configSpec = configSpecs[0] + if len(configSpecs) == 1 { + w.configSpecName = configSpecs[0].Name + return nil + } + + supportUpdatedTpl := make([]appsv1alpha1.ComponentConfigSpec, 0) + for _, configSpec := range configSpecs { + if ok, err := util.IsSupportReconfigureParams(configSpec, w.updatedParams, w.Dynamic); err == nil && ok { + supportUpdatedTpl = append(supportUpdatedTpl, configSpec) + } + } + if len(supportUpdatedTpl) == 1 { + w.configSpec = configSpecs[0] + w.configSpecName = supportUpdatedTpl[0].Name + return nil + } + return cfgcore.MakeError(multiConfigTemplateErrorMessage) +} + +func (w *configWrapper) fillConfigFile() error { + if w.configKey != "" { + return nil + } + + if w.configSpec.TemplateRef == "" { + return makeNotFoundTemplateErr(w.clusterName, w.componentName) + } + + cmObj := corev1.ConfigMap{} + cmKey := client.ObjectKey{ + Name: cfgcore.GetComponentCfgName(w.clusterName, w.componentName, w.configSpecName), + Namespace: w.Namespace, + } + if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), cmKey, w.Dynamic, &cmObj); err != nil { + return err + } + if len(cmObj.Data) == 0 { + return cfgcore.MakeError("not support reconfiguring because there is no config file.") + } + + keys := w.filterForReconfiguring(cmObj.Data) + if len(keys) == 1 { + w.configKey = keys[0] + return nil + } + return cfgcore.MakeError(multiConfigFileErrorMessage) +} + +func (w *configWrapper) filterForReconfiguring(data map[string]string) []string { + keys := make([]string, 0, len(data)) + for k := range data { + if cfgcore.CheckConfigTemplateReconfigureKey(w.configSpec, k) { + keys = append(keys, k) + } + } + return keys +} + +func newConfigWrapper(baseOptions create.BaseOptions, clusterName, componentName, configSpec, configKey string, params map[string]string) (*configWrapper, error) { + var ( + err error + clusterObj *appsv1alpha1.Cluster + clusterDefObj *appsv1alpha1.ClusterDefinition + ) + + if clusterObj, err = cluster.GetClusterByName(baseOptions.Dynamic, clusterName, baseOptions.Namespace); err != nil { + return nil, err + } + if clusterDefObj, err = cluster.GetClusterDefByName(baseOptions.Dynamic, clusterObj.Spec.ClusterDefRef); err != nil { + return nil, err + } + + w := &configWrapper{ + BaseOptions: baseOptions, + clusterObj: clusterObj, + clusterDefObj: clusterDefObj, + clusterName: clusterName, + + componentName: componentName, + configSpecName: configSpec, + configKey: configKey, + updatedParams: params, + } + + if w.clusterObj.Spec.ClusterVersionRef == "" { + return w, err + } + + clusterVerObj := &appsv1alpha1.ClusterVersion{} + if err := util.GetResourceObjectFromGVR(types.ClusterVersionGVR(), client.ObjectKey{ + Namespace: "", + Name: w.clusterObj.Spec.ClusterVersionRef, + }, w.Dynamic, clusterVerObj); err != nil { + return nil, err + } + + w.clusterVerObj = clusterVerObj + return w, nil +} diff --git a/internal/cli/cmd/cluster/errors.go b/internal/cli/cmd/cluster/errors.go index 23fdf7466..2e782da39 100644 --- a/internal/cli/cmd/cluster/errors.go +++ b/internal/cli/cmd/cluster/errors.go @@ -16,7 +16,10 @@ limitations under the License. package cluster -import cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +import ( + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" + cfgcore "github.com/apecloud/kubeblocks/internal/configuration" +) var ( clusterNotExistErrMessage = "cluster[name=%s] is not exist. Please check that is spelled correctly." @@ -26,12 +29,19 @@ var ( multiComponentsErrorMessage = "when multi component exist, must specify which component to use. Please using --component-name" multiConfigTemplateErrorMessage = "when multi config template exist, must specify which config template to use. Please using --config-spec" + multiConfigFileErrorMessage = "when multi config files exist, must specify which config file to update. Please using --config-file" notFoundValidConfigTemplateErrorMessage = "not find valid config template, component[name=%s] in the cluster[name=%s]" + notFoundConfigSpecErrorMessage = "not find config spec[%s], component[name=%s] in the cluster[name=%s]" + + notFoundConfigFileErrorMessage = "not find config file, file[name=%s] in the configspec[name=%s], all configfiles: %v" + notSupportFileUpdateErrorMessage = "not support file[%s] update, current support files: %v" + notCueSchemaPrompt = "The config template not define cue schema and parameter explain info cannot be generated." cue2openAPISchemaFailedPrompt = "The cue schema may not satisfy the conversion constraints of openAPISchema and parameter explain info cannot be generated." - restartConfirmPrompt = "This configure will restart the process, which may cause the cluster to be unavailable for a period of time." + restartConfirmPrompt = "The parameter change you modified needs to be restarted, which may cause the cluster to be unavailable for a period of time. Do you need to continue...\n, " + confirmApplyReconfigurePrompt = "Are you sure you want to apply these changes?\n" ) func makeClusterNotExistErr(clusterName string) error { @@ -42,10 +52,22 @@ func makeComponentNotExistErr(clusterName, component string) error { return cfgcore.MakeError(componentNotExistErrMessage, clusterName, component) } +func makeConfigSpecNotExistErr(clusterName, component, configSpec string) error { + return cfgcore.MakeError(notFoundConfigSpecErrorMessage, configSpec, component, clusterName) +} + func makeNotFoundTemplateErr(clusterName, component string) error { return cfgcore.MakeError(notFoundValidConfigTemplateErrorMessage, clusterName, component) } +func makeNotFoundConfigFileErr(configFile, configSpec string, all []string) error { + return cfgcore.MakeError(notFoundConfigFileErrorMessage, configFile, configSpec, all) +} + +func makeNotSupportConfigFileUpdateErr(configFile string, configSpec appsv1alpha1.ComponentConfigSpec) error { + return cfgcore.MakeError(notSupportFileUpdateErrorMessage, configFile, configSpec.Keys) +} + func makeMissingClusterNameErr() error { return cfgcore.MakeError(missingClusterArgErrMassage) } diff --git a/internal/cli/cmd/cluster/operations.go b/internal/cli/cmd/cluster/operations.go index 8e163ce20..86305f56d 100644 --- a/internal/cli/cmd/cluster/operations.go +++ b/internal/cli/cmd/cluster/operations.go @@ -29,7 +29,6 @@ import ( "k8s.io/cli-runtime/pkg/genericclioptions" cmdutil "k8s.io/kubectl/pkg/cmd/util" "k8s.io/kubectl/pkg/util/templates" - "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" "github.com/apecloud/kubeblocks/internal/cli/cluster" @@ -38,8 +37,6 @@ import ( "github.com/apecloud/kubeblocks/internal/cli/printer" "github.com/apecloud/kubeblocks/internal/cli/types" "github.com/apecloud/kubeblocks/internal/cli/util" - "github.com/apecloud/kubeblocks/internal/cli/util/prompt" - cfgcore "github.com/apecloud/kubeblocks/internal/configuration" ) type OperationsOptions struct { @@ -69,8 +66,6 @@ type OperationsOptions struct { Replicas int `json:"replicas"` // Reconfiguring options - URLPath string `json:"urlPath"` - Parameters []string `json:"parameters"` KeyValues map[string]string `json:"keyValues"` CfgTemplateName string `json:"cfgTemplateName"` CfgFile string `json:"cfgFile"` @@ -88,25 +83,15 @@ type OperationsOptions struct { func newBaseOperationsOptions(streams genericclioptions.IOStreams, opsType appsv1alpha1.OpsType, hasComponentNamesFlag bool) *OperationsOptions { return &OperationsOptions{ - BaseOptions: create.BaseOptions{IOStreams: streams}, - OpsType: opsType, // nil cannot be set to a map struct in CueLang, so init the map of KeyValues. KeyValues: map[string]string{}, + BaseOptions: create.BaseOptions{IOStreams: streams}, + OpsType: opsType, HasComponentNamesFlag: hasComponentNamesFlag, RequireConfirm: true, } } -var ( - createReconfigureExample = templates.Examples(` - # update component params - kbcli cluster configure --component= --config-spec= --config-file= --set max_connections=1000,general_log=OFF - - # update mysql max_connections, cluster name is mycluster - kbcli cluster configure mycluster --component=mysql --config-spec=mysql-3node-tpl --config-file=my.cnf --set max_connections=2000 - `) -) - // buildCommonFlags build common flags for operations command func (o *OperationsOptions) buildCommonFlags(cmd *cobra.Command) { cmd.Flags().StringVar(&o.OpsRequestName, "name", "", "OpsRequest name. if not specified, it will be randomly generated ") @@ -172,160 +157,6 @@ func (o *OperationsOptions) validateVolumeExpansion() error { return nil } -func (o *OperationsOptions) validateReconfiguring() error { - if len(o.ComponentNames) != 1 { - return cfgcore.MakeError("reconfiguring only support one component.") - } - componentName := o.ComponentNames[0] - if err := o.existClusterAndComponent(componentName); err != nil { - return err - } - - tplList, err := util.GetConfigTemplateList(o.Name, o.Namespace, o.Dynamic, componentName, true) - if err != nil { - return err - } - tpl, err := o.validateTemplateParam(tplList) - if err != nil { - return err - } - if err := o.validateConfigMapKey(tpl, componentName); err != nil { - return err - } - if err := o.validateConfigParams(tpl); err != nil { - return err - } - o.printConfigureTips() - return nil -} - -func (o *OperationsOptions) validateConfigParams(tpl *appsv1alpha1.ComponentConfigSpec) error { - configConstraintKey := client.ObjectKey{ - Namespace: "", - Name: tpl.ConfigConstraintRef, - } - configConstraint := appsv1alpha1.ConfigConstraint{} - if err := util.GetResourceObjectFromGVR(types.ConfigConstraintGVR(), configConstraintKey, o.Dynamic, &configConstraint); err != nil { - return err - } - - newConfigData, err := cfgcore.MergeAndValidateConfigs(configConstraint.Spec, map[string]string{o.CfgFile: ""}, tpl.Keys, []cfgcore.ParamPairs{{ - Key: o.CfgFile, - UpdatedParams: cfgcore.FromStringMap(o.KeyValues), - }}) - if err != nil { - return err - } - return o.checkChangedParamsAndDoubleConfirm(&configConstraint.Spec, newConfigData, tpl) -} - -func (o *OperationsOptions) checkChangedParamsAndDoubleConfirm(cc *appsv1alpha1.ConfigConstraintSpec, data map[string]string, tpl *appsv1alpha1.ComponentConfigSpec) error { - mockEmptyData := func(m map[string]string) map[string]string { - r := make(map[string]string, len(data)) - for key := range m { - r[key] = "" - } - return r - } - - configPatch, _, err := cfgcore.CreateConfigPatch(mockEmptyData(data), data, cc.FormatterConfig.Format, tpl.Keys, false) - if err != nil { - return err - } - - dynamicUpdated, err := cfgcore.IsUpdateDynamicParameters(cc, configPatch) - if err != nil { - return nil - } - if dynamicUpdated { - return nil - } - return o.confirmReconfigureWithRestart() -} - -func (o *OperationsOptions) confirmReconfigureWithRestart() error { - const confirmStr = "yes" - printer.Warning(o.Out, restartConfirmPrompt) - _, err := prompt.NewPrompt(fmt.Sprintf("Please type \"%s\" to confirm:", confirmStr), - func(input string) error { - if input != confirmStr { - return fmt.Errorf("typed \"%s\" does not match \"%s\"", input, confirmStr) - } - return nil - }, o.In).Run() - return err -} - -func (o *OperationsOptions) validateTemplateParam(tpls []appsv1alpha1.ComponentConfigSpec) (*appsv1alpha1.ComponentConfigSpec, error) { - if len(tpls) == 0 { - return nil, cfgcore.MakeError("not support reconfiguring because there is no config template.") - } - - if len(o.CfgTemplateName) == 0 && len(tpls) > 1 { - return nil, cfgcore.MakeError("when multi templates exist, must specify which template to use.") - } - - // Autofill Config template name. - if len(o.CfgTemplateName) == 0 && len(tpls) == 1 { - tpl := &tpls[0] - o.CfgTemplateName = tpl.Name - return tpl, nil - } - - for i := range tpls { - tpl := &tpls[i] - if tpl.Name == o.CfgTemplateName { - return tpl, nil - } - } - return nil, cfgcore.MakeError("specify template name[%s] is not exist.", o.CfgTemplateName) -} - -func (o *OperationsOptions) validateConfigMapKey(tpl *appsv1alpha1.ComponentConfigSpec, componentName string) error { - var ( - cmObj = corev1.ConfigMap{} - cmName = cfgcore.GetComponentCfgName(o.Name, componentName, tpl.Name) - ) - - if err := util.GetResourceObjectFromGVR(types.ConfigmapGVR(), client.ObjectKey{ - Name: cmName, - Namespace: o.Namespace, - }, o.Dynamic, &cmObj); err != nil { - return err - } - if len(cmObj.Data) == 0 { - return cfgcore.MakeError("not support reconfiguring because there is no config file.") - } - - // Autofill ConfigMap key - if o.CfgFile == "" && len(cmObj.Data) > 0 { - o.fillKeyForReconfiguring(tpl, cmObj.Data) - } - if _, ok := cmObj.Data[o.CfgFile]; !ok { - return cfgcore.MakeError("specify file name[%s] is not exist.", o.CfgFile) - } - return nil -} - -func (o *OperationsOptions) parseUpdatedParams() error { - if len(o.Parameters) == 0 && len(o.URLPath) == 0 { - return cfgcore.MakeError("reconfiguring required configure file or updated parameters.") - } - - o.KeyValues = make(map[string]string) - for _, param := range o.Parameters { - pp := strings.Split(param, ",") - for _, p := range pp { - fields := strings.SplitN(p, "=", 2) - if len(fields) != 2 { - return cfgcore.MakeError("updated parameter format: key=value") - } - o.KeyValues[fields[0]] = fields[1] - } - } - return nil -} - // Validate command flags or args is legal func (o *OperationsOptions) Validate() error { if o.Name == "" { @@ -338,11 +169,6 @@ func (o *OperationsOptions) Validate() error { return err } - // not require confirm for reconfigure - if o.OpsType == appsv1alpha1.ReconfiguringType { - return o.validateReconfiguring() - } - // common validate for componentOps if o.HasComponentNamesFlag && len(o.ComponentNames) == 0 { return fmt.Errorf(`missing components, please specify the "--components" flag for multi-components cluster`) @@ -364,108 +190,6 @@ func (o *OperationsOptions) Validate() error { return nil } -func (o *OperationsOptions) fillTemplateArgForReconfiguring() error { - if o.Name == "" { - return makeMissingClusterNameErr() - } - - if err := o.fillComponentNameForReconfiguring(); err != nil { - return err - } - - if err := o.parseUpdatedParams(); err != nil { - return err - } - if len(o.KeyValues) == 0 { - return cfgcore.MakeError(missingUpdatedParametersErrMessage) - } - - componentName := o.ComponentNames[0] - tplList, err := util.GetConfigTemplateList(o.Name, o.Namespace, o.Dynamic, componentName, true) - if err != nil { - return err - } - - if len(tplList) == 0 { - return makeNotFoundTemplateErr(o.Name, componentName) - } - - if len(tplList) == 1 { - o.CfgTemplateName = tplList[0].Name - return nil - } - - supportUpdatedTpl := make([]appsv1alpha1.ComponentConfigSpec, 0) - for _, tpl := range tplList { - if ok, err := util.IsSupportReconfigureParams(tpl, o.KeyValues, o.Dynamic); err == nil && ok { - supportUpdatedTpl = append(supportUpdatedTpl, tpl) - } - } - if len(supportUpdatedTpl) == 1 { - o.CfgTemplateName = supportUpdatedTpl[0].Name - return nil - } - - return cfgcore.MakeError(multiConfigTemplateErrorMessage) -} - -func (o *OperationsOptions) fillComponentNameForReconfiguring() error { - if len(o.ComponentNames) != 0 { - return nil - } - - componentNames, err := util.GetComponentsFromClusterCR(client.ObjectKey{ - Namespace: o.Namespace, - Name: o.Name, - }, o.Dynamic) - if err != nil { - return err - } - if len(componentNames) != 1 { - return cfgcore.MakeError(multiComponentsErrorMessage) - } - o.ComponentNames = componentNames - return nil -} - -func (o *OperationsOptions) existClusterAndComponent(componentName string) error { - clusterObj := appsv1alpha1.Cluster{} - if err := util.GetResourceObjectFromGVR(types.ClusterGVR(), client.ObjectKey{ - Namespace: o.Namespace, - Name: o.Name, - }, o.Dynamic, &clusterObj); err != nil { - return makeClusterNotExistErr(o.Name) - } - - for _, component := range clusterObj.Spec.ComponentSpecs { - if component.Name == componentName { - return nil - } - } - return makeComponentNotExistErr(o.Name, componentName) -} - -func (o *OperationsOptions) printConfigureTips() { - fmt.Println("Will updated configure file meta:") - printer.PrintLineWithTabSeparator( - printer.NewPair(" TemplateName", printer.BoldYellow(o.CfgTemplateName)), - printer.NewPair(" ConfigureFile", printer.BoldYellow(o.CfgFile)), - printer.NewPair("ComponentName", o.ComponentNames[0]), - printer.NewPair("ClusterName", o.Name)) -} - -func (o *OperationsOptions) fillKeyForReconfiguring(tpl *appsv1alpha1.ComponentConfigSpec, data map[string]string) { - keys := make([]string, 0, len(data)) - for k := range data { - if cfgcore.CheckConfigTemplateReconfigureKey(*tpl, k) { - keys = append(keys, k) - } - } - if len(keys) == 1 { - o.CfgFile = keys[0] - } -} - // buildOperationsInputs builds operations inputs func buildOperationsInputs(f cmdutil.Factory, o *OperationsOptions) create.Inputs { o.OpsTypeLower = strings.ToLower(string(o.OpsType)) @@ -681,24 +405,6 @@ func NewVolumeExpansionCmd(f cmdutil.Factory, streams genericclioptions.IOStream return create.BuildCommand(inputs) } -// NewReconfigureCmd creates a Reconfiguring command -func NewReconfigureCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { - o := newBaseOperationsOptions(streams, appsv1alpha1.ReconfiguringType, false) - inputs := buildOperationsInputs(f, o) - inputs.Use = "configure" - inputs.Short = "Reconfigure parameters with the specified components in the cluster." - inputs.Example = createReconfigureExample - inputs.BuildFlags = func(cmd *cobra.Command) { - o.buildCommonFlags(cmd) - cmd.Flags().StringSliceVar(&o.Parameters, "set", nil, "Specify updated parameter list. For details about the parameters, refer to kbcli sub command: 'kbcli cluster describe-config'.") - cmd.Flags().StringSliceVar(&o.ComponentNames, "component", nil, "Specify the name of Component to be updated. If the cluster has only one component, unset the parameter.") - cmd.Flags().StringVar(&o.CfgTemplateName, "config-spec", "", "Specify the name of the configuration template to be updated (e.g. for apecloud-mysql: --config-spec=mysql-3node-tpl). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") - cmd.Flags().StringVar(&o.CfgFile, "config-file", "", "Specify the name of the configuration file to be updated (e.g. for mysql: --config-file=my.cnf). What templates or configure files are available for this cluster can refer to kbcli sub command: 'kbcli cluster describe-config'.") - } - inputs.Complete = o.fillTemplateArgForReconfiguring - return create.BuildCommand(inputs) -} - var ( exposeExamples = templates.Examples(` # Expose a cluster to vpc diff --git a/internal/cli/cmd/cluster/operations_test.go b/internal/cli/cmd/cluster/operations_test.go index 2f489e2e1..eba9f5c00 100644 --- a/internal/cli/cmd/cluster/operations_test.go +++ b/internal/cli/cmd/cluster/operations_test.go @@ -21,8 +21,6 @@ import ( . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/runtime" "k8s.io/cli-runtime/pkg/genericclioptions" clientfake "k8s.io/client-go/rest/fake" cmdtesting "k8s.io/kubectl/pkg/cmd/testing" @@ -32,8 +30,6 @@ import ( "github.com/apecloud/kubeblocks/internal/cli/testing" "github.com/apecloud/kubeblocks/internal/cli/types" "github.com/apecloud/kubeblocks/internal/cli/util" - cfgcore "github.com/apecloud/kubeblocks/internal/configuration" - testapps "github.com/apecloud/kubeblocks/internal/testutil/apps" ) var _ = Describe("operations", func() { @@ -149,58 +145,6 @@ var _ = Describe("operations", func() { Expect(testing.ContainExpectStrings(capturedOutput, "kbcli cluster describe-ops")).Should(BeTrue()) }) - It("check params for reconfiguring operations", func() { - const ( - ns = "default" - clusterDefName = "test-clusterdef" - clusterVersionName = "test-clusterversion" - clusterName = "test-cluster" - statefulCompType = "replicasets" - statefulCompName = "mysql" - configSpecName = "mysql-config-tpl" - configVolumeName = "mysql-config" - ) - - By("Create configmap and config constraint obj") - configmap := testapps.NewCustomizedObj("resources/mysql-config-template.yaml", &corev1.ConfigMap{}, testapps.WithNamespace(ns)) - constraint := testapps.NewCustomizedObj("resources/mysql-config-constraint.yaml", - &appsv1alpha1.ConfigConstraint{}) - componentConfig := testapps.NewConfigMap(ns, cfgcore.GetComponentCfgName(clusterName, statefulCompName, configSpecName), testapps.SetConfigMapData("my.cnf", "")) - By("Create a clusterDefinition obj") - clusterDefObj := testapps.NewClusterDefFactory(clusterDefName). - AddComponent(testapps.StatefulMySQLComponent, statefulCompType). - AddConfigTemplate(configSpecName, configmap.Name, constraint.Name, ns, configVolumeName). - GetObject() - By("Create a clusterVersion obj") - clusterVersionObj := testapps.NewClusterVersionFactory(clusterVersionName, clusterDefObj.GetName()). - AddComponent(statefulCompType). - GetObject() - By("creating a cluster") - clusterObj := testapps.NewClusterFactory(ns, clusterName, - clusterDefObj.Name, ""). - AddComponent(statefulCompName, statefulCompType).GetObject() - - objs := []runtime.Object{configmap, constraint, clusterDefObj, clusterVersionObj, clusterObj, componentConfig} - ttf, o := NewFakeOperationsOptions(ns, clusterObj.Name, appsv1alpha1.ReconfiguringType, objs...) - defer ttf.Cleanup() - o.ComponentNames = []string{"replicasets", "proxy"} - By("validate reconfiguring when multi components") - Expect(o.Validate()).To(MatchError("reconfiguring only support one component.")) - - By("validate reconfiguring parameter") - o.ComponentNames = []string{statefulCompName} - Expect(o.parseUpdatedParams().Error()).To(ContainSubstring("reconfiguring required configure file or updated parameters")) - o.Parameters = []string{"abcd"} - - Expect(o.parseUpdatedParams().Error()).To(ContainSubstring("updated parameter format")) - o.Parameters = []string{"abcd=test"} - o.CfgTemplateName = configSpecName - o.IOStreams = streams - in.Write([]byte(o.Name + "\n")) - - Expect(o.Validate()).Should(Succeed()) - }) - It("list and delete operations", func() { clusterName := "wesql" args := []string{clusterName} diff --git a/internal/cli/util/util.go b/internal/cli/util/util.go index 89c72f61d..e11df119c 100644 --- a/internal/cli/util/util.go +++ b/internal/cli/util/util.go @@ -396,44 +396,53 @@ func GetConfigTemplateList(clusterName string, namespace string, cli dynamic.Int clusterVersionObj = appsv1alpha1.ClusterVersion{} ) - if err := GetResourceObjectFromGVR(types.ClusterGVR(), client.ObjectKey{ + clusterKey := client.ObjectKey{ Namespace: namespace, Name: clusterName, - }, cli, &clusterObj); err != nil { + } + if err := GetResourceObjectFromGVR(types.ClusterGVR(), clusterKey, cli, &clusterObj); err != nil { return nil, err } - - clusterDefName := clusterObj.Spec.ClusterDefRef - if err := GetResourceObjectFromGVR(types.ClusterDefGVR(), client.ObjectKey{ + clusterDefKey := client.ObjectKey{ Namespace: "", - Name: clusterDefName, - }, cli, &clusterDefObj); err != nil { + Name: clusterObj.Spec.ClusterDefRef, + } + if err := GetResourceObjectFromGVR(types.ClusterDefGVR(), clusterDefKey, cli, &clusterDefObj); err != nil { return nil, err } - clusterVersionName := clusterObj.Spec.ClusterVersionRef - if clusterVersionName != "" { - if err := GetResourceObjectFromGVR(types.ClusterVersionGVR(), client.ObjectKey{ - Namespace: "", - Name: clusterVersionName, - }, cli, &clusterVersionObj); err != nil { + clusterVerKey := client.ObjectKey{ + Namespace: "", + Name: clusterObj.Spec.ClusterVersionRef, + } + if clusterVerKey.Name != "" { + if err := GetResourceObjectFromGVR(types.ClusterVersionGVR(), clusterVerKey, cli, &clusterVersionObj); err != nil { return nil, err } } + return GetConfigTemplateListWithResource(clusterObj.Spec.ComponentSpecs, clusterDefObj.Spec.ComponentDefs, clusterVersionObj.Spec.ComponentVersions, componentName, reloadTpl) +} - tpls, err := cfgcore.GetConfigTemplatesFromComponent(clusterObj.Spec.ComponentSpecs, clusterDefObj.Spec.ComponentDefs, clusterVersionObj.Spec.ComponentVersions, componentName) +func GetConfigTemplateListWithResource(cComponents []appsv1alpha1.ClusterComponentSpec, + dComponents []appsv1alpha1.ClusterComponentDefinition, + vComponents []appsv1alpha1.ClusterComponentVersion, + componentName string, + reloadTpl bool) ([]appsv1alpha1.ComponentConfigSpec, error) { + + configSpecs, err := cfgcore.GetConfigTemplatesFromComponent(cComponents, dComponents, vComponents, componentName) if err != nil { return nil, err - } else if !reloadTpl { - return tpls, nil + } + if !reloadTpl { + return configSpecs, nil } - validTpls := make([]appsv1alpha1.ComponentConfigSpec, 0, len(tpls)) - for _, tpl := range tpls { - if len(tpl.ConfigConstraintRef) > 0 && len(tpl.TemplateRef) > 0 { - validTpls = append(validTpls, tpl) + validConfigSpecs := make([]appsv1alpha1.ComponentConfigSpec, 0, len(configSpecs)) + for _, configSpec := range configSpecs { + if configSpec.ConfigConstraintRef != "" && configSpec.TemplateRef != "" { + validConfigSpecs = append(validConfigSpecs, configSpec) } } - return validTpls, nil + return validConfigSpecs, nil } // GetResourceObjectFromGVR query the resource object using GVR. @@ -448,8 +457,8 @@ func GetResourceObjectFromGVR(gvr schema.GroupVersionResource, key client.Object return apiruntime.DefaultUnstructuredConverter.FromUnstructured(unstructuredObj.Object, k8sObj) } -// GetComponentsFromClusterCR returns name of component. -func GetComponentsFromClusterCR(key client.ObjectKey, cli dynamic.Interface) ([]string, error) { +// GetComponentsFromClusterName returns name of component. +func GetComponentsFromClusterName(key client.ObjectKey, cli dynamic.Interface) ([]string, error) { clusterObj := appsv1alpha1.Cluster{} clusterDefObj := appsv1alpha1.ClusterDefinition{} if err := GetResourceObjectFromGVR(types.ClusterGVR(), key, cli, &clusterObj); err != nil { @@ -463,8 +472,13 @@ func GetComponentsFromClusterCR(key client.ObjectKey, cli dynamic.Interface) ([] return nil, err } - componentNames := make([]string, 0, len(clusterObj.Spec.ComponentSpecs)) - for _, component := range clusterObj.Spec.ComponentSpecs { + return GetComponentsFromResource(clusterObj.Spec.ComponentSpecs, &clusterDefObj) +} + +// GetComponentsFromResource returns name of component. +func GetComponentsFromResource(componentSpecs []appsv1alpha1.ClusterComponentSpec, clusterDefObj *appsv1alpha1.ClusterDefinition) ([]string, error) { + componentNames := make([]string, 0, len(componentSpecs)) + for _, component := range componentSpecs { cdComponent := clusterDefObj.GetComponentDefByName(component.ComponentDefRef) if enableReconfiguring(cdComponent) { componentNames = append(componentNames, component.Name) From d462c24ec9b99a24385da10fccb5280f9623004b Mon Sep 17 00:00:00 2001 From: xingran Date: Tue, 11 Apr 2023 10:34:46 +0800 Subject: [PATCH 70/80] chore: remove postgresql unsupport high-availablity and rename postgresql-patroni-ha to postgresql (#2497) --- Makefile | 2 - deploy/postgresql-cluster/Chart.yaml | 4 +- .../postgresql-cluster/templates/cluster.yaml | 4 +- deploy/postgresql-cluster/values.yaml | 3 + .../postgresql-patroni-ha-cluster/.helmignore | 23 - .../postgresql-patroni-ha-cluster/Chart.yaml | 9 - .../templates/NOTES.txt | 2 - .../templates/_helpers.tpl | 62 - .../templates/cluster.yaml | 50 - .../postgresql-patroni-ha-cluster/values.yaml | 36 - deploy/postgresql-patroni-ha/.helmignore | 23 - deploy/postgresql-patroni-ha/Chart.yaml | 21 - .../config/pg14-config-constraint.cue | 1071 ----------------- .../config/pg14-config-effect-scope.yaml | 22 - .../config/pg14-config.tpl | 123 -- .../templates/_helpers.tpl | 98 -- .../templates/_tplvalues.tpl | 12 - .../templates/backuppolicytemplate.yaml | 15 - .../templates/backuptool.yaml | 57 - .../templates/clusterdefinition.yaml | 317 ----- .../templates/clusterversion.yaml | 18 - .../templates/configconstraint.yaml | 47 - .../templates/configmap.yaml | 26 - .../templates/metrics-configmap.yaml | 8 - .../templates/scripts.yaml | 69 -- deploy/postgresql-patroni-ha/values.yaml | 427 ------- deploy/postgresql/Chart.yaml | 4 +- .../config/pg14-config-effect-scope.yaml | 67 +- .../scripts/patroni-reload.tpl | 0 .../scripts/restart-parameter.yaml | 0 deploy/postgresql/templates/backuptool.yaml | 44 +- .../templates/clusterdefinition.yaml | 196 +-- .../postgresql/templates/clusterversion.yaml | 8 +- .../templates/configconstraint.yaml | 21 +- deploy/postgresql/templates/configmap.yaml | 10 +- .../templates/patroni-rbac.yaml | 0 .../templates/patroni-reload.yaml | 0 deploy/postgresql/templates/scripts.yaml | 127 +- deploy/postgresql/values.yaml | 12 +- 39 files changed, 248 insertions(+), 2790 deletions(-) delete mode 100644 deploy/postgresql-patroni-ha-cluster/.helmignore delete mode 100644 deploy/postgresql-patroni-ha-cluster/Chart.yaml delete mode 100644 deploy/postgresql-patroni-ha-cluster/templates/NOTES.txt delete mode 100644 deploy/postgresql-patroni-ha-cluster/templates/_helpers.tpl delete mode 100644 deploy/postgresql-patroni-ha-cluster/templates/cluster.yaml delete mode 100644 deploy/postgresql-patroni-ha-cluster/values.yaml delete mode 100644 deploy/postgresql-patroni-ha/.helmignore delete mode 100644 deploy/postgresql-patroni-ha/Chart.yaml delete mode 100644 deploy/postgresql-patroni-ha/config/pg14-config-constraint.cue delete mode 100644 deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml delete mode 100644 deploy/postgresql-patroni-ha/config/pg14-config.tpl delete mode 100644 deploy/postgresql-patroni-ha/templates/_helpers.tpl delete mode 100644 deploy/postgresql-patroni-ha/templates/_tplvalues.tpl delete mode 100644 deploy/postgresql-patroni-ha/templates/backuppolicytemplate.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/backuptool.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/clusterversion.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/configconstraint.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/configmap.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/metrics-configmap.yaml delete mode 100644 deploy/postgresql-patroni-ha/templates/scripts.yaml delete mode 100644 deploy/postgresql-patroni-ha/values.yaml rename deploy/{postgresql-patroni-ha => postgresql}/scripts/patroni-reload.tpl (100%) rename deploy/{postgresql-patroni-ha => postgresql}/scripts/restart-parameter.yaml (100%) rename deploy/{postgresql-patroni-ha => postgresql}/templates/patroni-rbac.yaml (100%) rename deploy/{postgresql-patroni-ha => postgresql}/templates/patroni-reload.yaml (100%) diff --git a/Makefile b/Makefile index b25afb41f..781fab822 100644 --- a/Makefile +++ b/Makefile @@ -385,8 +385,6 @@ bump-chart-ver: \ bump-single-chart-appver.nyancat \ bump-single-chart-ver.postgresql \ bump-single-chart-ver.postgresql-cluster \ - bump-single-chart-ver.postgresql-patroni-ha \ - bump-single-chart-ver.postgresql-patroni-ha-cluster \ bump-single-chart-ver.redis \ bump-single-chart-ver.redis-cluster \ bump-single-chart-ver.milvus \ diff --git a/deploy/postgresql-cluster/Chart.yaml b/deploy/postgresql-cluster/Chart.yaml index 84123966c..56c51fcc3 100644 --- a/deploy/postgresql-cluster/Chart.yaml +++ b/deploy/postgresql-cluster/Chart.yaml @@ -1,9 +1,9 @@ apiVersion: v2 name: pgcluster -description: A PostgreSQL cluster Helm chart for KubeBlocks. +description: A PostgreSQL (with Patroni HA) cluster Helm chart for KubeBlocks. type: application version: 0.5.0-alpha.3 -appVersion: "14.7.0" +appVersion: "15.2.0" diff --git a/deploy/postgresql-cluster/templates/cluster.yaml b/deploy/postgresql-cluster/templates/cluster.yaml index 56816780b..8740f5728 100644 --- a/deploy/postgresql-cluster/templates/cluster.yaml +++ b/deploy/postgresql-cluster/templates/cluster.yaml @@ -16,10 +16,12 @@ spec: {{- end }} componentSpecs: - name: postgresql # user-defined - componentDefRef: pg-replication # ref clusterdefinition components.name + componentDefRef: postgresql # ref clusterdefinition components.name monitor: {{ .Values.monitor.enabled | default false }} replicas: {{ .Values.replicaCount | default 2 }} primaryIndex: {{ .Values.primaryIndex | default 0 }} + switchPolicy: + type: {{ .Values.switchPolicy.type}} enabledLogs: {{ .Values.enabledLogs | toJson | indent 4 }} {{- with .Values.resources }} resources: diff --git a/deploy/postgresql-cluster/values.yaml b/deploy/postgresql-cluster/values.yaml index 23ab82c83..a318813a9 100644 --- a/deploy/postgresql-cluster/values.yaml +++ b/deploy/postgresql-cluster/values.yaml @@ -9,6 +9,9 @@ clusterVersionOverride: "" primaryIndex: 0 +switchPolicy: + type: Noop + monitor: enabled: false diff --git a/deploy/postgresql-patroni-ha-cluster/.helmignore b/deploy/postgresql-patroni-ha-cluster/.helmignore deleted file mode 100644 index 0e8a0eb36..000000000 --- a/deploy/postgresql-patroni-ha-cluster/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/deploy/postgresql-patroni-ha-cluster/Chart.yaml b/deploy/postgresql-patroni-ha-cluster/Chart.yaml deleted file mode 100644 index 56c51fcc3..000000000 --- a/deploy/postgresql-patroni-ha-cluster/Chart.yaml +++ /dev/null @@ -1,9 +0,0 @@ -apiVersion: v2 -name: pgcluster -description: A PostgreSQL (with Patroni HA) cluster Helm chart for KubeBlocks. - -type: application - -version: 0.5.0-alpha.3 - -appVersion: "15.2.0" diff --git a/deploy/postgresql-patroni-ha-cluster/templates/NOTES.txt b/deploy/postgresql-patroni-ha-cluster/templates/NOTES.txt deleted file mode 100644 index c3b3453e3..000000000 --- a/deploy/postgresql-patroni-ha-cluster/templates/NOTES.txt +++ /dev/null @@ -1,2 +0,0 @@ -1. Get the application URL by running these commands: - diff --git a/deploy/postgresql-patroni-ha-cluster/templates/_helpers.tpl b/deploy/postgresql-patroni-ha-cluster/templates/_helpers.tpl deleted file mode 100644 index 05a757a7e..000000000 --- a/deploy/postgresql-patroni-ha-cluster/templates/_helpers.tpl +++ /dev/null @@ -1,62 +0,0 @@ -{{/* -Expand the name of the chart. -*/}} -{{- define "postgresqlcluster.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "postgresqlcluster.fullname" -}} -{{- if .Values.fullnameOverride }} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- $name := default .Chart.Name .Values.nameOverride }} -{{- if contains $name .Release.Name }} -{{- .Release.Name | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} -{{- end }} -{{- end }} -{{- end }} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "postgresqlcluster.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Common labels -*/}} -{{- define "postgresqlcluster.labels" -}} -helm.sh/chart: {{ include "postgresqlcluster.chart" . }} -{{ include "postgresqlcluster.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "postgresqlcluster.selectorLabels" -}} -app.kubernetes.io/name: {{ include "postgresqlcluster.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - -{{/* -Create the name of the service account to use -*/}} -{{- define "postgresqlcluster.serviceAccountName" -}} -{{- if .Values.serviceAccount.create }} -{{- default (include "postgresqlcluster.fullname" .) .Values.serviceAccount.name }} -{{- else }} -{{- default "default" .Values.serviceAccount.name }} -{{- end }} -{{- end }} diff --git a/deploy/postgresql-patroni-ha-cluster/templates/cluster.yaml b/deploy/postgresql-patroni-ha-cluster/templates/cluster.yaml deleted file mode 100644 index d4eee14e9..000000000 --- a/deploy/postgresql-patroni-ha-cluster/templates/cluster.yaml +++ /dev/null @@ -1,50 +0,0 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 -kind: Cluster -metadata: - name: {{ .Release.Name }} - labels: {{ include "postgresqlcluster.labels" . | nindent 4 }} -spec: - clusterDefinitionRef: postgresql-ha # ref clusterdefinition.name - clusterVersionRef: postgresql-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} # ref clusterversion.name - terminationPolicy: {{ .Values.terminationPolicy }} - affinity: - {{- with .Values.topologyKeys }} - topologyKeys: {{ . | toYaml | nindent 6 }} - {{- end }} - {{- with $.Values.tolerations }} - tolerations: {{ . | toYaml | nindent 4 }} - {{- end }} - componentSpecs: - - name: postgresql # user-defined - componentDefRef: postgresql-ha # ref clusterdefinition components.name - monitor: {{ .Values.monitor.enabled | default false }} - replicas: {{ .Values.replicaCount | default 2 }} - primaryIndex: {{ .Values.primaryIndex | default 0 }} - switchPolicy: - type: {{ .Values.switchPolicy.type}} - enabledLogs: {{ .Values.enabledLogs | toJson | indent 4 }} - {{- with .Values.resources }} - resources: - {{- with .limits }} - limits: - cpu: {{ .cpu | quote }} - memory: {{ .memory | quote }} - {{- end }} - {{- with .requests }} - requests: - cpu: {{ .cpu | quote }} - memory: {{ .memory | quote }} - {{- end }} - {{- end }} - {{- if .Values.persistence.enabled }} - volumeClaimTemplates: - - name: data # ref clusterdefinition components.containers.volumeMounts.name - spec: - storageClassName: {{ .Values.persistence.data.storageClassName }} - accessModes: - - ReadWriteOnce - resources: - requests: - storage: {{ .Values.persistence.data.size }} - {{- end }} - diff --git a/deploy/postgresql-patroni-ha-cluster/values.yaml b/deploy/postgresql-patroni-ha-cluster/values.yaml deleted file mode 100644 index a318813a9..000000000 --- a/deploy/postgresql-patroni-ha-cluster/values.yaml +++ /dev/null @@ -1,36 +0,0 @@ -# Default values for wesqlcluster. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 2 -terminationPolicy: Delete - -clusterVersionOverride: "" - -primaryIndex: 0 - -switchPolicy: - type: Noop - -monitor: - enabled: false - -resources: { } - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - - # limits: - # cpu: 500m - # memory: 2Gi - # requests: - # cpu: 100m - # memory: 1Gi -persistence: - enabled: true - data: - storageClassName: - size: 1Gi -enabledLogs: - - running diff --git a/deploy/postgresql-patroni-ha/.helmignore b/deploy/postgresql-patroni-ha/.helmignore deleted file mode 100644 index 0e8a0eb36..000000000 --- a/deploy/postgresql-patroni-ha/.helmignore +++ /dev/null @@ -1,23 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*.orig -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/deploy/postgresql-patroni-ha/Chart.yaml b/deploy/postgresql-patroni-ha/Chart.yaml deleted file mode 100644 index 1cf6269e0..000000000 --- a/deploy/postgresql-patroni-ha/Chart.yaml +++ /dev/null @@ -1,21 +0,0 @@ -apiVersion: v2 -name: postgresql-ha -description: A PostgreSQL (with Patroni HA) cluster definition Helm chart for Kubernetes - -type: application - -version: 0.5.0-alpha.3 - -appVersion: "15.2.0" - -home: https://kubeblocks.io/ -icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png - - -maintainers: -- name: ApeCloud - url: https://github.com/apecloud/kubeblocks/ - - -sources: -- https://github.com/apecloud/kubeblocks/ diff --git a/deploy/postgresql-patroni-ha/config/pg14-config-constraint.cue b/deploy/postgresql-patroni-ha/config/pg14-config-constraint.cue deleted file mode 100644 index 107531fe5..000000000 --- a/deploy/postgresql-patroni-ha/config/pg14-config-constraint.cue +++ /dev/null @@ -1,1071 +0,0 @@ -// Copyright ApeCloud, Inc. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -// PostgreSQL parameters: https://postgresqlco.nf/doc/en/param/ -#PGParameter: { - // Sets the application name to be reported in statistics and logs. - application_name?: string - // Sets the shell command that will be called to archive a WAL file. - archive_command?: string - // (s) Forces a switch to the next xlog file if a new file has not been started within N seconds. - archive_timeout: int & >=0 & <=2147483647 | *300 @timeDurationResource(1s) - // Enable input of NULL elements in arrays. - array_nulls?: bool & false | true - // (s) Sets the maximum allowed time to complete client authentication. - authentication_timeout?: int & >=1 & <=600 @timeDurationResource() - // Use EXPLAIN ANALYZE for plan logging. - "auto_explain.log_analyze"?: bool & false | true - // Log buffers usage. - "auto_explain.log_buffers"?: bool & false | true - // EXPLAIN format to be used for plan logging. - "auto_explain.log_format"?: string & "text" | "xml" | "json" | "yaml" - - // (ms) Sets the minimum execution time above which plans will be logged. - "auto_explain.log_min_duration"?: int & >=-1 & <=2147483647 @timeDurationResource() - - // Log nested statements. - "auto_explain.log_nested_statements"?: bool & false | true - - // Collect timing data, not just row counts. - "auto_explain.log_timing"?: bool & false | true - - // Include trigger statistics in plans. - "auto_explain.log_triggers"?: bool & false | true - - // Use EXPLAIN VERBOSE for plan logging. - "auto_explain.log_verbose"?: bool & false | true - - // Fraction of queries to process. - "auto_explain.sample_rate"?: float & >=0 & <=1 - - // Starts the autovacuum subprocess. - autovacuum?: bool & false | true - - // Number of tuple inserts, updates or deletes prior to analyze as a fraction of reltuples. - autovacuum_analyze_scale_factor: float & >=0 & <=100 | *0.05 - - // Minimum number of tuple inserts, updates or deletes prior to analyze. - autovacuum_analyze_threshold?: int & >=0 & <=2147483647 - - // Age at which to autovacuum a table to prevent transaction ID wraparound. - autovacuum_freeze_max_age?: int & >=100000000 & <=750000000 - - // Sets the maximum number of simultaneously running autovacuum worker processes. - autovacuum_max_workers?: int & >=1 & <=8388607 - - // Multixact age at which to autovacuum a table to prevent multixact wraparound. - autovacuum_multixact_freeze_max_age?: int & >=10000000 & <=2000000000 - - // (s) Time to sleep between autovacuum runs. - autovacuum_naptime: int & >=1 & <=2147483 | *15 @timeDurationResource(1s) - - // (ms) Vacuum cost delay in milliseconds, for autovacuum. - autovacuum_vacuum_cost_delay?: int & >=-1 & <=100 - - // Vacuum cost amount available before napping, for autovacuum. - autovacuum_vacuum_cost_limit?: int & >=-1 & <=10000 - - // Number of tuple inserts prior to vacuum as a fraction of reltuples. - autovacuum_vacuum_insert_scale_factor?: float & >=0 & <=100 - - // Minimum number of tuple inserts prior to vacuum, or -1 to disable insert vacuums. - autovacuum_vacuum_insert_threshold?: int & >=-1 & <=2147483647 - - // Number of tuple updates or deletes prior to vacuum as a fraction of reltuples. - autovacuum_vacuum_scale_factor: float & >=0 & <=100 | *0.1 - - // Minimum number of tuple updates or deletes prior to vacuum. - autovacuum_vacuum_threshold?: int & >=0 & <=2147483647 - - // (kB) Sets the maximum memory to be used by each autovacuum worker process. - autovacuum_work_mem?: int & >=-1 & <=2147483647 @storeResource(1KB) - - // (8Kb) Number of pages after which previously performed writes are flushed to disk. - backend_flush_after?: int & >=0 & <=256 - - // Sets whether \ is allowed in string literals. - backslash_quote?: string & "safe_encoding" | "on" | "off" - - // Log backtrace for errors in these functions. - backtrace_functions?: string - - // (ms) Background writer sleep time between rounds. - bgwriter_delay?: int & >=10 & <=10000 @timeDurationResource() - - // (8Kb) Number of pages after which previously performed writes are flushed to disk. - bgwriter_flush_after?: int & >=0 & <=256 - - // Background writer maximum number of LRU pages to flush per round. - bgwriter_lru_maxpages?: int & >=0 & <=1000 - - // Multiple of the average buffer usage to free per round. - bgwriter_lru_multiplier?: float & >=0 & <=10 - - // Sets the output format for bytea. - bytea_output?: string & "escape" | "hex" - - // Check function bodies during CREATE FUNCTION. - check_function_bodies?: bool & false | true - - // Time spent flushing dirty buffers during checkpoint, as fraction of checkpoint interval. - checkpoint_completion_target: float & >=0 & <=1 | *0.9 - - // (8kB) Number of pages after which previously performed writes are flushed to disk. - checkpoint_flush_after?: int & >=0 & <=256 @storeResource(8KB) - - // (s) Sets the maximum time between automatic WAL checkpoints. - checkpoint_timeout?: int & >=30 & <=3600 @timeDurationResource(1s) - - // (s) Enables warnings if checkpoint segments are filled more frequently than this. - checkpoint_warning?: int & >=0 & <=2147483647 @timeDurationResource(1s) - - // time between checks for client disconnection while running queries - client_connection_check_interval?: int & >=0 & <=2147483647 @timeDurationResource() - - // Sets the clients character set encoding. - client_encoding?: string - - // Sets the message levels that are sent to the client. - client_min_messages?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "log" | "notice" | "warning" | "error" - - // Sets the delay in microseconds between transaction commit and flushing WAL to disk. - commit_delay?: int & >=0 & <=100000 - - // Sets the minimum concurrent open transactions before performing commit_delay. - commit_siblings?: int & >=0 & <=1000 - - // Enables in-core computation of a query identifier - compute_query_id?: string & "on" | "auto" - - // Sets the servers main configuration file. - config_file?: string - - // Enables the planner to use constraints to optimize queries. - constraint_exclusion?: string & "partition" | "on" | "off" - - // Sets the planners estimate of the cost of processing each index entry during an index scan. - cpu_index_tuple_cost?: float & >=0 & <=1.79769 - - // Sets the planners estimate of the cost of processing each operator or function call. - cpu_operator_cost?: float & >=0 & <=1.79769 - - // Sets the planners estimate of the cost of processing each tuple (row). - cpu_tuple_cost?: float & >=0 & <=1.79769 - - // Sets the database to store pg_cron metadata tables - "cron.database_name"?: string - - // Log all jobs runs into the job_run_details table - "cron.log_run"?: string & "on" | "off" - - // Log all cron statements prior to execution. - "cron.log_statement"?: string & "on" | "off" - - // Maximum number of jobs that can run concurrently. - "cron.max_running_jobs": int & >=0 & <=100 | *5 - - // Enables background workers for pg_cron - "cron.use_background_workers"?: string - - // Sets the planners estimate of the fraction of a cursors rows that will be retrieved. - cursor_tuple_fraction?: float & >=0 & <=1 - - // Sets the servers data directory. - data_directory?: string - - // Sets the display format for date and time values. - datestyle?: string - - // Enables per-database user names. - db_user_namespace?: bool & false | true - - // (ms) Sets the time to wait on a lock before checking for deadlock. - deadlock_timeout?: int & >=1 & <=2147483647 @timeDurationResource() - - // Indents parse and plan tree displays. - debug_pretty_print?: bool & false | true - - // Logs each querys parse tree. - debug_print_parse?: bool & false | true - - // Logs each querys execution plan. - debug_print_plan?: bool & false | true - - // Logs each querys rewritten parse tree. - debug_print_rewritten?: bool & false | true - - // Sets the default statistics target. - default_statistics_target?: int & >=1 & <=10000 - - // Sets the default tablespace to create tables and indexes in. - default_tablespace?: string - - // Sets the default TOAST compression method for columns of newly-created tables - default_toast_compression?: string & "pglz" | "lz4" - - // Sets the default deferrable status of new transactions. - default_transaction_deferrable?: bool & false | true - - // Sets the transaction isolation level of each new transaction. - default_transaction_isolation?: string & "serializable" | "repeatable read" | "read committed" | "read uncommitted" - - // Sets the default read-only status of new transactions. - default_transaction_read_only?: bool & false | true - - // (8kB) Sets the planners assumption about the size of the disk cache. - effective_cache_size?: int & >=1 & <=2147483647 @storeResource(8KB) - - // Number of simultaneous requests that can be handled efficiently by the disk subsystem. - effective_io_concurrency?: int & >=0 & <=1000 - - // Enables or disables the query planner's use of async-aware append plan types - enable_async_append?: bool & false | true - - // Enables the planners use of bitmap-scan plans. - enable_bitmapscan?: bool & false | true - - // Enables the planner's use of gather merge plans. - enable_gathermerge?: bool & false | true - - // Enables the planners use of hashed aggregation plans. - enable_hashagg?: bool & false | true - - // Enables the planners use of hash join plans. - enable_hashjoin?: bool & false | true - - // Enables the planner's use of incremental sort steps. - enable_incremental_sort?: bool & false | true - - // Enables the planner's use of index-only-scan plans. - enable_indexonlyscan?: bool & false | true - - // Enables the planners use of index-scan plans. - enable_indexscan?: bool & false | true - - // Enables the planners use of materialization. - enable_material?: bool & false | true - - // Enables the planner's use of memoization - enable_memoize?: bool & false | true - - // Enables the planners use of merge join plans. - enable_mergejoin?: bool & false | true - - // Enables the planners use of nested-loop join plans. - enable_nestloop?: bool & false | true - - // Enables the planner's use of parallel append plans. - enable_parallel_append?: bool & false | true - - // Enables the planner's user of parallel hash plans. - enable_parallel_hash?: bool & false | true - - // Enable plan-time and run-time partition pruning. - enable_partition_pruning?: bool & false | true - - // Enables partitionwise aggregation and grouping. - enable_partitionwise_aggregate?: bool & false | true - - // Enables partitionwise join. - enable_partitionwise_join?: bool & false | true - - // Enables the planners use of sequential-scan plans. - enable_seqscan?: bool & false | true - - // Enables the planners use of explicit sort steps. - enable_sort?: bool & false | true - - // Enables the planners use of TID scan plans. - enable_tidscan?: bool & false | true - - // Warn about backslash escapes in ordinary string literals. - escape_string_warning?: bool & false | true - - // Terminate session on any error. - exit_on_error?: bool & false | true - - // Sets the number of digits displayed for floating-point values. - extra_float_digits?: int & >=-15 & <=3 - - // Forces use of parallel query facilities. - force_parallel_mode?: bool & false | true - - // Sets the FROM-list size beyond which subqueries are not collapsed. - from_collapse_limit?: int & >=1 & <=2147483647 - - // Forces synchronization of updates to disk. - fsync: bool & false | true | *true - - // Writes full pages to WAL when first modified after a checkpoint. - full_page_writes: bool & false | true | *true - - // Enables genetic query optimization. - geqo?: bool & false | true - - // GEQO: effort is used to set the default for other GEQO parameters. - geqo_effort?: int & >=1 & <=10 - - // GEQO: number of iterations of the algorithm. - geqo_generations?: int & >=0 & <=2147483647 - - // GEQO: number of individuals in the population. - geqo_pool_size?: int & >=0 & <=2147483647 @storeResource() - - // GEQO: seed for random path selection. - geqo_seed?: float & >=0 & <=1 - - // GEQO: selective pressure within the population. - geqo_selection_bias?: float & >=1.5 & <=2 - - // Sets the threshold of FROM items beyond which GEQO is used. - geqo_threshold?: int & >=2 & <=2147483647 - - // Sets the maximum allowed result for exact search by GIN. - gin_fuzzy_search_limit?: int & >=0 & <=2147483647 - - // (kB) Sets the maximum size of the pending list for GIN index. - gin_pending_list_limit?: int & >=64 & <=2147483647 @storeResource(1KB) - - // Multiple of work_mem to use for hash tables. - hash_mem_multiplier?: float & >=1 & <=1000 - - // Sets the servers hba configuration file. - hba_file?: string - - // Force group aggregation for hll - "hll.force_groupagg"?: bool & false | true - - // Allows feedback from a hot standby to the primary that will avoid query conflicts. - hot_standby_feedback?: bool & false | true - - // Use of huge pages on Linux. - huge_pages?: string & "on" | "off" | "try" - - // Sets the servers ident configuration file. - ident_file?: string - - // (ms) Sets the maximum allowed duration of any idling transaction. - idle_in_transaction_session_timeout: int & >=0 & <=2147483647 | *86400000 @timeDurationResource() - - // Terminate any session that has been idle (that is, waiting for a client query), but not within an open transaction, for longer than the specified amount of time - idle_session_timeout?: int & >=0 & <=2147483647 @timeDurationResource() - - // Continues recovery after an invalid pages failure. - ignore_invalid_pages: bool & false | true | *false - - // Sets the display format for interval values. - intervalstyle?: string & "postgres" | "postgres_verbose" | "sql_standard" | "iso_8601" - - // Allow JIT compilation. - jit: bool & false | true | *false - - // Perform JIT compilation if query is more expensive. - jit_above_cost?: float & >=-1 & <=1.79769 - - // Perform JIT inlining if query is more expensive. - jit_inline_above_cost?: float & >=-1 & <=1.79769 - - // Optimize JITed functions if query is more expensive. - jit_optimize_above_cost?: float & >=-1 & <=1.79769 - - // Sets the FROM-list size beyond which JOIN constructs are not flattened. - join_collapse_limit?: int & >=1 & <=2147483647 - - // Sets the language in which messages are displayed. - lc_messages?: string - - // Sets the locale for formatting monetary amounts. - lc_monetary?: string - - // Sets the locale for formatting numbers. - lc_numeric?: string - - // Sets the locale for formatting date and time values. - lc_time?: string - - // Sets the host name or IP address(es) to listen to. - listen_addresses?: string - - // Enables backward compatibility mode for privilege checks on large objects. - lo_compat_privileges: bool & false | true | *false - - // (ms) Sets the minimum execution time above which autovacuum actions will be logged. - log_autovacuum_min_duration: int & >=-1 & <=2147483647 | *10000 @timeDurationResource() - - // Logs each checkpoint. - log_checkpoints: bool & false | true | *true - - // Logs each successful connection. - log_connections?: bool & false | true - - // Sets the destination for server log output. - log_destination?: string & "stderr" | "csvlog" - - // Sets the destination directory for log files. - log_directory?: string - - // Logs end of a session, including duration. - log_disconnections?: bool & false | true - - // Logs the duration of each completed SQL statement. - log_duration?: bool & false | true - - // Sets the verbosity of logged messages. - log_error_verbosity?: string & "terse" | "default" | "verbose" - - // Writes executor performance statistics to the server log. - log_executor_stats?: bool & false | true - - // Sets the file permissions for log files. - log_file_mode?: string - - // Sets the file name pattern for log files. - log_filename?: string - - // Start a subprocess to capture stderr output and/or csvlogs into log files. - logging_collector: bool & false | true | *true - - // Logs the host name in the connection logs. - log_hostname?: bool & false | true - - // (kB) Sets the maximum memory to be used for logical decoding. - logical_decoding_work_mem?: int & >=64 & <=2147483647 @storeResource(1KB) - - // Controls information prefixed to each log line. - log_line_prefix?: string - - // Logs long lock waits. - log_lock_waits?: bool & false | true - - // (ms) Sets the minimum execution time above which a sample of statements will be logged. Sampling is determined by log_statement_sample_rate. - log_min_duration_sample?: int & >=-1 & <=2147483647 @timeDurationResource() - - // (ms) Sets the minimum execution time above which statements will be logged. - log_min_duration_statement?: int & >=-1 & <=2147483647 @timeDurationResource() - - // Causes all statements generating error at or above this level to be logged. - log_min_error_statement?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "info" | "notice" | "warning" | "error" | "log" | "fatal" | "panic" - - // Sets the message levels that are logged. - log_min_messages?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "info" | "notice" | "warning" | "error" | "log" | "fatal" - - // When logging statements, limit logged parameter values to first N bytes. - log_parameter_max_length?: int & >=-1 & <=1073741823 - - // When reporting an error, limit logged parameter values to first N bytes. - log_parameter_max_length_on_error?: int & >=-1 & <=1073741823 - - // Writes parser performance statistics to the server log. - log_parser_stats?: bool & false | true - - // Writes planner performance statistics to the server log. - log_planner_stats?: bool & false | true - - // Controls whether a log message is produced when the startup process waits longer than deadlock_timeout for recovery conflicts - log_recovery_conflict_waits?: bool & false | true - - // Logs each replication command. - log_replication_commands?: bool & false | true - - // (min) Automatic log file rotation will occur after N minutes. - log_rotation_age: int & >=1 & <=1440 | *60 @timeDurationResource(1min) - - // (kB) Automatic log file rotation will occur after N kilobytes. - log_rotation_size?: int & >=0 & <=2097151 @storeResource(1KB) - - // Sets the type of statements logged. - log_statement?: string & "none" | "ddl" | "mod" | "all" - - // Fraction of statements exceeding log_min_duration_sample to be logged. - log_statement_sample_rate?: float & >=0 & <=1 - - // Writes cumulative performance statistics to the server log. - log_statement_stats?: bool & false | true - - // (kB) Log the use of temporary files larger than this number of kilobytes. - log_temp_files?: int & >=-1 & <=2147483647 @storeResource(1KB) - - // Sets the time zone to use in log messages. - log_timezone?: string - - // Set the fraction of transactions to log for new transactions. - log_transaction_sample_rate?: float & >=0 & <=1 - - // Truncate existing log files of same name during log rotation. - log_truncate_on_rotation: bool & false | true | *false - - // A variant of effective_io_concurrency that is used for maintenance work. - maintenance_io_concurrency?: int & >=0 & <=1000 - - // (kB) Sets the maximum memory to be used for maintenance operations. - maintenance_work_mem?: int & >=1024 & <=2147483647 @storeResource(1KB) - - // Sets the maximum number of concurrent connections. - max_connections?: int & >=6 & <=8388607 - - // Sets the maximum number of simultaneously open files for each server process. - max_files_per_process?: int & >=64 & <=2147483647 - - // Sets the maximum number of locks per transaction. - max_locks_per_transaction: int & >=10 & <=2147483647 | *64 - - // Maximum number of logical replication worker processes. - max_logical_replication_workers?: int & >=0 & <=262143 - - // Sets the maximum number of parallel processes per maintenance operation. - max_parallel_maintenance_workers?: int & >=0 & <=1024 - - // Sets the maximum number of parallel workers than can be active at one time. - max_parallel_workers?: int & >=0 & <=1024 - - // Sets the maximum number of parallel processes per executor node. - max_parallel_workers_per_gather?: int & >=0 & <=1024 - - // Sets the maximum number of predicate-locked tuples per page. - max_pred_locks_per_page?: int & >=0 & <=2147483647 - - // Sets the maximum number of predicate-locked pages and tuples per relation. - max_pred_locks_per_relation?: int & >=-2147483648 & <=2147483647 - - // Sets the maximum number of predicate locks per transaction. - max_pred_locks_per_transaction?: int & >=10 & <=2147483647 - - // Sets the maximum number of simultaneously prepared transactions. - max_prepared_transactions: int & >=0 & <=8388607 | *0 - - // Sets the maximum number of replication slots that the server can support. - max_replication_slots: int & >=5 & <=8388607 | *20 - - // (MB) Sets the maximum WAL size that can be reserved by replication slots. - max_slot_wal_keep_size?: int & >=-1 & <=2147483647 @storeResource(1MB) - - // (kB) Sets the maximum stack depth, in kilobytes. - max_stack_depth: int & >=100 & <=2147483647 | *6144 @storeResource(1KB) - - // (ms) Sets the maximum delay before canceling queries when a hot standby server is processing archived WAL data. - max_standby_archive_delay?: int & >=-1 & <=2147483647 @timeDurationResource() - - // (ms) Sets the maximum delay before canceling queries when a hot standby server is processing streamed WAL data. - max_standby_streaming_delay?: int & >=-1 & <=2147483647 @timeDurationResource() - - // Maximum number of synchronization workers per subscription - max_sync_workers_per_subscription?: int & >=0 & <=262143 - - // Sets the maximum number of simultaneously running WAL sender processes. - max_wal_senders: int & >=5 & <=8388607 | *20 - - // (MB) Sets the WAL size that triggers a checkpoint. - max_wal_size: int & >=128 & <=201326592 | *2048 @storeResource(1MB) - - // Sets the maximum number of concurrent worker processes. - max_worker_processes?: int & >=0 & <=262143 - - // Specifies the amount of memory that should be allocated at server startup for use by parallel queries - min_dynamic_shared_memory?: int & >=0 & <=715827882 @storeResource(1MB) - - // (8kB) Sets the minimum amount of index data for a parallel scan. - min_parallel_index_scan_size?: int & >=0 & <=715827882 @storeResource(8KB) - - // (8kB) Sets the minimum amount of table data for a parallel scan. - min_parallel_table_scan_size?: int & >=0 & <=715827882 @storeResource(8KB) - - // (MB) Sets the minimum size to shrink the WAL to. - min_wal_size: int & >=128 & <=201326592 | *192 @storeResource(1MB) - - // (min) Time before a snapshot is too old to read pages changed after the snapshot was taken. - old_snapshot_threshold?: int & >=-1 & <=86400 - - // Emulate oracle's date output behaviour. - "orafce.nls_date_format"?: string - - // Specify timezone used for sysdate function. - "orafce.timezone"?: string - - // Controls whether Gather and Gather Merge also run subplans. - parallel_leader_participation?: bool & false | true - - // Sets the planner's estimate of the cost of starting up worker processes for parallel query. - parallel_setup_cost?: float & >=0 & <=1.79769 - - // Sets the planner's estimate of the cost of passing each tuple (row) from worker to master backend. - parallel_tuple_cost?: float & >=0 & <=1.79769 - - // Encrypt passwords. - password_encryption?: string & "md5" | "scram-sha-256" - - // Specifies which classes of statements will be logged by session audit logging. - "pgaudit.log"?: string & "ddl" | "function" | "misc" | "read" | "role" | "write" | "none" | "all" | "-ddl" | "-function" | "-misc" | "-read" | "-role" | "-write" - - // Specifies that session logging should be enabled in the case where all relations in a statement are in pg_catalog. - "pgaudit.log_catalog"?: bool & false | true - - // Specifies the log level that will be used for log entries. - "pgaudit.log_level"?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "info" | "notice" | "warning" | "log" - - // Specifies that audit logging should include the parameters that were passed with the statement. - "pgaudit.log_parameter"?: bool & false | true - - // Specifies whether session audit logging should create a separate log entry for each relation (TABLE, VIEW, etc.) referenced in a SELECT or DML statement. - "pgaudit.log_relation"?: bool & false | true - - // Specifies that audit logging should include the rows retrieved or affected by a statement. - "pgaudit.log_rows": bool & false | true | *false - - // Specifies whether logging will include the statement text and parameters (if enabled). - "pgaudit.log_statement": bool & false | true | *true - - // Specifies whether logging will include the statement text and parameters with the first log entry for a statement/substatement combination or with every entry. - "pgaudit.log_statement_once"?: bool & false | true - - // Specifies the master role to use for object audit logging. - "pgaudit.role"?: string - - // It specifies whether to perform Recheck which is an internal process of full text search. - "pg_bigm.enable_recheck"?: string & "on" | "off" - - // It specifies the maximum number of 2-grams of the search keyword to be used for full text search. - "pg_bigm.gin_key_limit": int & >=0 & <=2147483647 | *0 - - // It specifies the minimum threshold used by the similarity search. - "pg_bigm.similarity_limit": float & >=0 & <=1 | *0.3 - - // Logs results of hint parsing. - "pg_hint_plan.debug_print"?: string & "off" | "on" | "detailed" | "verbose" - - // Force planner to use plans specified in the hint comment preceding to the query. - "pg_hint_plan.enable_hint"?: bool & false | true - - // Force planner to not get hint by using table lookups. - "pg_hint_plan.enable_hint_table"?: bool & false | true - - // Message level of debug messages. - "pg_hint_plan.message_level"?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "log" | "info" | "notice" | "warning" | "error" - - // Message level of parse errors. - "pg_hint_plan.parse_messages"?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "log" | "info" | "notice" | "warning" | "error" - - // Batch inserts if possible - "pglogical.batch_inserts"?: bool & false | true - - // Sets log level used for logging resolved conflicts. - "pglogical.conflict_log_level"?: string & "debug5" | "debug4" | "debug3" | "debug2" | "debug1" | "info" | "notice" | "warning" | "error" | "log" | "fatal" | "panic" - - // Sets method used for conflict resolution for resolvable conflicts. - "pglogical.conflict_resolution"?: string & "error" | "apply_remote" | "keep_local" | "last_update_wins" | "first_update_wins" - - // connection options to add to all peer node connections - "pglogical.extra_connection_options"?: string - - // pglogical specific synchronous commit value - "pglogical.synchronous_commit"?: bool & false | true - - // Use SPI instead of low-level API for applying changes - "pglogical.use_spi"?: bool & false | true - - // Starts the autoprewarm worker. - "pg_prewarm.autoprewarm"?: bool & false | true - - // Sets the interval between dumps of shared buffers - "pg_prewarm.autoprewarm_interval"?: int & >=0 & <=2147483 - - // Sets if the result value is normalized or not. - "pg_similarity.block_is_normalized"?: bool & false | true - - // Sets the threshold used by the Block similarity function. - "pg_similarity.block_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Block similarity function. - "pg_similarity.block_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.cosine_is_normalized"?: bool & false | true - - // Sets the threshold used by the Cosine similarity function. - "pg_similarity.cosine_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Cosine similarity function. - "pg_similarity.cosine_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.dice_is_normalized"?: bool & false | true - - // Sets the threshold used by the Dice similarity measure. - "pg_similarity.dice_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Dice similarity measure. - "pg_similarity.dice_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.euclidean_is_normalized"?: bool & false | true - - // Sets the threshold used by the Euclidean similarity measure. - "pg_similarity.euclidean_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Euclidean similarity measure. - "pg_similarity.euclidean_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.hamming_is_normalized"?: bool & false | true - - // Sets the threshold used by the Block similarity metric. - "pg_similarity.hamming_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.jaccard_is_normalized"?: bool & false | true - - // Sets the threshold used by the Jaccard similarity measure. - "pg_similarity.jaccard_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Jaccard similarity measure. - "pg_similarity.jaccard_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.jaro_is_normalized"?: bool & false | true - - // Sets the threshold used by the Jaro similarity measure. - "pg_similarity.jaro_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.jarowinkler_is_normalized"?: bool & false | true - - // Sets the threshold used by the Jarowinkler similarity measure. - "pg_similarity.jarowinkler_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.levenshtein_is_normalized"?: bool & false | true - - // Sets the threshold used by the Levenshtein similarity measure. - "pg_similarity.levenshtein_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.matching_is_normalized"?: bool & false | true - - // Sets the threshold used by the Matching Coefficient similarity measure. - "pg_similarity.matching_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Matching Coefficient similarity measure. - "pg_similarity.matching_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.mongeelkan_is_normalized"?: bool & false | true - - // Sets the threshold used by the Monge-Elkan similarity measure. - "pg_similarity.mongeelkan_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Monge-Elkan similarity measure. - "pg_similarity.mongeelkan_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets the gap penalty used by the Needleman-Wunsch similarity measure. - "pg_similarity.nw_gap_penalty"?: float & >=-9.22337e+18 & <=9.22337e+18 - - // Sets if the result value is normalized or not. - "pg_similarity.nw_is_normalized"?: bool & false | true - - // Sets the threshold used by the Needleman-Wunsch similarity measure. - "pg_similarity.nw_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.overlap_is_normalized"?: bool & false | true - - // Sets the threshold used by the Overlap Coefficient similarity measure. - "pg_similarity.overlap_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Overlap Coefficientsimilarity measure. - "pg_similarity.overlap_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.qgram_is_normalized"?: bool & false | true - - // Sets the threshold used by the Q-Gram similarity measure. - "pg_similarity.qgram_threshold"?: float & >=0 & <=1 - - // Sets the tokenizer for Q-Gram measure. - "pg_similarity.qgram_tokenizer"?: string & "alnum" | "gram" | "word" | "camelcase" - - // Sets if the result value is normalized or not. - "pg_similarity.swg_is_normalized"?: bool & false | true - - // Sets the threshold used by the Smith-Waterman-Gotoh similarity measure. - "pg_similarity.swg_threshold"?: float & >=0 & <=1 - - // Sets if the result value is normalized or not. - "pg_similarity.sw_is_normalized"?: bool & false | true - - // Sets the threshold used by the Smith-Waterman similarity measure. - "pg_similarity.sw_threshold"?: float & >=0 & <=1 - - // Sets the maximum number of statements tracked by pg_stat_statements. - "pg_stat_statements.max"?: int & >=100 & <=2147483647 - - // Save pg_stat_statements statistics across server shutdowns. - "pg_stat_statements.save"?: bool & false | true - - // Selects which statements are tracked by pg_stat_statements. - "pg_stat_statements.track"?: string & "none" | "top" | "all" - - // Selects whether planning duration is tracked by pg_stat_statements. - "pg_stat_statements.track_planning"?: bool & false | true - - // Selects whether utility commands are tracked by pg_stat_statements. - "pg_stat_statements.track_utility"?: bool & false | true - - // Sets the behavior for interacting with passcheck feature. - "pgtle.enable_password_check"?: string & "on" | "off" | "require" - - // Number of workers to use for a physical transport. - "pg_transport.num_workers"?: int & >=1 & <=32 - - // Specifies whether to report timing information during transport. - "pg_transport.timing"?: bool & false | true - - // (kB) Amount of memory each worker can allocate for a physical transport. - "pg_transport.work_mem"?: int & >=65536 & <=2147483647 @storeResource(1KB) - - // Controls the planner selection of custom or generic plan. - plan_cache_mode?: string & "auto" | "force_generic_plan" | "force_custom_plan" - - // Sets the TCP port the server listens on. - port?: int & >=1 & <=65535 - - // Enable for disable GDAL drivers used with PostGIS in Postgres 9.3.5 and above. - "postgis.gdal_enabled_drivers"?: string & "ENABLE_ALL" | "DISABLE_ALL" - - // When generating SQL fragments, quote all identifiers. - quote_all_identifiers?: bool & false | true - - // Sets the planners estimate of the cost of a nonsequentially fetched disk page. - random_page_cost?: float & >=0 & <=1.79769 - - // Lower threshold of Dice similarity. Molecules with similarity lower than threshold are not similar by # operation. - "rdkit.dice_threshold"?: float & >=0 & <=1 - - // Should stereochemistry be taken into account in substructure matching. If false, no stereochemistry information is used in substructure matches. - "rdkit.do_chiral_sss"?: bool & false | true - - // Should enhanced stereochemistry be taken into account in substructure matching. - "rdkit.do_enhanced_stereo_sss"?: bool & false | true - - // Lower threshold of Tanimoto similarity. Molecules with similarity lower than threshold are not similar by % operation. - "rdkit.tanimoto_threshold"?: float & >=0 & <=1 - - // When set to fsync, PostgreSQL will recursively open and synchronize all files in the data directory before crash recovery begins - recovery_init_sync_method?: string & "fsync" | "syncfs" - - // When set to on, which is the default, PostgreSQL will automatically remove temporary files after a backend crash - remove_temp_files_after_crash: float & >=0 & <=1 | *0 - - // Reinitialize server after backend crash. - restart_after_crash?: bool & false | true - - // Enable row security. - row_security?: bool & false | true - - // Sets the schema search order for names that are not schema-qualified. - search_path?: string - - // Sets the planners estimate of the cost of a sequentially fetched disk page. - seq_page_cost?: float & >=0 & <=1.79769 - - // Lists shared libraries to preload into each backend. - session_preload_libraries?: string & "auto_explain" | "orafce" | "pg_bigm" | "pg_hint_plan" | "pg_prewarm" | "pg_similarity" | "pg_stat_statements" | "pg_transport" | "plprofiler" - - // Sets the sessions behavior for triggers and rewrite rules. - session_replication_role?: string & "origin" | "replica" | "local" - - // (8kB) Sets the number of shared memory buffers used by the server. - shared_buffers?: int & >=16 & <=1073741823 @storeResource(8KB) - - // Lists shared libraries to preload into server. - // TODO support enum list, e.g. shared_preload_libraries = 'pg_stat_statements, auto_explain' - // shared_preload_libraries?: string & "auto_explain" | "orafce" | "pgaudit" | "pglogical" | "pg_bigm" | "pg_cron" | "pg_hint_plan" | "pg_prewarm" | "pg_similarity" | "pg_stat_statements" | "pg_tle" | "pg_transport" | "plprofiler" - - // Enables SSL connections. - ssl: bool & false | true | *true - - // Location of the SSL server authority file. - ssl_ca_file?: string - - // Location of the SSL server certificate file. - ssl_cert_file?: string - - // Sets the list of allowed SSL ciphers. - ssl_ciphers?: string - - // Location of the SSL server private key file - ssl_key_file?: string - - // Sets the maximum SSL/TLS protocol version to use. - ssl_max_protocol_version?: string & "TLSv1" | "TLSv1.1" | "TLSv1.2" - - // Sets the minimum SSL/TLS protocol version to use. - ssl_min_protocol_version?: string & "TLSv1" | "TLSv1.1" | "TLSv1.2" - - // Causes ... strings to treat backslashes literally. - standard_conforming_strings?: bool & false | true - - // (ms) Sets the maximum allowed duration of any statement. - statement_timeout?: int & >=0 & <=2147483647 @timeDurationResource() - - // Writes temporary statistics files to the specified directory. - stats_temp_directory?: string - - // Sets the number of connection slots reserved for superusers. - superuser_reserved_connections: int & >=0 & <=8388607 | *3 - - // Enable synchronized sequential scans. - synchronize_seqscans?: bool & false | true - - // Sets the current transactions synchronization level. - synchronous_commit?: string & "local" | "on" | "off" - - // Maximum number of TCP keepalive retransmits. - tcp_keepalives_count?: int & >=0 & <=2147483647 - - // (s) Time between issuing TCP keepalives. - tcp_keepalives_idle?: int & >=0 & <=2147483647 @timeDurationResource(1s) - - // (s) Time between TCP keepalive retransmits. - tcp_keepalives_interval?: int & >=0 & <=2147483647 @timeDurationResource(1s) - - // (8kB) Sets the maximum number of temporary buffers used by each session. - temp_buffers?: int & >=100 & <=1073741823 @storeResource(8KB) - - // (kB) Limits the total size of all temporary files used by each process. - temp_file_limit?: int & >=-1 & <=2147483647 @storeResource(8KB) - - // Sets the tablespace(s) to use for temporary tables and sort files. - temp_tablespaces?: string - - // Sets the time zone for displaying and interpreting time stamps. - timezone?: string - - // Collects information about executing commands. - track_activities?: bool & false | true - - // Sets the size reserved for pg_stat_activity.current_query, in bytes. - track_activity_query_size: int & >=100 & <=1048576 | *4096 - - // Collects transaction commit time. - track_commit_timestamp?: bool & false | true - - // Collects statistics on database activity. - track_counts?: bool & false | true - - // Collects function-level statistics on database activity. - track_functions?: string & "none" | "pl" | "all" - - // Collects timing statistics on database IO activity. - track_io_timing: bool & false | true | *true - - // Enables timing of WAL I/O calls. - track_wal_io_timing?: bool & false | true - - // Treats expr=NULL as expr IS NULL. - transform_null_equals?: bool & false | true - - // Sets the directory where the Unix-domain socket will be created. - unix_socket_directories?: string - - // Sets the owning group of the Unix-domain socket. - unix_socket_group?: string - - // Sets the access permissions of the Unix-domain socket. - unix_socket_permissions?: int & >=0 & <=511 - - // Updates the process title to show the active SQL command. - update_process_title: bool & false | true | *true - - // (ms) Vacuum cost delay in milliseconds. - vacuum_cost_delay?: int & >=0 & <=100 @timeDurationResource() - - // Vacuum cost amount available before napping. - vacuum_cost_limit?: int & >=1 & <=10000 - - // Vacuum cost for a page dirtied by vacuum. - vacuum_cost_page_dirty?: int & >=0 & <=10000 - - // Vacuum cost for a page found in the buffer cache. - vacuum_cost_page_hit?: int & >=0 & <=10000 - - // Vacuum cost for a page not found in the buffer cache. - vacuum_cost_page_miss: int & >=0 & <=10000 | *5 - - // Number of transactions by which VACUUM and HOT cleanup should be deferred, if any. - vacuum_defer_cleanup_age?: int & >=0 & <=1000000 - - // Specifies the maximum age (in transactions) that a table's pg_class.relfrozenxid field can attain before VACUUM takes extraordinary measures to avoid system-wide transaction ID wraparound failure - vacuum_failsafe_age: int & >=0 & <=1200000000 | *1200000000 - - // Minimum age at which VACUUM should freeze a table row. - vacuum_freeze_min_age?: int & >=0 & <=1000000000 - - // Age at which VACUUM should scan whole table to freeze tuples. - vacuum_freeze_table_age?: int & >=0 & <=2000000000 - - // Specifies the maximum age (in transactions) that a table's pg_class.relminmxid field can attain before VACUUM takes extraordinary measures to avoid system-wide multixact ID wraparound failure - vacuum_multixact_failsafe_age: int & >=0 & <=1200000000 | *1200000000 - - // Minimum age at which VACUUM should freeze a MultiXactId in a table row. - vacuum_multixact_freeze_min_age?: int & >=0 & <=1000000000 - - // Multixact age at which VACUUM should scan whole table to freeze tuples. - vacuum_multixact_freeze_table_age?: int & >=0 & <=2000000000 - - // (8kB) Sets the number of disk-page buffers in shared memory for WAL. - wal_buffers?: int & >=-1 & <=262143 @storeResource(8KB) - - // Compresses full-page writes written in WAL file. - wal_compression: bool & false | true | *true - - // (MB) Sets the size of WAL files held for standby servers. - wal_keep_size: int & >=0 & <=2147483647 | *2048 @storeResource(1MB) - - // Sets whether a WAL receiver should create a temporary replication slot if no permanent slot is configured. - wal_receiver_create_temp_slot: bool & false | true | *false - - // (s) Sets the maximum interval between WAL receiver status reports to the primary. - wal_receiver_status_interval?: int & >=0 & <=2147483 @timeDurationResource(1s) - - // (ms) Sets the maximum wait time to receive data from the primary. - wal_receiver_timeout: int & >=0 & <=3600000 | *30000 @timeDurationResource() - - // (ms) Sets the maximum time to wait for WAL replication. - wal_sender_timeout: int & >=0 & <=3600000 | *30000 @timeDurationResource() - - // (kB) Size of new file to fsync instead of writing WAL. - wal_skip_threshold?: int & >=0 & <=2147483647 @storeResource(1KB) - - // Selects the method used for forcing WAL updates to disk. - wal_sync_method?: string & "fsync" | "fdatasync" | "open_sync" | "open_datasync" - - // (ms) WAL writer sleep time between WAL flushes. - wal_writer_delay?: int & >=1 & <=10000 @timeDurationResource() - - // (8Kb) Amount of WAL written out by WAL writer triggering a flush. - wal_writer_flush_after?: int & >=0 & <=2147483647 @storeResource(8KB) - - // (kB) Sets the maximum memory to be used for query workspaces. - work_mem?: int & >=64 & <=2147483647 @storeResource(1KB) - - // Sets how binary values are to be encoded in XML. - xmlbinary?: string & "base64" | "hex" - - // Sets whether XML data in implicit parsing and serialization operations is to be considered as documents or content fragments. - xmloption?: string & "content" | "document" - - ... -} - -configuration: #PGParameter & { -} diff --git a/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml b/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml deleted file mode 100644 index 6e4d69046..000000000 --- a/deploy/postgresql-patroni-ha/config/pg14-config-effect-scope.yaml +++ /dev/null @@ -1,22 +0,0 @@ -# Patroni bootstrap parameters -staticParameters: - - archive_command - - shared_buffers - - logging_collector - - log_destination - - log_directory - - log_filename - - log_file_mode - - log_rotation_age - - log_truncate_on_rotation - - ssl - - ssl_ca_file - - ssl_crl_file - - ssl_cert_file - - ssl_key_file - - shared_preload_libraries - - bg_mon.listen_address - - bg_mon.history_buckets - - pg_stat_statements.track_utility - - extwlist.extensions - - extwlist.custom_path \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/config/pg14-config.tpl b/deploy/postgresql-patroni-ha/config/pg14-config.tpl deleted file mode 100644 index c38b79027..000000000 --- a/deploy/postgresql-patroni-ha/config/pg14-config.tpl +++ /dev/null @@ -1,123 +0,0 @@ -# - Connection Settings - - -{{- $buffer_unit := "B" }} -{{- $shared_buffers := 1073741824 }} -{{- $max_connections := 10000 }} -{{- $phy_memory := getContainerMemory ( index $.podSpec.containers 0 ) }} -{{- if gt $phy_memory 0 }} -{{- $shared_buffers = div $phy_memory 4 }} -{{- $max_connections = min ( div $phy_memory 9531392 ) 5000 }} -{{- end -}} - -{{- if ge $shared_buffers 1024 }} -{{- $shared_buffers = div $shared_buffers 1024 }} -{{- $buffer_unit = "KB" }} -{{- end -}} - -{{- if ge $shared_buffers 1024 }} -{{- $shared_buffers = div $shared_buffers 1024 }} -{{- $buffer_unit = "MB" }} -{{- end -}} - -{{- if ge $shared_buffers 1024 }} -{{- $shared_buffers = div $shared_buffers 1024 }} -{{ $buffer_unit = "GB" }} -{{- end -}} - -listen_addresses = '*' -port = '5432' -#archive_command = 'wal_dir=/pg/arcwal; [[ $(date +%H%M) == 1200 ]] && rm -rf ${wal_dir}/$(date -d"yesterday" +%Y%m%d); /bin/mkdir -p ${wal_dir}/$(date +%Y%m%d) && /usr/bin/lz4 -q -z %p > ${wal_dir}/$(date +%Y%m%d)/%f.lz4' -#archive_mode = 'True' -auto_explain.log_analyze = 'True' -auto_explain.log_min_duration = '1s' -auto_explain.log_nested_statements = 'True' -auto_explain.log_timing = 'True' -auto_explain.log_verbose = 'True' -autovacuum_analyze_scale_factor = '0.05' -autovacuum_freeze_max_age = '100000000' -autovacuum_max_workers = '1' -autovacuum_naptime = '1min' -autovacuum_vacuum_cost_delay = '-1' -autovacuum_vacuum_cost_limit = '-1' -autovacuum_vacuum_scale_factor = '0.1' -bgwriter_delay = '10ms' -bgwriter_lru_maxpages = '800' -bgwriter_lru_multiplier = '5.0' -checkpoint_completion_target = '0.95' -checkpoint_timeout = '10min' -commit_delay = '20' -commit_siblings = '10' -deadlock_timeout = '50ms' -default_statistics_target = '500' -effective_cache_size = '12GB' -hot_standby = 'on' -hot_standby_feedback = 'True' -huge_pages = 'try' -idle_in_transaction_session_timeout = '1h' -listen_addresses = '0.0.0.0' -log_autovacuum_min_duration = '1s' -log_checkpoints = 'True' -log_destination = 'csvlog' -log_directory = 'log' -log_filename = 'postgresql-%Y-%m-%d.log' -log_lock_waits = 'True' -log_min_duration_statement = '100' -log_replication_commands = 'True' -log_statement = 'ddl' -logging_collector = 'True' -#maintenance_work_mem = '3952MB' -max_connections = '{{ $max_connections }}' -max_locks_per_transaction = '128' -max_logical_replication_workers = '8' -max_parallel_maintenance_workers = '2' -max_parallel_workers = '8' -max_parallel_workers_per_gather = '0' -max_prepared_transactions = '0' -max_replication_slots = '16' -max_standby_archive_delay = '10min' -max_standby_streaming_delay = '3min' -max_sync_workers_per_subscription = '6' -max_wal_senders = '24' -max_wal_size = '100GB' -max_worker_processes = '8' -min_wal_size = '20GB' -password_encryption = 'md5' -pg_stat_statements.max = '5000' -pg_stat_statements.track = 'all' -pg_stat_statements.track_planning = 'False' -pg_stat_statements.track_utility = 'False' -random_page_cost = '1.1' -#auto generated -shared_buffers = '{{ printf "%d%s" $shared_buffers $buffer_unit }}' -shared_preload_libraries = 'pg_stat_statements, auto_explain' -superuser_reserved_connections = '10' -temp_file_limit = '100GB' -#timescaledb.max_background_workers = '6' -#timescaledb.telemetry_level = 'off' -track_activity_query_size = '8192' -track_commit_timestamp = 'True' -track_functions = 'all' -track_io_timing = 'True' -vacuum_cost_delay = '2ms' -vacuum_cost_limit = '10000' -vacuum_defer_cleanup_age = '50000' -wal_buffers = '16MB' -wal_keep_size = '20GB' -wal_level = 'replica' -wal_log_hints = 'on' -wal_receiver_status_interval = '1s' -wal_receiver_timeout = '60s' -wal_writer_delay = '20ms' -wal_writer_flush_after = '1MB' -work_mem = '32MB' - -{{- if $.component.tls }} -{{- $ca_file := getCAFile }} -{{- $cert_file := getCertFile }} -{{- $key_file := getKeyFile }} -# tls -ssl=ON -ssl_ca_file={{ $ca_file }} -ssl_cert_file={{ $cert_file }} -ssl_key_file={{ $key_file }} -{{- end }} \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/templates/_helpers.tpl b/deploy/postgresql-patroni-ha/templates/_helpers.tpl deleted file mode 100644 index 6ee64a159..000000000 --- a/deploy/postgresql-patroni-ha/templates/_helpers.tpl +++ /dev/null @@ -1,98 +0,0 @@ -{{/* -Expand the name of the chart. -*/}} -{{- define "postgresql.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "postgresql.fullname" -}} -{{- if .Values.fullnameOverride }} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- $name := default .Chart.Name .Values.nameOverride }} -{{- if contains $name .Release.Name }} -{{- .Release.Name | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} -{{- end }} -{{- end }} -{{- end }} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "postgresql.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} -{{- end }} - -{{/* -Common labels -*/}} -{{- define "postgresql.labels" -}} -helm.sh/chart: {{ include "postgresql.chart" . }} -{{ include "postgresql.selectorLabels" . }} -{{- if .Chart.AppVersion }} -app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} -{{- end }} -app.kubernetes.io/managed-by: {{ .Release.Service }} -{{- end }} - -{{/* -Selector labels -*/}} -{{- define "postgresql.selectorLabels" -}} -app.kubernetes.io/name: {{ include "postgresql.name" . }} -app.kubernetes.io/instance: {{ .Release.Name }} -{{- end }} - -{{/* -Create the name of the service account to use -*/}} -{{- define "postgresql.serviceAccountName" -}} -{{- if .Values.serviceAccount.create }} -{{- default (include "postgresql.fullname" .) .Values.serviceAccount.name }} -{{- else }} -{{- default "default" .Values.serviceAccount.name }} -{{- end }} -{{- end }} - -{{/* -Return true if a configmap object should be created for PostgreSQL primary with the configuration -*/}} -{{- define "postgresql.primary.createConfigmap" -}} -{{- if and (or .Values.primary.configuration .Values.primary.pgHbaConfiguration) (not .Values.primary.existingConfigmap) }} - {{- true -}} -{{- else -}} -{{- end -}} -{{- end -}} - -{{/* -Return PostgreSQL service port -*/}} -{{- define "postgresql.service.port" -}} -{{- .Values.primary.service.ports.postgresql -}} -{{- end -}} - -{{/* -Return the name for a custom database to create -*/}} -{{- define "postgresql.database" -}} -{{- .Values.auth.database -}} -{{- end -}} - -{{/* -Get the password key. -*/}} -{{/* TODO: use $(RANDOM_PASSWD) instead */}} -{{- define "postgresql.postgresPassword" -}} -{{- if or (.Release.IsInstall) (not (lookup "apps.kubeblocks.io/v1alpha1" "ClusterDefinition" "" "postgresql")) -}} -{{ .Values.auth.postgresPassword | default (randAlphaNum 10) }} -{{- else -}} -{{ index (lookup "apps.kubeblocks.io/v1alpha1" "ClusterDefinition" "" "postgresql").spec.connectionCredential "password"}} -{{- end }} -{{- end }} diff --git a/deploy/postgresql-patroni-ha/templates/_tplvalues.tpl b/deploy/postgresql-patroni-ha/templates/_tplvalues.tpl deleted file mode 100644 index 4e2d01631..000000000 --- a/deploy/postgresql-patroni-ha/templates/_tplvalues.tpl +++ /dev/null @@ -1,12 +0,0 @@ -{{/* -Renders a value that contains template. -Usage: -{{ include "tplvalues.render" ( dict "value" .Values.path.to.the.Value "context" $) }} -*/}} -{{- define "tplvalues.render" -}} - {{- if typeIs "string" .value }} - {{- tpl .value .context }} - {{- else }} - {{- tpl (.value | toYaml) .context }} - {{- end }} -{{- end -}} diff --git a/deploy/postgresql-patroni-ha/templates/backuppolicytemplate.yaml b/deploy/postgresql-patroni-ha/templates/backuppolicytemplate.yaml deleted file mode 100644 index e1a44b496..000000000 --- a/deploy/postgresql-patroni-ha/templates/backuppolicytemplate.yaml +++ /dev/null @@ -1,15 +0,0 @@ -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -kind: BackupPolicyTemplate -metadata: - name: backup-policy-template-postgresql-ha - labels: - clusterdefinition.kubeblocks.io/name: postgresql-ha - {{- include "postgresql.labels" . | nindent 4 }} -spec: - # which backup tool to perform database backup, only support one tool. - backupToolName: volumesnapshot - ttl: 168h0m0s - - credentialKeyword: - userKeyword: username - passwordKeyword: password diff --git a/deploy/postgresql-patroni-ha/templates/backuptool.yaml b/deploy/postgresql-patroni-ha/templates/backuptool.yaml deleted file mode 100644 index 539e7e788..000000000 --- a/deploy/postgresql-patroni-ha/templates/backuptool.yaml +++ /dev/null @@ -1,57 +0,0 @@ -apiVersion: dataprotection.kubeblocks.io/v1alpha1 -kind: BackupTool -metadata: - name: postgres-ha-basebackup - labels: - clusterdefinition.kubeblocks.io/name: postgresql-ha - {{- include "postgresql.labels" . | nindent 4 }} -spec: - image: registry.cn-hangzhou.aliyuncs.com/apecloud/postgresql:14.7.0 - deployKind: job - resources: - limits: - cpu: "1" - memory: 2Gi - requests: - cpu: "1" - memory: 128Mi - env: - - name: RESTORE_DATA_DIR - value: /home/postgres/pgdata/kb_restore - - name: TMP_DATA_DIR - value: /home/postgres/pgdata/kb_restore/tmp_data - - name: TMP_ARCH_DATA_DIR - value: /home/postgres/pgdata/kb_restore/arch - - name: DATA_DIR - value: /home/postgres/pgdata/pgroot/data - physical: - restoreCommands: - - | - #!/bin/sh - set -e - # create a new directory for restore - mkdir -p ${RESTORE_DATA_DIR} && rm -rf ${RESTORE_DATA_DIR}/* - cd ${RESTORE_DATA_DIR} - touch kb_restore.sh && touch kb_restore.signal - echo "mkdir -p ${DATA_DIR}/../arch" >> kb_restore.sh - echo "mv -f ${TMP_DATA_DIR}/* ${DATA_DIR}/" >> kb_restore.sh - echo "mv -f ${TMP_ARCH_DATA_DIR}/* ${DATA_DIR}/../arch" >> kb_restore.sh - echo "rm -rf ${RESTORE_DATA_DIR}" >> kb_restore.sh - - # extract the data file to the temporary data directory - mkdir -p ${TMP_DATA_DIR} && mkdir -p ${TMP_ARCH_DATA_DIR} - rm -rf ${TMP_ARCH_DATA_DIR}/* && rm -rf ${TMP_DATA_DIR}/* - cd ${BACKUP_DIR}/${BACKUP_NAME} - tar -xvf base.tar.gz -C ${TMP_DATA_DIR}/ - tar -xvf pg_wal.tar.gz -C ${TMP_ARCH_DATA_DIR}/ - echo "done!" - incrementalRestoreCommands: [] - logical: - restoreCommands: [] - incrementalRestoreCommands: [] - backupCommands: - - > - set -e; - mkdir -p ${BACKUP_DIR}/${BACKUP_NAME}/; - echo ${DB_PASSWORD} | pg_basebackup -Ft -Pv -Xs -z -D ${BACKUP_DIR}/${BACKUP_NAME} -Z5 -h ${DB_HOST} -U standby -W; - incrementalBackupCommands: [] diff --git a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml b/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml deleted file mode 100644 index 2015c1273..000000000 --- a/deploy/postgresql-patroni-ha/templates/clusterdefinition.yaml +++ /dev/null @@ -1,317 +0,0 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 -kind: ClusterDefinition -metadata: - name: postgresql-ha - labels: - {{- include "postgresql.labels" . | nindent 4 }} -spec: - type: postgresql - connectionCredential: - username: postgres - password: "$(RANDOM_PASSWD)" - endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-postgresql)" - host: "$(SVC_FQDN)" - port: "$(SVC_PORT_tcp-postgresql)" - componentDefs: - - name: postgresql-ha - workloadType: Replication - characterType: postgresql - customLabelSpecs: - - key: apps.kubeblocks.postgres.patroni/scope - value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" - resources: - - gvk: "v1/Pod" - selector: - app.kubernetes.io/managed-by: kubeblocks - - gvk: "apps/v1/StatefulSet" - selector: - app.kubernetes.io/managed-by: kubeblocks - probes: - roleChangedProbe: - failureThreshold: 2 - periodSeconds: 1 - timeoutSeconds: 1 - monitor: - builtIn: false - exporterConfig: - scrapePath: /metrics - scrapePort: 9187 - logConfigs: - {{- range $name,$pattern := .Values.logConfigs }} - - name: {{ $name }} - filePathPattern: {{ $pattern }} - {{- end }} - configSpecs: - - name: postgresql-ha-configuration - templateRef: postgresql-ha-configuration - constraintRef: postgresql14-ha-cc - keys: - - postgresql.conf - namespace: {{ .Release.Namespace }} - volumeName: postgresql-config - defaultMode: 0777 - - name: postgresql-ha-custom-metrics - templateRef: postgresql-ha-custom-metrics - namespace: {{ .Release.Namespace }} - volumeName: postgresql-custom-metrics - defaultMode: 0777 - scriptSpecs: - - name: pg-patroni-scripts - templateRef: pg-patroni-scripts - namespace: {{ .Release.Namespace }} - volumeName: scripts - defaultMode: 0777 - service: - ports: - - name: tcp-postgresql - port: 5432 - targetPort: tcp-postgresql - - name: http-metrics-postgresql - port: 9187 - targetPort: http-metrics - volumeTypes: - - name: data - type: data - podSpec: - serviceAccountName: operator - securityContext: - runAsUser: 0 - fsGroup: 103 - runAsGroup: 103 - initContainers: - - name: pg-init-container - imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} - command: - - /kb-scripts/init_container.sh - volumeMounts: - - name: data - mountPath: /home/postgres/pgdata - mode: 0777 - - name: postgresql-config - mountPath: /home/postgres/conf - mode: 0777 - - name: scripts - mountPath: /kb-scripts - mode: 0777 - containers: - - name: postgresql - imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} - securityContext: - runAsUser: 0 - command: - - /kb-scripts/setup.sh - readinessProbe: - failureThreshold: 6 - initialDelaySeconds: 10 - periodSeconds: 10 - successThreshold: 1 - timeoutSeconds: 5 - exec: - command: - - /bin/sh - - -c - - -ee - - | - exec pg_isready -U {{ default "postgres" | quote }} -h 127.0.0.1 -p 5432 - [ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ] - volumeMounts: - - name: dshm - mountPath: /dev/shm - - name: data - mountPath: /home/postgres/pgdata - mode: 0777 - - name: postgresql-config - mountPath: /home/postgres/conf - mode: 0777 - - name: scripts - mountPath: /kb-scripts - ports: - - name: tcp-postgresql - containerPort: 5432 - - name: patroni - containerPort: 8008 - env: ## refer https://github.com/zalando/spilo/blob/master/ENVIRONMENT.rst - - name: DCS_ENABLE_KUBERNETES_API - value: "true" - - name: KUBERNETES_USE_CONFIGMAPS - value: "true" - - name: SCOPE - value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" - - name: KUBERNETES_SCOPE_LABEL - value: "apps.kubeblocks.postgres.patroni/scope" - - name: KUBERNETES_ROLE_LABEL - value: "apps.kubeblocks.postgres.patroni/role" - - name: KUBERNETES_LABELS - value: '{"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)","app.kubernetes.io/managed-by":"kubeblocks"}' - - name: RESTORE_DATA_DIR - value: /home/postgres/pgdata/kb_restore - - name: SPILO_CONFIGURATION - value: | ## https://github.com/zalando/patroni#yaml-configuration - bootstrap: - initdb: - - auth-host: md5 - - auth-local: trust - - name: ALLOW_NOSSL - value: "true" - - name: PGROOT - value: /home/postgres/pgdata/pgroot - - name: POD_IP - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: status.podIP - - name: POD_NAMESPACE - valueFrom: - fieldRef: - apiVersion: v1 - fieldPath: metadata.namespace - - name: PGUSER_SUPERUSER - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: username - - name: PGPASSWORD_SUPERUSER - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - - name: PGUSER_ADMIN - value: superadmin - - name: PGPASSWORD_ADMIN - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - - name: PGPASSWORD_STANDBY - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - - name: PGUSER - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: username - - name: PGPASSWORD - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - - name: metrics - image: {{ .Values.metrics.image.registry | default "docker.io" }}/{{ .Values.metrics.image.repository }}:{{ .Values.metrics.image.tag }} - imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }} - securityContext: - runAsUser: 0 - env: - {{- $database := "postgres" }} - {{- $sslmode := "disable" }} - - name: DATA_SOURCE_URI - value: {{ printf "127.0.0.1:5432/%s?sslmode=%s" $database $sslmode }} - - name: DATA_SOURCE_PASS - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - - name: DATA_SOURCE_USER - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: username - command: - - "/opt/bitnami/postgres-exporter/bin/postgres_exporter" - - "--auto-discover-databases" - - "--extend.query-path=/opt/conf/custom-metrics.yaml" - - "--exclude-databases=template0,template1" - - "--log.level=info" - ports: - - name: http-metrics - containerPort: 9187 - livenessProbe: - failureThreshold: 6 - initialDelaySeconds: 5 - periodSeconds: 10 - successThreshold: 1 - timeoutSeconds: 5 - httpGet: - path: / - port: http-metrics - readinessProbe: - failureThreshold: 6 - initialDelaySeconds: 5 - periodSeconds: 10 - successThreshold: 1 - timeoutSeconds: 5 - httpGet: - path: / - port: http-metrics - volumeMounts: - - name: postgresql-custom-metrics - mountPath: /opt/conf - volumes: - - name: dshm - emptyDir: - medium: Memory - {{- with .Values.shmVolume.sizeLimit }} - sizeLimit: {{ . }} - {{- end }} - systemAccounts: - cmdExecutorConfig: - image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ default .Values.image.tag }} - command: - - psql - args: - - -h$(KB_ACCOUNT_ENDPOINT) - - -c - - $(KB_ACCOUNT_STATEMENT) - env: - - name: PGUSER - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: username - - name: PGPASSWORD - valueFrom: - secretKeyRef: - name: $(CONN_CREDENTIAL_SECRET_NAME) - key: password - passwordConfig: - length: 10 - numDigits: 5 - numSymbols: 0 - letterCase: MixedCases - accounts: - - name: kbadmin - provisionPolicy: - type: CreateByStmt - scope: AnyPods - statements: - creation: CREATE USER $(USERNAME) SUPERUSER PASSWORD '$(PASSWD)'; - deletion: DROP USER IF EXISTS $(USERNAME); - - name: kbdataprotection - provisionPolicy: - type: CreateByStmt - scope: AnyPods - statements: - creation: CREATE USER $(USERNAME) SUPERUSER PASSWORD '$(PASSWD)'; - deletion: DROP USER IF EXISTS $(USERNAME); - - name: kbprobe - provisionPolicy: - type: CreateByStmt - scope: AnyPods - statements: - creation: CREATE USER $(USERNAME) WITH PASSWORD '$(PASSWD)'; GRANT pg_monitor TO $(USERNAME); - deletion: DROP USER IF EXISTS $(USERNAME); - - name: kbmonitoring - provisionPolicy: - type: CreateByStmt - scope: AnyPods - statements: - creation: CREATE USER $(USERNAME) WITH PASSWORD '$(PASSWD)'; GRANT pg_monitor TO $(USERNAME); - deletion: DROP USER IF EXISTS $(USERNAME); - - name: kbreplicator - provisionPolicy: - type: CreateByStmt - scope: AnyPods - statements: - creation: CREATE USER $(USERNAME) WITH REPLICATION PASSWORD '$(PASSWD)'; - deletion: DROP USER IF EXISTS $(USERNAME); diff --git a/deploy/postgresql-patroni-ha/templates/clusterversion.yaml b/deploy/postgresql-patroni-ha/templates/clusterversion.yaml deleted file mode 100644 index 905173c61..000000000 --- a/deploy/postgresql-patroni-ha/templates/clusterversion.yaml +++ /dev/null @@ -1,18 +0,0 @@ -apiVersion: apps.kubeblocks.io/v1alpha1 -kind: ClusterVersion -metadata: - name: postgresql-{{ default .Chart.AppVersion .Values.clusterVersionOverride }} - labels: - {{- include "postgresql.labels" . | nindent 4 }} -spec: - clusterDefinitionRef: postgresql-ha - componentVersions: - - componentDefRef: postgresql-ha - versionsContext: - initContainers: - - name: pg-init-container - image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} - containers: - - name: postgresql - image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} ---- diff --git a/deploy/postgresql-patroni-ha/templates/configconstraint.yaml b/deploy/postgresql-patroni-ha/templates/configconstraint.yaml deleted file mode 100644 index b9445e995..000000000 --- a/deploy/postgresql-patroni-ha/templates/configconstraint.yaml +++ /dev/null @@ -1,47 +0,0 @@ -{{- $cc := .Files.Get "config/pg14-config-effect-scope.yaml" | fromYaml }} -apiVersion: apps.kubeblocks.io/v1alpha1 -kind: ConfigConstraint -metadata: - name: postgresql14-ha-cc - labels: - {{- include "postgresql.labels" . | nindent 4 }} -spec: - reloadOptions: - tplScriptTrigger: - scriptConfigMapRef: patroni-reload-script - namespace: {{ .Release.Namespace }} - - # top level mysql configuration type - cfgSchemaTopLevelName: PGParameter - - # ConfigurationSchema that impose restrictions on engine parameter's rule - configurationSchema: - # schema: auto generate from mmmcue scripts - # example: ../../internal/configuration/testdata/mysql_openapi.json - cue: |- - {{- .Files.Get "config/pg14-config-constraint.cue" | nindent 6 }} - - ## require db instance restart - ## staticParameters - {{- if hasKey $cc "staticParameters" }} - staticParameters: - {{- $params := get $cc "staticParameters" }} - {{- range $params }} - - {{ . }} - {{- end }} - {{- end}} - - ## define immutable parameter list, this feature is not currently supported. - {{- if hasKey $cc "immutableParameters" }} - immutableParameters: - {{- $params := get $cc "immutableParameters" }} - {{- range $params }} - - {{ . }} - {{- end }} - {{- end}} - - - - # configuration file format - formatterConfig: - format: properties diff --git a/deploy/postgresql-patroni-ha/templates/configmap.yaml b/deploy/postgresql-patroni-ha/templates/configmap.yaml deleted file mode 100644 index 679124ec5..000000000 --- a/deploy/postgresql-patroni-ha/templates/configmap.yaml +++ /dev/null @@ -1,26 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: postgresql-ha-configuration - labels: - {{- include "postgresql.labels" . | nindent 4 }} -data: - postgresql.conf: |- - {{- .Files.Get "config/pg14-config.tpl" | nindent 4 }} - # TODO: check if it should trust all - pg_hba.conf: | - host all all 0.0.0.0/0 trust - host all all ::/0 trust - local all all trust - host all all 127.0.0.1/32 trust - host all all ::1/128 trust - host replication all 0.0.0.0/0 md5 - host replication all ::/0 md5 - kb_restore.conf: | - method: kb_restore_from_backup - kb_restore_from_backup: - command: sh /home/postgres/pgdata/kb_restore/kb_restore.sh - keep_existing_recovery_conf: false - recovery_conf: - restore_command: cp /home/postgres/pgdata/pgroot/arch/%f %p - recovery_target_timeline: latest diff --git a/deploy/postgresql-patroni-ha/templates/metrics-configmap.yaml b/deploy/postgresql-patroni-ha/templates/metrics-configmap.yaml deleted file mode 100644 index 2ed1e4e15..000000000 --- a/deploy/postgresql-patroni-ha/templates/metrics-configmap.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: postgresql-ha-custom-metrics - labels: - {{- include "postgresql.labels" . | nindent 4 }} -data: - custom-metrics.yaml: {{ toYaml .Values.metrics.customMetrics | quote }} diff --git a/deploy/postgresql-patroni-ha/templates/scripts.yaml b/deploy/postgresql-patroni-ha/templates/scripts.yaml deleted file mode 100644 index 5d22881a8..000000000 --- a/deploy/postgresql-patroni-ha/templates/scripts.yaml +++ /dev/null @@ -1,69 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: pg-patroni-scripts - labels: - {{- include "postgresql.labels" . | nindent 4 }} -data: - init_container.sh: | - #!/bin/bash - set -o errexit - set -ex - mkdir -p /home/postgres/pgdata/conf - chmod +777 -R /home/postgres/pgdata/conf - cp /home/postgres/conf/postgresql.conf /home/postgres/pgdata/conf - chmod +777 /home/postgres/pgdata/conf/postgresql.conf - generate_patroni_yaml.py: | - #!/usr/bin/env python3 - # -*- coding: utf-8 -*- - import os - import sys - import yaml - def write_file(config, filename, overwrite): - if not overwrite and os.path.exists(filename): - pass - else: - with open(filename, 'w') as f: - f.write(config) - def read_file_lines(file): - ret = [] - for line in file.readlines(): - line = line.strip() - if line and not line.startswith('#'): - ret.append(line) - return ret - def main(filename): - restore_dir = os.environ.get('RESTORE_DATA_DIR', '') - local_config = yaml.safe_load( - os.environ.get('SPILO_CONFIGURATION', os.environ.get('PATRONI_CONFIGURATION', ''))) or {} - if not 'postgresql' in local_config: - local_config['postgresql'] = {} - postgresql = local_config['postgresql'] - postgresql['config_dir'] = '/home/postgres/pgdata/conf' - postgresql['custom_conf'] = '/home/postgres/conf/postgresql.conf' - # TODO add local postgresql.parameters - # add pg_hba.conf - with open('/home/postgres/conf/pg_hba.conf', 'r') as f: - lines = read_file_lines(f) - if lines: - postgresql['pg_hba'] = lines - if restore_dir and os.path.isfile(os.path.join(restore_dir, 'kb_restore.signal')): - if not 'bootstrap' in local_config: - local_config['bootstrap'] = {} - with open('/home/postgres/conf/kb_restore.conf', 'r') as f: - local_config['bootstrap'].update(yaml.safe_load(f)) - write_file(yaml.dump(local_config, default_flow_style=False), filename, True) - if __name__ == '__main__': - main(sys.argv[1]) - setup.sh: | - #!/bin/bash - set -o errexit - set -ex - KB_PRIMARY_POD_NAME_PREFIX=${KB_PRIMARY_POD_NAME%%\.*} - if [ "$KB_PRIMARY_POD_NAME_PREFIX" != "$KB_POD_NAME" ]; then - sleep 3 - fi - python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml - export SPILO_CONFIGURATION=$(cat tmp_patroni.yaml) - # export SCOPE="$KB_CLUSTER_NAME-$KB_CLUSTER_NAME" - exec /launch.sh init \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/values.yaml b/deploy/postgresql-patroni-ha/values.yaml deleted file mode 100644 index 01afa1177..000000000 --- a/deploy/postgresql-patroni-ha/values.yaml +++ /dev/null @@ -1,427 +0,0 @@ -## @section PostgreSQL common parameters -## - -## Bitnami PostgreSQL image version -## ref: https://hub.docker.com/r/bitnami/postgresql/tags/ -## @param image.registry PostgreSQL image registry -## @param image.repository PostgreSQL image repository -## @param image.tag PostgreSQL image tag (immutable tags are recommended) -## @param image.digest PostgreSQL image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag -## @param image.pullPolicy PostgreSQL image pull policy -## @param image.pullSecrets Specify image pull secrets -## @param image.debug Specify if debug values should be set -## -image: - registry: registry.cn-hangzhou.aliyuncs.com - repository: apecloud/spilo - tag: 15.2.0 - digest: "" - ## Specify a imagePullPolicy - ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' - ## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images - ## - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ - ## Example: - ## pullSecrets: - ## - myRegistryKeySecretName - ## - pullSecrets: [ ] - ## Set to true if you would like to see extra information on logs - ## - debug: false -## Authentication parameters -## -auth: - ## @param auth.postgresPassword Password for the "postgres" admin user, leave empty - ## for random generated password. - ## - postgresPassword: - ## @param auth.database Name for a custom database to create - ## - database: "custom_db" -## Audit settings -## @param audit.logHostname Log client hostnames -## @param audit.logConnections Add client log-in operations to the log file -## @param audit.logDisconnections Add client log-outs operations to the log file -## @param audit.pgAuditLog Add operations to log using the pgAudit extension -## @param audit.pgAuditLogCatalog Log catalog using pgAudit -## @param audit.clientMinMessages Message log level to share with the user -## @param audit.logLinePrefix Template for log line prefix (default if not set) -## @param audit.logTimezone Timezone for the log timestamps -## -audit: - logHostname: false - logConnections: false - logDisconnections: false - pgAuditLog: "" - pgAuditLogCatalog: "off" - clientMinMessages: error - logLinePrefix: "" - logTimezone: "" - -## Set PostgreSQL preload extension shared libraries. -## @param postgresqlSharedPreloadLibraries Shared preload libraries (comma-separated list) -## -postgresqlSharedPreloadLibraries: "pg_stat_statements, auto_explain" -## Start PostgreSQL pod(s) without limitations on shm memory. -## By default, docker and containerd (and possibly other container runtimes) limit `/dev/shm` to `64M` -## -shmVolume: - ## @param shmVolume.enabled Enable emptyDir volume for /dev/shm for PostgreSQL pod(s) - ## - enabled: true - ## @param shmVolume.sizeLimit Set this to enable a size limit on the shm tmpfs - ## Note: the size of the tmpfs counts against container's memory limit - ## e.g: - ## sizeLimit: 1Gi - ## - sizeLimit: "" - -## @section PostgreSQL Primary parameters -## -primary: - ## @param primary.name Name of the primary database (eg primary, master, leader, ...) - ## - name: primary - ## configEnabled: true - -## @section Metrics Parameters -metrics: - ## @param metrics.image.registry PostgreSQL Prometheus Exporter image registry - ## @param metrics.image.repository PostgreSQL Prometheus Exporter image repository - ## @param metrics.image.tag PostgreSQL Prometheus Exporter image tag (immutable tags are recommended) - ## @param metrics.image.digest PostgreSQL image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag - ## @param metrics.image.pullPolicy PostgreSQL Prometheus Exporter image pull policy - ## @param metrics.image.pullSecrets Specify image pull secrets - ## - image: - registry: registry.cn-hangzhou.aliyuncs.com - repository: apecloud/postgres-exporter - tag: 0.11.1-debian-11-r66 - digest: "" - pullPolicy: IfNotPresent - ## Optionally specify an array of imagePullSecrets. - ## Secrets must be manually created in the namespace. - ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ - ## Example: - ## pullSecrets: - ## - myRegistryKeySecretName - ## - pullSecrets: [ ] - - ## @param metrics.customMetrics Define additional custom metrics - ## ref: https://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file - ## customMetrics: - ## pg_database: - ## query: "SELECT d.datname AS name, CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT') THEN pg_catalog.pg_database_size(d.datname) ELSE 0 END AS size_bytes FROM pg_catalog.pg_database d where datname not in ('template0', 'template1', 'postgres')" - ## metrics: - ## - name: - ## usage: "LABEL" - ## description: "Name of the database" - ## - size_bytes: - ## usage: "GAUGE" - ## description: "Size of the database in bytes" - ## - customMetrics: - pg_postmaster: - query: "SELECT pg_postmaster_start_time as start_time_seconds from pg_postmaster_start_time()" - master: true - metrics: - - start_time_seconds: - usage: "GAUGE" - description: "Time at which postmaster started" - - pg_stat_user_tables: - query: | - SELECT - current_database() datname, - schemaname, - relname, - seq_scan, - seq_tup_read, - idx_scan, - idx_tup_fetch, - n_tup_ins, - n_tup_upd, - n_tup_del, - n_tup_hot_upd, - n_live_tup, - n_dead_tup, - n_mod_since_analyze, - n_ins_since_vacuum, - COALESCE(last_vacuum, '1970-01-01Z') as last_vacuum, - COALESCE(last_autovacuum, '1970-01-01Z') as last_autovacuum, - COALESCE(last_analyze, '1970-01-01Z') as last_analyze, - COALESCE(last_autoanalyze, '1970-01-01Z') as last_autoanalyze, - vacuum_count, - autovacuum_count, - analyze_count, - autoanalyze_count - FROM - pg_stat_user_tables - metrics: - - datname: - usage: "LABEL" - description: "Name of current database" - - schemaname: - usage: "LABEL" - description: "Name of the schema that this table is in" - - relname: - usage: "LABEL" - description: "Name of this table" - - seq_scan: - usage: "COUNTER" - description: "Number of sequential scans initiated on this table" - - seq_tup_read: - usage: "COUNTER" - description: "Number of live rows fetched by sequential scans" - - idx_scan: - usage: "COUNTER" - description: "Number of index scans initiated on this table" - - idx_tup_fetch: - usage: "COUNTER" - description: "Number of live rows fetched by index scans" - - n_tup_ins: - usage: "COUNTER" - description: "Number of rows inserted" - - n_tup_upd: - usage: "COUNTER" - description: "Number of rows updated" - - n_tup_del: - usage: "COUNTER" - description: "Number of rows deleted" - - n_tup_hot_upd: - usage: "COUNTER" - description: "Number of rows HOT updated (i.e., with no separate index update required)" - - n_live_tup: - usage: "GAUGE" - description: "Estimated number of live rows" - - n_dead_tup: - usage: "GAUGE" - description: "Estimated number of dead rows" - - n_mod_since_analyze: - usage: "GAUGE" - description: "Estimated number of rows changed since last analyze" - - n_ins_since_vacuum: - usage: "GAUGE" - description: "Estimated number of rows inserted since this table was last vacuumed" - - last_vacuum: - usage: "GAUGE" - description: "Last time at which this table was manually vacuumed (not counting VACUUM FULL)" - - last_autovacuum: - usage: "GAUGE" - description: "Last time at which this table was vacuumed by the autovacuum daemon" - - last_analyze: - usage: "GAUGE" - description: "Last time at which this table was manually analyzed" - - last_autoanalyze: - usage: "GAUGE" - description: "Last time at which this table was analyzed by the autovacuum daemon" - - vacuum_count: - usage: "COUNTER" - description: "Number of times this table has been manually vacuumed (not counting VACUUM FULL)" - - autovacuum_count: - usage: "COUNTER" - description: "Number of times this table has been vacuumed by the autovacuum daemon" - - analyze_count: - usage: "COUNTER" - description: "Number of times this table has been manually analyzed" - - autoanalyze_count: - usage: "COUNTER" - description: "Number of times this table has been analyzed by the autovacuum daemon" - - pg_statio_user_tables: - query: | - SELECT - current_database() datname, - schemaname, - relname, - heap_blks_read, - heap_blks_hit, - idx_blks_read, - idx_blks_hit, - toast_blks_read, - toast_blks_hit, - tidx_blks_read, - tidx_blks_hit - FROM - pg_statio_user_tables - metrics: - - datname: - usage: "LABEL" - description: "Name of current database" - - schemaname: - usage: "LABEL" - description: "Name of the schema that this table is in" - - relname: - usage: "LABEL" - description: "Name of this table" - - heap_blks_read: - usage: "COUNTER" - description: "Number of disk blocks read from this table" - - heap_blks_hit: - usage: "COUNTER" - description: "Number of buffer hits in this table" - - idx_blks_read: - usage: "COUNTER" - description: "Number of disk blocks read from all indexes on this table" - - idx_blks_hit: - usage: "COUNTER" - description: "Number of buffer hits in all indexes on this table" - - toast_blks_read: - usage: "COUNTER" - description: "Number of disk blocks read from this table's TOAST table (if any)" - - toast_blks_hit: - usage: "COUNTER" - description: "Number of buffer hits in this table's TOAST table (if any)" - - tidx_blks_read: - usage: "COUNTER" - description: "Number of disk blocks read from this table's TOAST table indexes (if any)" - - tidx_blks_hit: - usage: "COUNTER" - description: "Number of buffer hits in this table's TOAST table indexes (if any)" - - # WARNING: This set of metrics can be very expensive on a busy server as every unique query executed will create an additional time series - pg_stat_statements: - query: | - SELECT - t2.rolname, - t3.datname, - queryid, - plans, - total_plan_time / 1000 as total_plan_time_seconds, - min_plan_time / 1000 as min_plan_time_seconds, - max_plan_time / 1000 as max_plan_time_seconds, - mean_plan_time / 1000 as mean_plan_time_seconds, - stddev_plan_time / 1000 as stddev_plan_time_seconds, - calls, - total_exec_time / 1000 as total_exec_time_seconds, - min_exec_time / 1000 as min_exec_time_seconds, - max_exec_time / 1000 as max_exec_time_seconds, - mean_exec_time / 1000 as mean_exec_time_seconds, - stddev_exec_time / 1000 as stddev_exec_time_seconds, - rows, - shared_blks_hit, - shared_blks_read, - shared_blks_dirtied, - shared_blks_written, - local_blks_hit, - local_blks_read, - local_blks_dirtied, - local_blks_written, - temp_blks_read, - temp_blks_written, - blk_read_time / 1000 as blk_read_time_seconds, - blk_write_time / 1000 as blk_write_time_seconds, - wal_records, - wal_fpi, - wal_bytes - FROM - pg_stat_statements t1 - JOIN - pg_roles t2 - ON (t1.userid=t2.oid) - JOIN - pg_database t3 - ON (t1.dbid=t3.oid) - WHERE t2.rolname != 'rdsadmin' - master: true - metrics: - - rolname: - usage: "LABEL" - description: "Name of user" - - datname: - usage: "LABEL" - description: "Name of database" - - queryid: - usage: "LABEL" - description: "Query ID" - - plans: - usage: "COUNTER" - description: "Number of times the statement was planned" - - total_plan_time_seconds: - usage: "COUNTER" - description: "Total time spent planning the statement" - - min_plan_time_seconds: - usage: "GAUGE" - description: "Minimum time spent planning the statement" - - max_plan_time_seconds: - usage: "GAUGE" - description: "Maximum time spent planning the statement" - - mean_plan_time_seconds: - usage: "GAUGE" - description: "Mean time spent planning the statement" - - stddev_plan_time_seconds: - usage: "GAUGE" - description: "Population standard deviation of time spent planning the statement" - - calls: - usage: "COUNTER" - description: "Number of times executed" - - total_exec_time_seconds: - usage: "COUNTER" - description: "Total time spent in the statement" - - min_exec_time_seconds: - usage: "GAUGE" - description: "Minimum time spent in the statement" - - max_exec_time_seconds: - usage: "GAUGE" - description: "Maximum time spent in the statement" - - mean_exec_time_seconds: - usage: "GAUGE" - description: "Mean time spent in the statement" - - stddev_exec_time_seconds: - usage: "GAUGE" - description: "Population standard deviation of time spent in the statement" - - rows: - usage: "COUNTER" - description: "Total number of rows retrieved or affected by the statement" - - shared_blks_hit: - usage: "COUNTER" - description: "Total number of shared block cache hits by the statement" - - shared_blks_read: - usage: "COUNTER" - description: "Total number of shared blocks read by the statement" - - shared_blks_dirtied: - usage: "COUNTER" - description: "Total number of shared blocks dirtied by the statement" - - shared_blks_written: - usage: "COUNTER" - description: "Total number of shared blocks written by the statement" - - local_blks_hit: - usage: "COUNTER" - description: "Total number of local block cache hits by the statement" - - local_blks_read: - usage: "COUNTER" - description: "Total number of local blocks read by the statement" - - local_blks_dirtied: - usage: "COUNTER" - description: "Total number of local blocks dirtied by the statement" - - local_blks_written: - usage: "COUNTER" - description: "Total number of local blocks written by the statement" - - temp_blks_read: - usage: "COUNTER" - description: "Total number of temp blocks read by the statement" - - temp_blks_written: - usage: "COUNTER" - description: "Total number of temp blocks written by the statement" - - blk_read_time_seconds: - usage: "COUNTER" - description: "Total time the statement spent reading blocks, in milliseconds (if track_io_timing is enabled, otherwise zero)" - - blk_write_time_seconds: - usage: "COUNTER" - description: "Total time the statement spent writing blocks, in milliseconds (if track_io_timing is enabled, otherwise zero)" - - wal_records: - usage: "COUNTER" - description: "Total number of WAL records generated by the statement" - - wal_fpi: - usage: "COUNTER" - description: "Total number of WAL full page images generated by the statement" - - wal_bytes: - usage: "COUNTER" - description: "Total amount of WAL generated by the statement in bytes" -logConfigs: - running: /postgresql/data/log/postgresql-* \ No newline at end of file diff --git a/deploy/postgresql/Chart.yaml b/deploy/postgresql/Chart.yaml index 27d08942a..a2c45dc38 100644 --- a/deploy/postgresql/Chart.yaml +++ b/deploy/postgresql/Chart.yaml @@ -1,12 +1,12 @@ apiVersion: v2 name: postgresql -description: A PostgreSQL cluster definition Helm chart for Kubernetes +description: A PostgreSQL (with Patroni HA) cluster definition Helm chart for Kubernetes type: application version: 0.5.0-alpha.3 -appVersion: "14.7.0" +appVersion: "15.2.0" home: https://kubeblocks.io/ icon: https://github.com/apecloud/kubeblocks/raw/main/img/logo.png diff --git a/deploy/postgresql/config/pg14-config-effect-scope.yaml b/deploy/postgresql/config/pg14-config-effect-scope.yaml index 3aeb21fe8..6e4d69046 100644 --- a/deploy/postgresql/config/pg14-config-effect-scope.yaml +++ b/deploy/postgresql/config/pg14-config-effect-scope.yaml @@ -1,47 +1,22 @@ +# Patroni bootstrap parameters staticParameters: -# - autovacuum_freeze_max_age -# - autovacuum_max_workers -# - autovacuum_multixact_freeze_max_age -# - config_file -# - cron.database_name -# - cron.log_run -# - cron.log_statement -# - cron.max_running_jobs -# - cron.use_background_workers -# - data_directory -# - hba_file -# - huge_pages -# - ident_file -# - ignore_invalid_pages -# - listen_addresses -# - logging_collector -# - max_connections -# - max_files_per_process -# - max_locks_per_transaction -# - max_logical_replication_workers -# - max_pred_locks_per_transaction -# - max_prepared_transactions -# - max_replication_slots -# - max_wal_senders -# - max_worker_processes -# - min_dynamic_shared_memory -# - old_snapshot_threshold -# - pglogical.batch_inserts -# - pglogical.synchronous_commit -# - pglogical.use_spi -# - pg_prewarm.autoprewarm -# - pg_stat_statements.max -# - port -# - postgis.gdal_enabled_drivers -# - recovery_init_sync_method -# - session_preload_libraries -# - shared_buffers -# - shared_preload_libraries -# - superuser_reserved_connections -# - track_activity_query_size -# - track_commit_timestamp -# - unix_socket_directories -# - unix_socket_group -# - unix_socket_permissions -# - wal_buffers -# - wal_compression \ No newline at end of file + - archive_command + - shared_buffers + - logging_collector + - log_destination + - log_directory + - log_filename + - log_file_mode + - log_rotation_age + - log_truncate_on_rotation + - ssl + - ssl_ca_file + - ssl_crl_file + - ssl_cert_file + - ssl_key_file + - shared_preload_libraries + - bg_mon.listen_address + - bg_mon.history_buckets + - pg_stat_statements.track_utility + - extwlist.extensions + - extwlist.custom_path \ No newline at end of file diff --git a/deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl b/deploy/postgresql/scripts/patroni-reload.tpl similarity index 100% rename from deploy/postgresql-patroni-ha/scripts/patroni-reload.tpl rename to deploy/postgresql/scripts/patroni-reload.tpl diff --git a/deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml b/deploy/postgresql/scripts/restart-parameter.yaml similarity index 100% rename from deploy/postgresql-patroni-ha/scripts/restart-parameter.yaml rename to deploy/postgresql/scripts/restart-parameter.yaml diff --git a/deploy/postgresql/templates/backuptool.yaml b/deploy/postgresql/templates/backuptool.yaml index d9b7164c1..90e618194 100644 --- a/deploy/postgresql/templates/backuptool.yaml +++ b/deploy/postgresql/templates/backuptool.yaml @@ -1,7 +1,7 @@ apiVersion: dataprotection.kubeblocks.io/v1alpha1 kind: BackupTool metadata: - name: pg-basebackup + name: postgres-basebackup labels: clusterdefinition.kubeblocks.io/name: postgresql {{- include "postgresql.labels" . | nindent 4 }} @@ -16,30 +16,34 @@ spec: cpu: "1" memory: 128Mi env: + - name: RESTORE_DATA_DIR + value: /home/postgres/pgdata/kb_restore + - name: TMP_DATA_DIR + value: /home/postgres/pgdata/kb_restore/tmp_data + - name: TMP_ARCH_DATA_DIR + value: /home/postgres/pgdata/kb_restore/arch - name: DATA_DIR - value: /postgresql/data + value: /home/postgres/pgdata/pgroot/data physical: restoreCommands: - | + #!/bin/sh set -e - mkdir -p ${DATA_DIR} - res=`ls -A ${DATA_DIR}` - if [ ! -z ${res} ]; then - echo "${DATA_DIR} is not empty! Please make sure that the directory is empty before restoring the backup." - exit 1 - fi - mkdir -p ${DATA_DIR} && mkdir -p ${DATA_DIR}/../arch + # create a new directory for restore + mkdir -p ${RESTORE_DATA_DIR} && rm -rf ${RESTORE_DATA_DIR}/* + cd ${RESTORE_DATA_DIR} + touch kb_restore.sh && touch kb_restore.signal + echo "mkdir -p ${DATA_DIR}/../arch" >> kb_restore.sh + echo "mv -f ${TMP_DATA_DIR}/* ${DATA_DIR}/" >> kb_restore.sh + echo "mv -f ${TMP_ARCH_DATA_DIR}/* ${DATA_DIR}/../arch" >> kb_restore.sh + echo "rm -rf ${RESTORE_DATA_DIR}" >> kb_restore.sh + + # extract the data file to the temporary data directory + mkdir -p ${TMP_DATA_DIR} && mkdir -p ${TMP_ARCH_DATA_DIR} + rm -rf ${TMP_ARCH_DATA_DIR}/* && rm -rf ${TMP_DATA_DIR}/* cd ${BACKUP_DIR}/${BACKUP_NAME} - tar -xvf base.tar.gz -C ${DATA_DIR}/ - tar -xvf pg_wal.tar.gz -C ${DATA_DIR}/../arch - chmod -R 777 ${DATA_DIR}/../arch - cd ${DATA_DIR} - echo "set recovery configuration..." - echo "restore_command = 'cp ${DATA_DIR}/../arch/%f %p'" >> postgresql.auto.conf - echo "recovery_target_timeline = 'latest'" >> postgresql.auto.conf - init_scripts_dir=${DATA_DIR}/../init-scripts - mkdir -p ${init_scripts_dir} - echo "touch ${DATA_DIR}/recovery.signal" > ${init_scripts_dir}/kb_restore.sh + tar -xvf base.tar.gz -C ${TMP_DATA_DIR}/ + tar -xvf pg_wal.tar.gz -C ${TMP_ARCH_DATA_DIR}/ echo "done!" incrementalRestoreCommands: [] logical: @@ -49,5 +53,5 @@ spec: - > set -e; mkdir -p ${BACKUP_DIR}/${BACKUP_NAME}/; - echo ${DB_PASSWORD} | pg_basebackup -Ft -Pv -Xs -z -D ${BACKUP_DIR}/${BACKUP_NAME} -Z5 -h ${DB_HOST} -U ${DB_USER} -W; + echo ${DB_PASSWORD} | pg_basebackup -Ft -Pv -Xs -z -D ${BACKUP_DIR}/${BACKUP_NAME} -Z5 -h ${DB_HOST} -U standby -W; incrementalBackupCommands: [] diff --git a/deploy/postgresql/templates/clusterdefinition.yaml b/deploy/postgresql/templates/clusterdefinition.yaml index 61c05ff01..30642c4d3 100644 --- a/deploy/postgresql/templates/clusterdefinition.yaml +++ b/deploy/postgresql/templates/clusterdefinition.yaml @@ -8,14 +8,24 @@ spec: type: postgresql connectionCredential: username: postgres - password: {{ (include "postgresql.postgresPassword" .) | quote }} + password: "$(RANDOM_PASSWD)" endpoint: "$(SVC_FQDN):$(SVC_PORT_tcp-postgresql)" host: "$(SVC_FQDN)" port: "$(SVC_PORT_tcp-postgresql)" componentDefs: - - name: pg-replication + - name: postgresql workloadType: Replication characterType: postgresql + customLabelSpecs: + - key: apps.kubeblocks.postgres.patroni/scope + value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" + resources: + - gvk: "v1/Pod" + selector: + app.kubernetes.io/managed-by: kubeblocks + - gvk: "apps/v1/StatefulSet" + selector: + app.kubernetes.io/managed-by: kubeblocks probes: roleChangedProbe: failureThreshold: 2 @@ -56,45 +66,43 @@ spec: - name: tcp-postgresql port: 5432 targetPort: tcp-postgresql + - name: http-metrics-postgresql + port: 9187 + targetPort: http-metrics volumeTypes: - name: data type: data podSpec: + serviceAccountName: operator securityContext: - fsGroup: 1001 - containers: - - name: postgresql + runAsUser: 0 + fsGroup: 103 + runAsGroup: 103 + initContainers: + - name: pg-init-container imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} - securityContext: - runAsUser: 0 + command: + - /kb-scripts/init_container.sh volumeMounts: - - name: dshm - mountPath: /dev/shm - name: data - mountPath: /postgresql + mountPath: /home/postgres/pgdata + mode: 0777 - name: postgresql-config - mountPath: /postgresql/conf + mountPath: /home/postgres/conf + mode: 0777 - name: scripts - mountPath: /scripts - ports: - - name: tcp-postgresql - containerPort: 5432 + mountPath: /kb-scripts + mode: 0777 + containers: + - name: postgresql + imagePullPolicy: {{ default .Values.image.pullPolicy "IfNotPresent" }} + securityContext: + runAsUser: 0 command: - - /scripts/setup.sh - livenessProbe: - failureThreshold: 6 - initialDelaySeconds: 30 - periodSeconds: 10 - successThreshold: 1 - timeoutSeconds: 5 - exec: - command: - - /bin/sh - - -c - - exec pg_isready -U {{ default "postgres" | quote }} -h 127.0.0.1 -p 5432 + - /kb-scripts/setup.sh readinessProbe: failureThreshold: 6 - initialDelaySeconds: 5 + initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 5 @@ -106,83 +114,89 @@ spec: - | exec pg_isready -U {{ default "postgres" | quote }} -h 127.0.0.1 -p 5432 [ -f /postgresql/tmp/.initialized ] || [ -f /postgresql/.initialized ] - env: - - name: BITNAMI_DEBUG - value: "false" - - name: POSTGRESQL_PORT_NUMBER - value: "5432" - - name: POSTGRESQL_VOLUME_DIR - value: /postgresql - - name: PGDATA - value: /postgresql/data - - name: PGUSER + volumeMounts: + - name: dshm + mountPath: /dev/shm + - name: data + mountPath: /home/postgres/pgdata + mode: 0777 + - name: postgresql-config + mountPath: /home/postgres/conf + mode: 0777 + - name: scripts + mountPath: /kb-scripts + ports: + - name: tcp-postgresql + containerPort: 5432 + - name: patroni + containerPort: 8008 + env: ## refer https://github.com/zalando/spilo/blob/master/ENVIRONMENT.rst + - name: DCS_ENABLE_KUBERNETES_API + value: "true" + - name: KUBERNETES_USE_CONFIGMAPS + value: "true" + - name: SCOPE + value: "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-patroni" + - name: KUBERNETES_SCOPE_LABEL + value: "apps.kubeblocks.postgres.patroni/scope" + - name: KUBERNETES_ROLE_LABEL + value: "apps.kubeblocks.postgres.patroni/role" + - name: KUBERNETES_LABELS + value: '{"app.kubernetes.io/instance":"$(KB_CLUSTER_NAME)","apps.kubeblocks.io/component-name":"$(KB_COMP_NAME)","app.kubernetes.io/managed-by":"kubeblocks"}' + - name: RESTORE_DATA_DIR + value: /home/postgres/pgdata/kb_restore + - name: SPILO_CONFIGURATION + value: | ## https://github.com/zalando/patroni#yaml-configuration + bootstrap: + initdb: + - auth-host: md5 + - auth-local: trust + - name: ALLOW_NOSSL + value: "true" + - name: PGROOT + value: /home/postgres/pgdata/pgroot + - name: POD_IP + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: status.podIP + - name: POD_NAMESPACE + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: metadata.namespace + - name: PGUSER_SUPERUSER valueFrom: secretKeyRef: name: $(CONN_CREDENTIAL_SECRET_NAME) key: username - - name: PGPASSWORD + - name: PGPASSWORD_SUPERUSER valueFrom: secretKeyRef: name: $(CONN_CREDENTIAL_SECRET_NAME) key: password - # Authentication - - name: POSTGRES_USER + - name: PGUSER_ADMIN + value: superadmin + - name: PGPASSWORD_ADMIN valueFrom: secretKeyRef: name: $(CONN_CREDENTIAL_SECRET_NAME) - key: username - - name: POSTGRES_POSTGRES_PASSWORD + key: password + - name: PGPASSWORD_STANDBY valueFrom: secretKeyRef: name: $(CONN_CREDENTIAL_SECRET_NAME) key: password - - name: POSTGRES_PASSWORD + - name: PGUSER + valueFrom: + secretKeyRef: + name: $(CONN_CREDENTIAL_SECRET_NAME) + key: username + - name: PGPASSWORD valueFrom: secretKeyRef: name: $(CONN_CREDENTIAL_SECRET_NAME) key: password - - name: POSTGRES_DB - value: {{ (include "postgresql.database" .) | quote }} - # Audit - - name: POSTGRESQL_LOG_HOSTNAME - value: {{ .Values.audit.logHostname | quote }} - - name: POSTGRESQL_LOG_CONNECTIONS - value: {{ .Values.audit.logConnections | quote }} - - name: POSTGRESQL_LOG_DISCONNECTIONS - value: {{ .Values.audit.logDisconnections | quote }} - {{- if .Values.audit.logLinePrefix }} - - name: POSTGRESQL_LOG_LINE_PREFIX - value: {{ .Values.audit.logLinePrefix | quote }} - {{- end }} - {{- if .Values.audit.logTimezone }} - - name: POSTGRESQL_LOG_TIMEZONE - value: {{ .Values.audit.logTimezone | quote }} - {{- end }} - {{- if .Values.audit.pgAuditLog }} - - name: POSTGRESQL_PGAUDIT_LOG - value: {{ .Values.audit.pgAuditLog | quote }} - {{- end }} - - name: POSTGRESQL_PGAUDIT_LOG_CATALOG - value: {{ .Values.audit.pgAuditLogCatalog | quote }} - # Others - - name: POSTGRESQL_CLIENT_MIN_MESSAGES - value: {{ .Values.audit.clientMinMessages | quote }} - - name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES - value: {{ .Values.postgresqlSharedPreloadLibraries | quote }} - {{- if .Values.primary.extraEnvVars }} - {{- include "tplvalues.render" (dict "value" .Values.primary.extraEnvVars "context" $) | nindent 12 }} - {{- end }} - {{- if or .Values.primary.extraEnvVarsCM .Values.primary.extraEnvVarsSecret }} - envFrom: - {{- if .Values.primary.extraEnvVarsCM }} - - configMapRef: - name: {{ .Values.primary.extraEnvVarsCM }} - {{- end }} - {{- if .Values.primary.extraEnvVarsSecret }} - - secretRef: - name: {{ .Values.primary.extraEnvVarsSecret }} - {{- end }} - {{- end }} - name: metrics image: {{ .Values.metrics.image.registry | default "docker.io" }}/{{ .Values.metrics.image.repository }}:{{ .Values.metrics.image.tag }} imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }} @@ -242,13 +256,13 @@ spec: {{- end }} systemAccounts: cmdExecutorConfig: - image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ default .Values.image.tag .Chart.AppVersion }} + image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ default .Values.image.tag }} command: - - psql + - psql args: - - -h$(KB_ACCOUNT_ENDPOINT) - - -c - - $(KB_ACCOUNT_STATEMENT) + - -h$(KB_ACCOUNT_ENDPOINT) + - -c + - $(KB_ACCOUNT_STATEMENT) env: - name: PGUSER valueFrom: diff --git a/deploy/postgresql/templates/clusterversion.yaml b/deploy/postgresql/templates/clusterversion.yaml index 797ae6b60..9b611bd20 100644 --- a/deploy/postgresql/templates/clusterversion.yaml +++ b/deploy/postgresql/templates/clusterversion.yaml @@ -7,8 +7,12 @@ metadata: spec: clusterDefinitionRef: postgresql componentVersions: - - componentDefRef: pg-replication + - componentDefRef: postgresql versionsContext: + initContainers: + - name: pg-init-container + image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} containers: - name: postgresql - image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ default .Chart.AppVersion .Values.image.tag }} + image: {{ .Values.image.registry | default "docker.io" }}/{{ .Values.image.repository }}:{{ .Values.image.tag }} +--- diff --git a/deploy/postgresql/templates/configconstraint.yaml b/deploy/postgresql/templates/configconstraint.yaml index ee646442e..d5da741fa 100644 --- a/deploy/postgresql/templates/configconstraint.yaml +++ b/deploy/postgresql/templates/configconstraint.yaml @@ -7,9 +7,9 @@ metadata: {{- include "postgresql.labels" . | nindent 4 }} spec: reloadOptions: - unixSignalTrigger: - signal: SIGHUP - processName: postgres + tplScriptTrigger: + scriptConfigMapRef: patroni-reload-script + namespace: {{ .Release.Namespace }} # top level mysql configuration type cfgSchemaTopLevelName: PGParameter @@ -31,16 +31,17 @@ spec: {{- end }} {{- end}} - ## reload parameters - ## dynamicParameters - {{- if hasKey $cc "dynamicParameters" }} - dynamicParameters: - {{- $params := get $cc "dynamicParameters" }} + ## define immutable parameter list, this feature is not currently supported. + {{- if hasKey $cc "immutableParameters" }} + immutableParameters: + {{- $params := get $cc "immutableParameters" }} {{- range $params }} - {{ . }} {{- end }} - {{- end}} + {{- end}} + + - # configuration file format + # configuration file format formatterConfig: format: properties diff --git a/deploy/postgresql/templates/configmap.yaml b/deploy/postgresql/templates/configmap.yaml index abccb7ad3..f4965bf73 100644 --- a/deploy/postgresql/templates/configmap.yaml +++ b/deploy/postgresql/templates/configmap.yaml @@ -15,4 +15,12 @@ data: host all all 127.0.0.1/32 trust host all all ::1/128 trust host replication all 0.0.0.0/0 md5 - host replication all ::/0 md5 \ No newline at end of file + host replication all ::/0 md5 + kb_restore.conf: | + method: kb_restore_from_backup + kb_restore_from_backup: + command: sh /home/postgres/pgdata/kb_restore/kb_restore.sh + keep_existing_recovery_conf: false + recovery_conf: + restore_command: cp /home/postgres/pgdata/pgroot/arch/%f %p + recovery_target_timeline: latest diff --git a/deploy/postgresql-patroni-ha/templates/patroni-rbac.yaml b/deploy/postgresql/templates/patroni-rbac.yaml similarity index 100% rename from deploy/postgresql-patroni-ha/templates/patroni-rbac.yaml rename to deploy/postgresql/templates/patroni-rbac.yaml diff --git a/deploy/postgresql-patroni-ha/templates/patroni-reload.yaml b/deploy/postgresql/templates/patroni-reload.yaml similarity index 100% rename from deploy/postgresql-patroni-ha/templates/patroni-reload.yaml rename to deploy/postgresql/templates/patroni-reload.yaml diff --git a/deploy/postgresql/templates/scripts.yaml b/deploy/postgresql/templates/scripts.yaml index 81d4f7ac9..45f775fad 100644 --- a/deploy/postgresql/templates/scripts.yaml +++ b/deploy/postgresql/templates/scripts.yaml @@ -5,78 +5,65 @@ metadata: labels: {{- include "postgresql.labels" . | nindent 4 }} data: - post_start.sh: | - #!/bin/sh - export PGPASSWORD=$POSTGRES_POSTGRES_PASSWORD - echo "wait for the database to be accessible, trying to connect." - while : - do - echo "start to connect postgres." - psql -Upostgres -c "SELECT 1;" >/dev/null 2>&1 - if [ $? -eq 0 ]; then - echo "postgres connect success, break now." - if [ -f ${PGDATA}/recovery.signal ]; then - psql -Upostgres -c "select pg_wal_replay_resume();" - fi - if [ -f ${PGDATA}/../init-scripts/restore.sh ]; then - echo "restore.sh exist, remove it." - rm -rf ${PGDATA}/../init-scripts/restore.sh - fi - break - fi - echo "postgres connect fail, sleep and try again." - sleep 1 - done - echo "create extension pg_stat_statements." - psql -Upostgres -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;" 2>&1 - if [ $? -eq 0 ];then - echo "create extension pg_stat_statements success." - else - exit 1 - fi + init_container.sh: | + #!/bin/bash + set -o errexit + set -ex + mkdir -p /home/postgres/pgdata/conf + chmod +777 -R /home/postgres/pgdata/conf + cp /home/postgres/conf/postgresql.conf /home/postgres/pgdata/conf + chmod +777 /home/postgres/pgdata/conf/postgresql.conf + generate_patroni_yaml.py: | + #!/usr/bin/env python3 + # -*- coding: utf-8 -*- + import os + import sys + import yaml + def write_file(config, filename, overwrite): + if not overwrite and os.path.exists(filename): + pass + else: + with open(filename, 'w') as f: + f.write(config) + def read_file_lines(file): + ret = [] + for line in file.readlines(): + line = line.strip() + if line and not line.startswith('#'): + ret.append(line) + return ret + def main(filename): + restore_dir = os.environ.get('RESTORE_DATA_DIR', '') + local_config = yaml.safe_load( + os.environ.get('SPILO_CONFIGURATION', os.environ.get('PATRONI_CONFIGURATION', ''))) or {} + if not 'postgresql' in local_config: + local_config['postgresql'] = {} + postgresql = local_config['postgresql'] + postgresql['config_dir'] = '/home/postgres/pgdata/conf' + postgresql['custom_conf'] = '/home/postgres/conf/postgresql.conf' + # TODO add local postgresql.parameters + # add pg_hba.conf + with open('/home/postgres/conf/pg_hba.conf', 'r') as f: + lines = read_file_lines(f) + if lines: + postgresql['pg_hba'] = lines + if restore_dir and os.path.isfile(os.path.join(restore_dir, 'kb_restore.signal')): + if not 'bootstrap' in local_config: + local_config['bootstrap'] = {} + with open('/home/postgres/conf/kb_restore.conf', 'r') as f: + local_config['bootstrap'].update(yaml.safe_load(f)) + write_file(yaml.dump(local_config, default_flow_style=False), filename, True) + if __name__ == '__main__': + main(sys.argv[1]) setup.sh: | #!/bin/bash set -o errexit - set -o nounset - KB_PRIMARY_POD_NAME_PREFIX="${KB_PRIMARY_POD_NAME%%\.*}" - # debug - echo "KB_PRIMARY_POD_NAME=$KB_PRIMARY_POD_NAME" - echo "KB_PRIMARY_POD_NAME_PREFIX=$KB_PRIMARY_POD_NAME_PREFIX" - echo "KB_POD_NAME=$KB_POD_NAME" + set -ex + KB_PRIMARY_POD_NAME_PREFIX=${KB_PRIMARY_POD_NAME%%\.*} if [ "$KB_PRIMARY_POD_NAME_PREFIX" != "$KB_POD_NAME" ]; then - export POSTGRES_REPLICATION_MODE=slave - # TODO: use replicator instead - export POSTGRES_REPLICATION_USER=$POSTGRES_USER - export POSTGRES_REPLICATION_PASSWORD=$POSTGRES_PASSWORD - export POSTGRES_CLUSTER_APP_NAME=my-application - export POSTGRES_MASTER_HOST=$KB_PRIMARY_POD_NAME - export POSTGRES_MASTER_PORT_NUMBER="5432" - . /opt/bitnami/scripts/libos.sh - . /opt/bitnami/scripts/libpostgresql.sh - . /opt/bitnami/scripts/postgresql-env.sh - # add permission to daemon user - chmod a+w "$POSTGRESQL_VOLUME_DIR" - # Ensure 'daemon' user exists when running as 'root' - am_i_root && ensure_user_exists "$POSTGRESQL_DAEMON_USER" --group "$POSTGRESQL_DAEMON_GROUP" - if [ ! -d ${PGDATA} ]; then - # pg_basebackup - postgresql_slave_init_db - fi - else - # bitnami scripts will execute all the *.sh files in the /docker-entrypoint-initdb.d directory after setup - cp /scripts/post_start.sh /docker-entrypoint-initdb.d + sleep 3 fi - if [ -f ${PGDATA}/../init-scripts/kb_restore.sh ]; then - # add recovery.signal/standby.signal to trigger recovery - cp ${PGDATA}/../init-scripts/kb_restore.sh /docker-entrypoint-preinitdb.d - fi - /opt/bitnami/scripts/postgresql/entrypoint.sh /opt/bitnami/scripts/postgresql/run.sh - backup-log-collector.sh: | - #!/bin/bash - set -o errexit - set -o nounset - LOG_START_TIME=$(pg_waldump $(ls -tr /postgresql/data/pg_wal/ | grep '[[:digit:]]$'|head -n 1) --rmgr=Transaction 2>/dev/null |head -n 1|awk -F ' COMMIT ' '{print $2}'|awk -F ';' '{print $1}') - LOG_STOP_TIME=$(pg_waldump $(ls -t /postgresql/data/pg_wal/ | grep '[[:digit:]]$'|head -n 1) --rmgr=Transaction 2>/dev/null |tail -n 1|awk -F ' COMMIT ' '{print $2}'|awk -F ';' '{print $1}') - LOG_START_TIME=$(date -d "$LOG_START_TIME" -u '+%Y-%m-%dT%H:%M:%SZ') - LOG_STOP_TIME=$(date -d "$LOG_STOP_TIME" -u '+%Y-%m-%dT%H:%M:%SZ') - printf "{\"startTime\": \"$LOG_START_TIME\" ,\"stopTime\": \"$LOG_STOP_TIME\"}" \ No newline at end of file + python3 /kb-scripts/generate_patroni_yaml.py tmp_patroni.yaml + export SPILO_CONFIGURATION=$(cat tmp_patroni.yaml) + # export SCOPE="$KB_CLUSTER_NAME-$KB_CLUSTER_NAME" + exec /launch.sh init \ No newline at end of file diff --git a/deploy/postgresql/values.yaml b/deploy/postgresql/values.yaml index 1a8cc25a7..33cf5582a 100644 --- a/deploy/postgresql/values.yaml +++ b/deploy/postgresql/values.yaml @@ -1,8 +1,5 @@ ## @section PostgreSQL common parameters -## -## Bitnami PostgreSQL image version -## ref: https://hub.docker.com/r/bitnami/postgresql/tags/ ## @param image.registry PostgreSQL image registry ## @param image.repository PostgreSQL image repository ## @param image.tag PostgreSQL image tag (immutable tags are recommended) @@ -13,9 +10,9 @@ ## image: registry: registry.cn-hangzhou.aliyuncs.com - repository: apecloud/postgresql - tag: - + repository: apecloud/spilo + tag: 15.2.0 + digest: "" ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images @@ -67,7 +64,7 @@ audit: ## postgresqlSharedPreloadLibraries: "pg_stat_statements, auto_explain" ## Start PostgreSQL pod(s) without limitations on shm memory. -## By default docker and containerd (and possibly other container runtimes) limit `/dev/shm` to `64M` +## By default, docker and containerd (and possibly other container runtimes) limit `/dev/shm` to `64M` ## shmVolume: ## @param shmVolume.enabled Enable emptyDir volume for /dev/shm for PostgreSQL pod(s) @@ -101,6 +98,7 @@ metrics: registry: registry.cn-hangzhou.aliyuncs.com repository: apecloud/postgres-exporter tag: 0.11.1-debian-11-r66 + digest: "" pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. From 9f97ab14013aff101800598ff2c0c61d03fc408e Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Tue, 11 Apr 2023 10:45:50 +0800 Subject: [PATCH 71/80] fix: failed to start for the replication workload (#2482) (#2489) --- internal/controller/plan/template_wrapper.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/internal/controller/plan/template_wrapper.go b/internal/controller/plan/template_wrapper.go index 65ff48a53..174908b93 100644 --- a/internal/controller/plan/template_wrapper.go +++ b/internal/controller/plan/template_wrapper.go @@ -99,6 +99,7 @@ func (wrapper *renderWrapper) renderConfigTemplate(task *intctrltypes.ReconcileT return err } if !enableRerender { + wrapper.addVolumeMountMeta(configSpec.ComponentTemplateSpec, cmName) continue } @@ -111,7 +112,6 @@ func (wrapper *renderWrapper) renderConfigTemplate(task *intctrltypes.ReconcileT return err } updateCMConfigSpecLabels(cm, configSpec) - if err := wrapper.addRenderedObject(configSpec.ComponentTemplateSpec, cm, scheme); err != nil { return err } @@ -127,6 +127,7 @@ func (wrapper *renderWrapper) renderScriptTemplate(task *intctrltypes.ReconcileT Name: cmName, Namespace: wrapper.cluster.Namespace, }, generics.ToGVK(&corev1.ConfigMap{})) != nil { + wrapper.addVolumeMountMeta(templateSpec, cmName) continue } @@ -150,12 +151,14 @@ func (wrapper *renderWrapper) addRenderedObject(templateSpec appsv1alpha1.Compon } cfgcore.SetParametersUpdateSource(cm, constant.ReconfigureManagerSource) + wrapper.renderedObjs = append(wrapper.renderedObjs, cm) + wrapper.addVolumeMountMeta(templateSpec, cm.Name) + return nil +} - cmName := cm.Name +func (wrapper *renderWrapper) addVolumeMountMeta(templateSpec appsv1alpha1.ComponentTemplateSpec, cmName string) { wrapper.volumes[cmName] = templateSpec - wrapper.renderedObjs = append(wrapper.renderedObjs, cm) wrapper.templateAnnotations[cfgcore.GenerateTPLUniqLabelKeyWithConfig(templateSpec.Name)] = cmName - return nil } func updateCMConfigSpecLabels(cm *corev1.ConfigMap, configSpec appsv1alpha1.ComponentConfigSpec) { From b07b0891621d69a55eab6c9d4b242afa9d95d99a Mon Sep 17 00:00:00 2001 From: dingben <1162833047@qq.com> Date: Tue, 11 Apr 2023 13:24:25 +0800 Subject: [PATCH 72/80] feat: kbcli support show cluster's label (#2474) --- internal/cli/cluster/cluster.go | 1 + internal/cli/cluster/printer.go | 44 +++++++++++++++++++++------- internal/cli/cluster/printer_test.go | 20 ++++++++++--- internal/cli/cluster/types.go | 1 + internal/cli/cmd/cluster/list.go | 6 +++- internal/cli/util/util.go | 9 ++++++ 6 files changed, 66 insertions(+), 15 deletions(-) diff --git a/internal/cli/cluster/cluster.go b/internal/cli/cluster/cluster.go index 49fd9cab1..1128ac0eb 100644 --- a/internal/cli/cluster/cluster.go +++ b/internal/cli/cluster/cluster.go @@ -204,6 +204,7 @@ func (o *ClusterObjects) GetClusterInfo() *ClusterInfo { CreatedTime: util.TimeFormat(&c.CreationTimestamp), InternalEP: types.None, ExternalEP: types.None, + Labels: util.CombineLabels(c.Labels), } if o.ClusterDef == nil { diff --git a/internal/cli/cluster/printer.go b/internal/cli/cluster/printer.go index 754373b10..16cb8ccc9 100644 --- a/internal/cli/cluster/printer.go +++ b/internal/cli/cluster/printer.go @@ -35,26 +35,39 @@ const ( PrintEvents PrintType = "events" ) +type PrinterOptions struct { + ShowLabels bool +} + type tblInfo struct { header []interface{} - addRow func(tbl *printer.TablePrinter, objs *ClusterObjects) + addRow func(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) getOptions GetOptions } var mapTblInfo = map[PrintType]tblInfo{ PrintClusters: { header: []interface{}{"NAME", "NAMESPACE", "CLUSTER-DEFINITION", "VERSION", "TERMINATION-POLICY", "STATUS", "CREATED-TIME"}, - addRow: func(tbl *printer.TablePrinter, objs *ClusterObjects) { + addRow: func(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) { c := objs.GetClusterInfo() - tbl.AddRow(c.Name, c.Namespace, c.ClusterDefinition, c.ClusterVersion, c.TerminationPolicy, c.Status, c.CreatedTime) + info := []interface{}{c.Name, c.Namespace, c.ClusterDefinition, c.ClusterVersion, c.TerminationPolicy, c.Status, c.CreatedTime} + if opt.ShowLabels { + info = append(info, c.Labels) + } + + tbl.AddRow(info...) }, getOptions: GetOptions{}, }, PrintWide: { header: []interface{}{"NAME", "NAMESPACE", "CLUSTER-DEFINITION", "VERSION", "TERMINATION-POLICY", "STATUS", "INTERNAL-ENDPOINTS", "EXTERNAL-ENDPOINTS", "CREATED-TIME"}, - addRow: func(tbl *printer.TablePrinter, objs *ClusterObjects) { + addRow: func(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) { c := objs.GetClusterInfo() - tbl.AddRow(c.Name, c.Namespace, c.ClusterDefinition, c.ClusterVersion, c.TerminationPolicy, c.Status, c.InternalEP, c.ExternalEP, c.CreatedTime) + info := []interface{}{c.Name, c.Namespace, c.ClusterDefinition, c.ClusterVersion, c.TerminationPolicy, c.Status, c.InternalEP, c.ExternalEP, c.CreatedTime} + if opt.ShowLabels { + info = append(info, c.Labels) + } + tbl.AddRow(info...) }, getOptions: GetOptions{WithClusterDef: true, WithService: true, WithPod: true}, }, @@ -78,18 +91,29 @@ var mapTblInfo = map[PrintType]tblInfo{ // Printer prints cluster info type Printer struct { tbl *printer.TablePrinter + opt *PrinterOptions tblInfo } -func NewPrinter(out io.Writer, printType PrintType) *Printer { +func NewPrinter(out io.Writer, printType PrintType, opt *PrinterOptions) *Printer { p := &Printer{tbl: printer.NewTablePrinter(out)} p.tblInfo = mapTblInfo[printType] + + if opt == nil { + opt = &PrinterOptions{} + } + p.opt = opt + + if opt.ShowLabels { + p.tblInfo.header = append(p.tblInfo.header, "LABELS") + } + p.tbl.SetHeader(p.tblInfo.header...) return p } func (p *Printer) AddRow(objs *ClusterObjects) { - p.addRow(p.tbl, objs) + p.addRow(p.tbl, objs, p.opt) } func (p *Printer) Print() { @@ -100,14 +124,14 @@ func (p *Printer) GetterOptions() GetOptions { return p.getOptions } -func AddComponentRow(tbl *printer.TablePrinter, objs *ClusterObjects) { +func AddComponentRow(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) { components := objs.GetComponentInfo() for _, c := range components { tbl.AddRow(c.Name, c.NameSpace, c.Cluster, c.Type, c.Image) } } -func AddInstanceRow(tbl *printer.TablePrinter, objs *ClusterObjects) { +func AddInstanceRow(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) { instances := objs.GetInstanceInfo() for _, instance := range instances { tbl.AddRow(instance.Name, instance.Namespace, instance.Cluster, instance.Component, @@ -117,7 +141,7 @@ func AddInstanceRow(tbl *printer.TablePrinter, objs *ClusterObjects) { } } -func AddEventRow(tbl *printer.TablePrinter, objs *ClusterObjects) { +func AddEventRow(tbl *printer.TablePrinter, objs *ClusterObjects, opt *PrinterOptions) { events := util.SortEventsByLastTimestamp(objs.Events, "") for _, event := range *events { e := event.(*corev1.Event) diff --git a/internal/cli/cluster/printer_test.go b/internal/cli/cluster/printer_test.go index 98048a0ce..7cf3d84c7 100644 --- a/internal/cli/cluster/printer_test.go +++ b/internal/cli/cluster/printer_test.go @@ -33,20 +33,32 @@ var _ = Describe("printer", func() { return nil } + printerWithLabels := &PrinterOptions{ + ShowLabels: true, + } + It("print cluster info", func() { - Expect(printObjs(NewPrinter(os.Stdout, PrintClusters), objs)).Should(Succeed()) + Expect(printObjs(NewPrinter(os.Stdout, PrintClusters, nil), objs)).Should(Succeed()) + }) + + It("print cluster info with label", func() { + Expect(printObjs(NewPrinter(os.Stdout, PrintClusters, printerWithLabels), objs)).Should(Succeed()) }) It("print cluster wide info", func() { - Expect(printObjs(NewPrinter(os.Stdout, PrintWide), objs)).Should(Succeed()) + Expect(printObjs(NewPrinter(os.Stdout, PrintWide, nil), objs)).Should(Succeed()) + }) + + It("print cluster wide info with label", func() { + Expect(printObjs(NewPrinter(os.Stdout, PrintWide, printerWithLabels), objs)).Should(Succeed()) }) It("print component info", func() { - Expect(printObjs(NewPrinter(os.Stdout, PrintComponents), objs)).Should(Succeed()) + Expect(printObjs(NewPrinter(os.Stdout, PrintComponents, nil), objs)).Should(Succeed()) }) It("print instance info", func() { - Expect(printObjs(NewPrinter(os.Stdout, PrintInstances), objs)).Should(Succeed()) + Expect(printObjs(NewPrinter(os.Stdout, PrintInstances, nil), objs)).Should(Succeed()) }) }) }) diff --git a/internal/cli/cluster/types.go b/internal/cli/cluster/types.go index 2bec57d1e..9c9f17dc3 100644 --- a/internal/cli/cluster/types.go +++ b/internal/cli/cluster/types.go @@ -46,6 +46,7 @@ type ClusterInfo struct { InternalEP string `json:"internalEP,omitempty"` ExternalEP string `json:"externalEP,omitempty"` CreatedTime string `json:"age,omitempty"` + Labels string `json:"labels,omitempty"` } type ComponentInfo struct { diff --git a/internal/cli/cmd/cluster/list.go b/internal/cli/cmd/cluster/list.go index 9d86bd3eb..f005148c6 100644 --- a/internal/cli/cmd/cluster/list.go +++ b/internal/cli/cmd/cluster/list.go @@ -181,7 +181,11 @@ func run(o *list.ListOptions, printType cluster.PrintType) error { return err } - p := cluster.NewPrinter(o.IOStreams.Out, printType) + opt := &cluster.PrinterOptions{ + ShowLabels: o.ShowLabels, + } + + p := cluster.NewPrinter(o.IOStreams.Out, printType, opt) for _, info := range infos { if err = addRow(dynamic, client, info.Namespace, info.Name, p); err != nil { return err diff --git a/internal/cli/util/util.go b/internal/cli/util/util.go index e11df119c..ea1533671 100644 --- a/internal/cli/util/util.go +++ b/internal/cli/util/util.go @@ -658,3 +658,12 @@ func GetK8SProvider(client kubernetes.Interface) (K8sProvider, error) { func BuildAddonReleaseName(addon string) string { return fmt.Sprintf("%s-%s", types.AddonReleasePrefix, addon) } + +// CombineLabels combines labels into a string +func CombineLabels(labels map[string]string) string { + var labelStr string + for k, v := range labels { + labelStr += fmt.Sprintf("%s=%s,", k, v) + } + return strings.TrimSuffix(labelStr, ",") +} From ae62dd6100a914794dbf562196ca8d009e62a647 Mon Sep 17 00:00:00 2001 From: "L.DongMing" Date: Tue, 11 Apr 2023 13:25:17 +0800 Subject: [PATCH 73/80] feat: delete clusters and uninstall kubeblocks when playground destroy (#2457) --- internal/cli/cmd/accounts/delete.go | 2 +- internal/cli/cmd/cli.go | 1 - internal/cli/cmd/kubeblocks/install.go | 14 +- internal/cli/cmd/kubeblocks/uninstall.go | 39 +-- internal/cli/cmd/kubeblocks/uninstall_test.go | 6 +- internal/cli/cmd/playground/base.go | 24 +- internal/cli/cmd/playground/destroy.go | 259 ++++++++++++++++-- internal/cli/cmd/playground/init.go | 89 +++--- internal/cli/cmd/playground/kubeconfig.go | 8 + internal/cli/cmd/playground/types.go | 7 + internal/cli/cmd/playground/util.go | 59 +++- 11 files changed, 400 insertions(+), 108 deletions(-) diff --git a/internal/cli/cmd/accounts/delete.go b/internal/cli/cmd/accounts/delete.go index fed84d3e7..1e13bb110 100644 --- a/internal/cli/cmd/accounts/delete.go +++ b/internal/cli/cmd/accounts/delete.go @@ -41,7 +41,7 @@ func (o *DeleteUserOptions) AddFlags(cmd *cobra.Command) { cmd.Flags().StringVarP(&o.info.UserName, "username", "u", "", "Required. Specify the name of user") } -func (o DeleteUserOptions) Validate(args []string) error { +func (o *DeleteUserOptions) Validate(args []string) error { if err := o.AccountBaseOptions.Validate(args); err != nil { return err } diff --git a/internal/cli/cmd/cli.go b/internal/cli/cmd/cli.go index 914337911..394fca51f 100644 --- a/internal/cli/cmd/cli.go +++ b/internal/cli/cmd/cli.go @@ -120,7 +120,6 @@ func initConfig() { viper.SetConfigType("yaml") viper.AddConfigPath(fmt.Sprintf("/etc/%s/", cliName)) viper.AddConfigPath(fmt.Sprintf("$HOME/.%s/", cliName)) - viper.AddConfigPath(".") viper.AutomaticEnv() // read in environment variables that match viper.SetEnvPrefix(cliName) diff --git a/internal/cli/cmd/kubeblocks/install.go b/internal/cli/cmd/kubeblocks/install.go index 8e35b1ef6..06f09ecef 100644 --- a/internal/cli/cmd/kubeblocks/install.go +++ b/internal/cli/cmd/kubeblocks/install.go @@ -185,7 +185,7 @@ func (o *InstallOptions) Install() error { o.ValueOpts.Values = append(o.ValueOpts.Values, fmt.Sprintf(kMonitorParam, o.Monitor)) // add helm repo - spinner := printer.Spinner(o.Out, "%-40s", "Add and update repo "+types.KubeBlocksRepoName) + spinner := printer.Spinner(o.Out, "%-50s", "Add and update repo "+types.KubeBlocksRepoName) defer spinner(false) // Add repo, if exists, will update it if err = helm.AddRepo(&repo.Entry{Name: types.KubeBlocksRepoName, URL: util.GetHelmChartRepoURL()}); err != nil { @@ -194,7 +194,7 @@ func (o *InstallOptions) Install() error { spinner(true) // install KubeBlocks chart - spinner = printer.Spinner(o.Out, "%-40s", "Install KubeBlocks "+o.Version) + spinner = printer.Spinner(o.Out, "%-50s", "Install KubeBlocks "+o.Version) defer spinner(false) if err = o.installChart(); err != nil { return err @@ -268,13 +268,13 @@ func (o *InstallOptions) waitAddonsEnabled() error { } okMsg := func(msg string) string { - return fmt.Sprintf("%-40s %s\n", msg, printer.BoldGreen("OK")) + return fmt.Sprintf("%-50s %s\n", msg, printer.BoldGreen("OK")) } failMsg := func(msg string) string { - return fmt.Sprintf("%-40s %s\n", msg, printer.BoldRed("FAIL")) + return fmt.Sprintf("%-50s %s\n", msg, printer.BoldRed("FAIL")) } suffixMsg := func(msg string) string { - return fmt.Sprintf(" %-40s", msg) + return fmt.Sprintf(" %-50s", msg) } // create spinner @@ -336,7 +336,7 @@ func (o *InstallOptions) waitAddonsEnabled() error { } // timeout to wait for all auto-install addons to be enabled - s.FinalMSG = fmt.Sprintf("%-40s %s\n", msg, printer.BoldRed("TIMEOUT")) + s.FinalMSG = fmt.Sprintf("%-50s %s\n", msg, printer.BoldRed("TIMEOUT")) s.Stop() return nil } @@ -470,7 +470,7 @@ func (o *InstallOptions) createVolumeSnapshotClass() error { options.BaseOptions.IOStreams = o.IOStreams options.BaseOptions.Quiet = true - spinner := printer.Spinner(o.Out, "%-40s", "Configure VolumeSnapshotClass") + spinner := printer.Spinner(o.Out, "%-50s", "Configure VolumeSnapshotClass") defer spinner(false) if err := options.Complete(); err != nil { diff --git a/internal/cli/cmd/kubeblocks/uninstall.go b/internal/cli/cmd/kubeblocks/uninstall.go index 927742d2a..e702f3572 100644 --- a/internal/cli/cmd/kubeblocks/uninstall.go +++ b/internal/cli/cmd/kubeblocks/uninstall.go @@ -54,24 +54,25 @@ var ( kbcli kubeblocks uninstall`) ) -type uninstallOptions struct { - factory cmdutil.Factory +type UninstallOptions struct { + Factory cmdutil.Factory Options - // autoApprove if true, skip interactive approval - autoApprove bool + // AutoApprove if true, skip interactive approval + AutoApprove bool removePVs bool removePVCs bool - removeNamespace bool + RemoveNamespace bool addons []*extensionsv1alpha1.Addon + Quiet bool } func newUninstallCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *cobra.Command { - o := &uninstallOptions{ + o := &UninstallOptions{ Options: Options{ IOStreams: streams, }, - factory: f, + Factory: f, } cmd := &cobra.Command{ Use: "uninstall", @@ -80,21 +81,21 @@ func newUninstallCmd(f cmdutil.Factory, streams genericclioptions.IOStreams) *co Example: uninstallExample, Run: func(cmd *cobra.Command, args []string) { util.CheckErr(o.Complete(f, cmd)) - util.CheckErr(o.preCheck()) - util.CheckErr(o.uninstall()) + util.CheckErr(o.PreCheck()) + util.CheckErr(o.Uninstall()) }, } - cmd.Flags().BoolVar(&o.autoApprove, "auto-approve", false, "Skip interactive approval before uninstalling KubeBlocks") + cmd.Flags().BoolVar(&o.AutoApprove, "auto-approve", false, "Skip interactive approval before uninstalling KubeBlocks") cmd.Flags().BoolVar(&o.removePVs, "remove-pvs", false, "Remove PersistentVolume or not") cmd.Flags().BoolVar(&o.removePVCs, "remove-pvcs", false, "Remove PersistentVolumeClaim or not") - cmd.Flags().BoolVar(&o.removeNamespace, "remove-namespace", false, "Remove default created \"kb-system\" namespace or not") + cmd.Flags().BoolVar(&o.RemoveNamespace, "remove-namespace", false, "Remove default created \"kb-system\" namespace or not") return cmd } -func (o *uninstallOptions) preCheck() error { +func (o *UninstallOptions) PreCheck() error { // wait user to confirm - if !o.autoApprove { + if !o.AutoApprove { printer.Warning(o.Out, "uninstall will remove all KubeBlocks resources.\n") if err := confirmUninstall(o.In); err != nil { return err @@ -143,8 +144,10 @@ func (o *uninstallOptions) preCheck() error { kbNamespace, err := util.GetKubeBlocksNamespace(o.Client) if err != nil { printer.Warning(o.Out, "failed to locate KubeBlocks meta, will clean up all KubeBlocks resources.\n") - fmt.Fprintf(o.Out, "to find out the namespace where KubeBlocks is installed, please use:\n\t'kbcli kubeblocks status'\n") - fmt.Fprintf(o.Out, "to uninstall KubeBlocks completely, please use:\n\t`kbcli kubeblocks uninstall -n `\n") + if !o.Quiet { + fmt.Fprintf(o.Out, "to find out the namespace where KubeBlocks is installed, please use:\n\t'kbcli kubeblocks status'\n") + fmt.Fprintf(o.Out, "to uninstall KubeBlocks completely, please use:\n\t`kbcli kubeblocks uninstall -n `\n") + } } else if o.Namespace != kbNamespace { o.Namespace = kbNamespace fmt.Fprintf(o.Out, "Uninstall KubeBlocks in namespace \"%s\"\n", kbNamespace) @@ -152,7 +155,7 @@ func (o *uninstallOptions) preCheck() error { return nil } -func (o *uninstallOptions) uninstall() error { +func (o *UninstallOptions) Uninstall() error { printSpinner := func(spinner func(result bool), err error) { if err == nil || apierrors.IsNotFound(err) || strings.Contains(err.Error(), "release: not found") { @@ -216,7 +219,7 @@ func (o *uninstallOptions) uninstall() error { } // delete namespace if it is default namespace - if o.Namespace == types.DefaultNamespace && o.removeNamespace { + if o.Namespace == types.DefaultNamespace && o.RemoveNamespace { printSpinner(newSpinner("Remove namespace "+types.DefaultNamespace), deleteNamespace(o.Client, types.DefaultNamespace)) } @@ -226,7 +229,7 @@ func (o *uninstallOptions) uninstall() error { } // uninstallAddons uninstall all KubeBlocks addons -func (o *uninstallOptions) uninstallAddons() error { +func (o *UninstallOptions) uninstallAddons() error { var ( allErrs []error stop bool diff --git a/internal/cli/cmd/kubeblocks/uninstall_test.go b/internal/cli/cmd/kubeblocks/uninstall_test.go index a25cbfcf6..947f652c2 100644 --- a/internal/cli/cmd/kubeblocks/uninstall_test.go +++ b/internal/cli/cmd/kubeblocks/uninstall_test.go @@ -62,7 +62,7 @@ var _ = Describe("kubeblocks uninstall", func() { }) It("run uninstall", func() { - o := uninstallOptions{ + o := UninstallOptions{ Options: Options{ IOStreams: streams, HelmCfg: helm.NewFakeConfig(namespace), @@ -70,8 +70,8 @@ var _ = Describe("kubeblocks uninstall", func() { Client: testing.FakeClientSet(), Dynamic: testing.FakeDynamicClient(testing.FakeVolumeSnapshotClass()), }, - autoApprove: true, + AutoApprove: true, } - Expect(o.uninstall()).Should(Succeed()) + Expect(o.Uninstall()).Should(Succeed()) }) }) diff --git a/internal/cli/cmd/playground/base.go b/internal/cli/cmd/playground/base.go index 23dbe876e..dfaea297e 100644 --- a/internal/cli/cmd/playground/base.go +++ b/internal/cli/cmd/playground/base.go @@ -18,29 +18,37 @@ package playground import ( "fmt" + "os" + "path/filepath" "time" cp "github.com/apecloud/kubeblocks/internal/cli/cloudprovider" ) type baseOptions struct { - startTime time.Time - prevCluster *cp.K8sClusterInfo + startTime time.Time + // prevCluster is the previous cluster info + prevCluster *cp.K8sClusterInfo + // kubeConfigPath is the tmp kubeconfig path that will be used when int and destroy + kubeConfigPath string + // stateFilePath is the state file path stateFilePath string } func (o *baseOptions) validate() error { - var err error - - if err = initPlaygroundDir(); err != nil { + playgroundDir, err := initPlaygroundDir() + if err != nil { return err } - o.stateFilePath, err = stateFilePath() - if err != nil { - return err + o.kubeConfigPath = filepath.Join(playgroundDir, "kubeconfig") + if _, err = os.Stat(o.kubeConfigPath); err == nil { + if err = os.Remove(o.kubeConfigPath); err != nil { + return err + } } + o.stateFilePath = filepath.Join(playgroundDir, stateFileName) o.prevCluster, err = readClusterInfoFromFile(o.stateFilePath) if err != nil { return err diff --git a/internal/cli/cmd/playground/destroy.go b/internal/cli/cmd/playground/destroy.go index 249953ab8..72870977e 100644 --- a/internal/cli/cmd/playground/destroy.go +++ b/internal/cli/cmd/playground/destroy.go @@ -17,18 +17,32 @@ limitations under the License. package playground import ( + "context" "fmt" + "os" "strings" "time" "github.com/spf13/cobra" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + apitypes "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" + "k8s.io/klog/v2" cmdutil "k8s.io/kubectl/pkg/cmd/util" "k8s.io/kubectl/pkg/util/templates" + appsv1alpha1 "github.com/apecloud/kubeblocks/apis/apps/v1alpha1" cp "github.com/apecloud/kubeblocks/internal/cli/cloudprovider" + "github.com/apecloud/kubeblocks/internal/cli/cmd/kubeblocks" "github.com/apecloud/kubeblocks/internal/cli/printer" + "github.com/apecloud/kubeblocks/internal/cli/types" "github.com/apecloud/kubeblocks/internal/cli/util" + "github.com/apecloud/kubeblocks/internal/cli/util/helm" "github.com/apecloud/kubeblocks/internal/cli/util/prompt" ) @@ -84,7 +98,7 @@ func (o *destroyOptions) destroy() error { // destroyLocal destroy local k3d cluster that will destroy all resources func (o *destroyOptions) destroyLocal() error { provider := cp.NewLocalCloudProvider(o.Out, o.ErrOut) - spinner := printer.Spinner(o.Out, "Delete playground k3d cluster %s", o.prevCluster.ClusterName) + spinner := printer.Spinner(o.Out, "%-50s", "Delete playground k3d cluster "+o.prevCluster.ClusterName) defer spinner(false) if err := provider.DeleteK8sCluster(o.prevCluster); err != nil { if !strings.Contains(err.Error(), "no cluster found") && @@ -105,36 +119,17 @@ func (o *destroyOptions) destroyLocal() error { // namespace that will destroy all resources created by KubeBlocks, avoid to leave // some resources func (o *destroyOptions) destroyCloud() error { - cpPath, err := cloudProviderRepoDir() - if err != nil { - return err - } - - provider, err := cp.New(o.prevCluster.CloudProvider, cpPath, o.Out, o.ErrOut) - if err != nil { - return err - } + var err error // start to destroy cluster - printer.Warning(o.Out, `This action will directly delete the kubernetes cluster, which may - result in some residual resources, such as Volume, please confirm and manually - clean up related resources after this action. - - In order to minimize resource residue, you can use the following commands - to clean up the clusters and uninstall KubeBlocks before this action. - - # list all clusters created by KubeBlocks - kbcli cluster list -A - - # delete clusters - kbcli cluster delete - - # uninstall KubeBlocks and remove PVC and PV - kbcli kubeblocks uninstall --remove-pvcs --remove-pvs + printer.Warning(o.Out, `This action will uninstall KubeBlocks and delete the kubernetes cluster, + there may be residual resources, please confirm and manually clean up related + resources after this action. `) - fmt.Fprintf(o.Out, "Do you really want to destroy the kubernetes cluster %s?\n This is no undo. Only 'yes' will be accepted to confirm.\n\n", o.prevCluster.ClusterName) + fmt.Fprintf(o.Out, "Do you really want to destroy the kubernetes cluster %s?\n%s\n\n This is no undo. Only 'yes' will be accepted to confirm.\n\n", + o.prevCluster.ClusterName, o.prevCluster.String()) // confirm to destroy entered, _ := prompt.NewPrompt("Enter a value:", nil, o.In).Run() @@ -145,16 +140,42 @@ func (o *destroyOptions) destroyCloud() error { o.startTime = time.Now() + // for cloud provider, we should delete all clusters created by KubeBlocks first, + // uninstall KubeBlocks and remove the KubeBlocks namespace, then destroy the + // playground cluster, avoid to leave some resources. + // delete all clusters created by KubeBlocks, MUST BE VERY CAUTIOUS, use the right + // kubeconfig and context, otherwise, it will delete the wrong cluster. + if err = o.deleteClustersAndUninstallKB(); err != nil { + if strings.Contains(err.Error(), kubeClusterUnreachableErr.Error()) { + printer.Warning(o.Out, err.Error()) + } else { + return err + } + } + + // destroy playground kubernetes cluster + cpPath, err := cloudProviderRepoDir() + if err != nil { + return err + } + + provider, err := cp.New(o.prevCluster.CloudProvider, cpPath, o.Out, o.ErrOut) + if err != nil { + return err + } + fmt.Fprintf(o.Out, "Destroy %s %s cluster %s...\n", o.prevCluster.CloudProvider, cp.K8sService(o.prevCluster.CloudProvider), o.prevCluster.ClusterName) if err = provider.DeleteK8sCluster(o.prevCluster); err != nil { return err } + // remove the cluster kubeconfig from the use default kubeconfig if err = o.removeKubeConfig(); err != nil { return err } + // at last, remove the state file if err = o.removeStateFile(); err != nil { return err } @@ -163,13 +184,195 @@ func (o *destroyOptions) destroyCloud() error { return nil } +func (o *destroyOptions) deleteClustersAndUninstallKB() error { + var err error + + if o.prevCluster.KubeConfig == "" { + fmt.Fprintf(o.Out, "No kubeconfig found for kubernetes cluster %s in %s \n", + o.prevCluster.ClusterName, o.stateFilePath) + return nil + } + + // write kubeconfig content to a temporary file and use it + if err = writeAndUseKubeConfig(o.prevCluster.KubeConfig, o.kubeConfigPath, o.Out); err != nil { + return err + } + + client, dynamic, err := getKubeClient() + if err != nil { + return err + } + + // delete all clusters created by KubeBlocks + if err = o.deleteClusters(dynamic); err != nil { + return err + } + + // uninstall KubeBlocks and remove namespace created by KubeBlocks + return o.uninstallKubeBlocks(client, dynamic) +} + +// delete all clusters created by KubeBlocks +func (o *destroyOptions) deleteClusters(dynamic dynamic.Interface) error { + var err error + ctx := context.Background() + + // get all clusters in all namespaces + getClusters := func() (*unstructured.UnstructuredList, error) { + return dynamic.Resource(types.ClusterGVR()).Namespace(metav1.NamespaceAll). + List(context.Background(), metav1.ListOptions{}) + } + + // get all clusters and check if satisfy the checkFn + checkClusters := func(checkFn func(cluster *appsv1alpha1.Cluster) bool) (bool, error) { + res := true + clusters, err := getClusters() + if err != nil { + return false, err + } + for _, item := range clusters.Items { + cluster := &appsv1alpha1.Cluster{} + if err = runtime.DefaultUnstructuredConverter.FromUnstructured(item.Object, cluster); err != nil { + return false, err + } + if !checkFn(cluster) { + res = false + break + } + } + return res, nil + } + + // delete all clusters + deleteClusters := func(clusters *unstructured.UnstructuredList) error { + for _, cluster := range clusters.Items { + if err = dynamic.Resource(types.ClusterGVR()).Namespace(cluster.GetNamespace()). + Delete(ctx, cluster.GetName(), *metav1.NewDeleteOptions(0)); err != nil { + return err + } + } + return nil + } + + spinner := printer.Spinner(o.Out, fmt.Sprintf("%-50s", "Delete clusters created by KubeBlocks")) + defer spinner(false) + + // get all clusters + clusters, err := getClusters() + if clusters == nil || len(clusters.Items) == 0 { + spinner(true) + return nil + } + + checkWipeOut := false + // set all cluster termination policy to WipeOut to delete all resources, otherwise + // the cluster will be deleted but the resources will be left + for _, item := range clusters.Items { + cluster := &appsv1alpha1.Cluster{} + if err = runtime.DefaultUnstructuredConverter.FromUnstructured(item.Object, cluster); err != nil { + return err + } + if cluster.Spec.TerminationPolicy == appsv1alpha1.WipeOut { + continue + } + + // terminate policy is not WipeOut, set it to WipeOut + klog.V(1).Infof("Set cluster %s termination policy to WipeOut", cluster.Name) + if _, err = dynamic.Resource(types.ClusterGVR()).Namespace(cluster.Namespace).Patch(ctx, cluster.Name, apitypes.JSONPatchType, + []byte(fmt.Sprintf("[{\"op\": \"replace\", \"path\": \"/spec/terminationPolicy\", \"value\": \"%s\" }]", + appsv1alpha1.WipeOut)), metav1.PatchOptions{}); err != nil { + return err + } + + // set some cluster termination policy to WipeOut, need to check again + checkWipeOut = true + } + + // check all clusters termination policy is WipeOut + if checkWipeOut { + if err = wait.PollImmediate(5*time.Second, 5*time.Minute, func() (bool, error) { + return checkClusters(func(cluster *appsv1alpha1.Cluster) bool { + if cluster.Spec.TerminationPolicy != appsv1alpha1.WipeOut { + klog.V(1).Infof("Cluster %s termination policy is %s", cluster.Name, cluster.Spec.TerminationPolicy) + } + return cluster.Spec.TerminationPolicy == appsv1alpha1.WipeOut + }) + }); err != nil { + return err + } + } + + // delete all clusters + if err = deleteClusters(clusters); err != nil { + return err + } + + // check and wait all clusters are deleted + if err = wait.PollImmediate(5*time.Second, 5*time.Minute, func() (bool, error) { + return checkClusters(func(cluster *appsv1alpha1.Cluster) bool { + // always return false if any cluster is not deleted + klog.V(1).Infof("Cluster %s is not deleted", cluster.Name) + return false + }) + }); err != nil { + return err + } + + spinner(true) + return nil +} + +func (o *destroyOptions) uninstallKubeBlocks(client kubernetes.Interface, dynamic dynamic.Interface) error { + var err error + uninstall := kubeblocks.UninstallOptions{ + Options: kubeblocks.Options{ + IOStreams: o.IOStreams, + Client: client, + Dynamic: dynamic, + }, + AutoApprove: true, + RemoveNamespace: true, + Quiet: true, + } + + uninstall.HelmCfg = helm.NewConfig("", o.kubeConfigPath, "", klog.V(1).Enabled()) + if err = uninstall.PreCheck(); err != nil { + return err + } + if err = uninstall.Uninstall(); err != nil { + return err + } + return nil +} + func (o *destroyOptions) removeKubeConfig() error { - spinner := printer.Spinner(o.Out, "Remove kubeconfig from %s", defaultKubeConfigPath) + spinner := printer.Spinner(o.Out, "%-50s", "Remove kubeconfig from "+defaultKubeConfigPath) defer spinner(false) if err := kubeConfigRemove(o.prevCluster.KubeConfig, defaultKubeConfigPath); err != nil { - return err + if os.IsNotExist(err) { + spinner(true) + return nil + } else { + return err + } } spinner(true) + + clusterContext, err := kubeConfigCurrentContext(o.prevCluster.KubeConfig) + if err != nil { + return err + } + + // check if current context in kubeconfig is deleted, if yes, notify user to set current context + currentContext, err := kubeConfigCurrentContextFromFile(defaultKubeConfigPath) + if err != nil { + return err + } + + // current context is deleted, notify user to set current context like kubectl + if currentContext == clusterContext { + printer.Warning(o.Out, "this removed your active context, use \"kubectl config use-context\" to select a different one\n") + } return nil } diff --git a/internal/cli/cmd/playground/init.go b/internal/cli/cmd/playground/init.go index 47867099b..b050070c1 100644 --- a/internal/cli/cmd/playground/init.go +++ b/internal/cli/cmd/playground/init.go @@ -19,12 +19,14 @@ package playground import ( "fmt" "os" + "path/filepath" "strings" "time" "github.com/pkg/errors" "github.com/spf13/cobra" "golang.org/x/exp/slices" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/rand" "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/cli-runtime/pkg/genericclioptions" @@ -150,24 +152,29 @@ func (o *initOptions) local() error { } } - if err = writeClusterInfoToFile(o.stateFilePath, clusterInfo); err != nil { + if err = writeClusterInfo(o.stateFilePath, clusterInfo); err != nil { return errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", o.stateFilePath, clusterInfo) } // create a local kubernetes cluster (k3d cluster) to deploy KubeBlocks - spinner := printer.Spinner(o.Out, "%-40s", "Create k3d cluster: "+clusterInfo.ClusterName) + spinner := printer.Spinner(o.Out, "%-50s", "Create k3d cluster: "+clusterInfo.ClusterName) defer spinner(false) if err = provider.CreateK8sCluster(clusterInfo); err != nil { return errors.Wrap(err, "failed to set up k3d cluster") } spinner(true) - if err = o.setKubeConfig(provider); err != nil { + clusterInfo, err = o.writeStateFile(provider) + if err != nil { + return err + } + + if err = o.setKubeConfig(clusterInfo); err != nil { return err } // install KubeBlocks and create a database cluster - return o.installKBAndCluster(clusterInfo.ClusterName) + return o.installKBAndCluster(clusterInfo) } // bootstraps a playground in the remote cloud @@ -198,7 +205,7 @@ func (o *initOptions) cloud() error { } fmt.Fprintf(o.Out, "\nWrite cluster info to state file %s\n", o.stateFilePath) - if err := writeClusterInfoToFile(o.stateFilePath, clusterInfo); err != nil { + if err := writeClusterInfo(o.stateFilePath, clusterInfo); err != nil { return errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", o.stateFilePath, clusterInfo) } @@ -229,12 +236,19 @@ func (o *initOptions) cloud() error { } printer.PrintBlankLine(o.Out) - // write cluster kubeconfig to local kubeconfig file and switch current context to it - if err = o.setKubeConfig(provider); err != nil { + // write cluster info to state file and get new cluster info with kubeconfig + clusterInfo, err = o.writeStateFile(provider) + if err != nil { return err } - return o.installKBAndCluster(clusterInfo.ClusterName) + // write cluster kubeconfig to default kubeconfig file and switch current context to it + if err = o.setKubeConfig(clusterInfo); err != nil { + return err + } + + // install KubeBlocks and create a database cluster + return o.installKBAndCluster(clusterInfo) } // confirmToContinue confirms to continue init or not if there is an existed kubernetes cluster @@ -276,31 +290,45 @@ func printGuide() { fmt.Fprintf(os.Stdout, guideStr, kbClusterName) } -func (o *initOptions) setKubeConfig(provider cp.Interface) error { - // get kubernetes cluster info with the kubeconfig and write it to state file +// writeStateFile writes cluster info to state file and return the new cluster info with kubeconfig +func (o *initOptions) writeStateFile(provider cp.Interface) (*cp.K8sClusterInfo, error) { clusterInfo, err := provider.GetClusterInfo() if err != nil { - return err + return nil, err } if clusterInfo.KubeConfig == "" { - return errors.New("failed to get kubernetes cluster kubeconfig") + return nil, errors.New("failed to get kubernetes cluster kubeconfig") } - if err = writeClusterInfoToFile(o.stateFilePath, clusterInfo); err != nil { - return errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", + if err = writeClusterInfo(o.stateFilePath, clusterInfo); err != nil { + return nil, errors.Wrapf(err, "failed to write kubernetes cluster info to state file %s:\n %v", o.stateFilePath, clusterInfo) } + return clusterInfo, nil +} - // merge created kubernetes cluster kubeconfig to ~/.kube/config and set it as default - spinner := printer.Spinner(o.Out, "Write kubeconfig to %s", defaultKubeConfigPath) +// merge created kubernetes cluster kubeconfig to ~/.kube/config and set it as default +func (o *initOptions) setKubeConfig(info *cp.K8sClusterInfo) error { + spinner := printer.Spinner(o.Out, "%-50s", "Merge kubeconfig to "+defaultKubeConfigPath) defer spinner(false) - if err = kubeConfigWrite(clusterInfo.KubeConfig, defaultKubeConfigPath, + + // check if the default kubeconfig file exists, if not, create it + if _, err := os.Stat(defaultKubeConfigPath); os.IsNotExist(err) { + if err = os.MkdirAll(filepath.Dir(defaultKubeConfigPath), 0755); err != nil { + return errors.Wrapf(err, "failed to create directory %s", filepath.Dir(defaultKubeConfigPath)) + } + if err = os.WriteFile(defaultKubeConfigPath, []byte{}, 0644); err != nil { + return errors.Wrapf(err, "failed to create file %s", defaultKubeConfigPath) + } + } + + if err := kubeConfigWrite(info.KubeConfig, defaultKubeConfigPath, writeKubeConfigOptions{UpdateExisting: true, UpdateCurrentContext: true}); err != nil { - return errors.Wrapf(err, "failed to write cluster %s kubeconfig", clusterInfo.ClusterName) + return errors.Wrapf(err, "failed to write cluster %s kubeconfig", info.ClusterName) } spinner(true) - currentContext, err := kubeConfigCurrentContext(clusterInfo.KubeConfig) - spinner = printer.Spinner(o.Out, "Switch current context to %s", currentContext) + currentContext, err := kubeConfigCurrentContext(info.KubeConfig) + spinner = printer.Spinner(o.Out, "%-50s", "Switch current context to "+currentContext) defer spinner(false) if err != nil { return err @@ -310,7 +338,7 @@ func (o *initOptions) setKubeConfig(provider cp.Interface) error { return nil } -func (o *initOptions) installKBAndCluster(k8sClusterName string) error { +func (o *initOptions) installKBAndCluster(info *cp.K8sClusterInfo) error { var err error // when the kubernetes cluster is not ready, the runtime will output the error @@ -321,36 +349,33 @@ func (o *initOptions) installKBAndCluster(k8sClusterName string) error { } } - // playground always use the default kubeconfig at ~/.kube/config - if err = util.SetKubeConfig(defaultKubeConfigPath); err != nil { + // write kubeconfig content to a temporary file and use it + if err = writeAndUseKubeConfig(info.KubeConfig, o.kubeConfigPath, o.Out); err != nil { return err } // create helm config - o.helmCfg = helm.NewConfig("", defaultKubeConfigPath, "", klog.V(1).Enabled()) + o.helmCfg = helm.NewConfig("", o.kubeConfigPath, "", klog.V(1).Enabled()) - // Install KubeBlocks - if err = o.installKubeBlocks(k8sClusterName); err != nil { + // install KubeBlocks + if err = o.installKubeBlocks(info.ClusterName); err != nil { return errors.Wrap(err, "failed to install KubeBlocks") } - // Install database cluster + // install database cluster clusterInfo := "ClusterDefinition: " + o.clusterDef if o.clusterVersion != "" { clusterInfo += ", ClusterVersion: " + o.clusterVersion } spinner := printer.Spinner(o.Out, "Create cluster %s (%s)", kbClusterName, clusterInfo) defer spinner(false) - if err = o.createCluster(); err != nil { + if err = o.createCluster(); err != nil && !apierrors.IsAlreadyExists(err) { return errors.Wrapf(err, "failed to create cluster %s", kbClusterName) } spinner(true) - // Print guide information fmt.Fprintf(os.Stdout, "\nKubeBlocks playground init SUCCESSFULLY!\n\n") - if k8sClusterName != "" { - fmt.Fprintf(os.Stdout, "Kubernetes cluster \"%s\" has been created.\n", k8sClusterName) - } + fmt.Fprintf(os.Stdout, "Kubernetes cluster \"%s\" has been created.\n", info.ClusterName) fmt.Fprintf(os.Stdout, "Cluster \"%s\" has been created.\n", kbClusterName) // output elapsed time diff --git a/internal/cli/cmd/playground/kubeconfig.go b/internal/cli/cmd/playground/kubeconfig.go index f12dde2e5..91d67a1c4 100644 --- a/internal/cli/cmd/playground/kubeconfig.go +++ b/internal/cli/cmd/playground/kubeconfig.go @@ -219,3 +219,11 @@ func kubeConfigCurrentContext(kubeConfigStr string) (string, error) { } return kubeConfig.CurrentContext, nil } + +func kubeConfigCurrentContextFromFile(kubeConfigPath string) (string, error) { + kubeConfig, err := clientcmd.LoadFromFile(kubeConfigPath) + if err != nil { + return "", fmt.Errorf("failed to load kubeconfig: %w", err) + } + return kubeConfig.CurrentContext, nil +} diff --git a/internal/cli/cmd/playground/types.go b/internal/cli/cmd/playground/types.go index 95e01ae0b..90913fe24 100644 --- a/internal/cli/cmd/playground/types.go +++ b/internal/cli/cmd/playground/types.go @@ -17,6 +17,8 @@ limitations under the License. package playground import ( + "github.com/pkg/errors" + "github.com/apecloud/kubeblocks/internal/cli/cloudprovider" "github.com/apecloud/kubeblocks/internal/cli/util" ) @@ -47,6 +49,11 @@ var ( defaultKubeConfigPath = util.ConfigPath("config") ) +// errors +var ( + kubeClusterUnreachableErr = errors.New("Kubernetes cluster unreachable") +) + var guideStr = ` 1. Basic commands for cluster: diff --git a/internal/cli/cmd/playground/util.go b/internal/cli/cmd/playground/util.go index f81333c0b..d60993e87 100644 --- a/internal/cli/cmd/playground/util.go +++ b/internal/cli/cmd/playground/util.go @@ -19,13 +19,18 @@ package playground import ( "encoding/json" "fmt" + "io" "os" "path/filepath" "strings" "github.com/pkg/errors" + "k8s.io/cli-runtime/pkg/genericclioptions" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes" cp "github.com/apecloud/kubeblocks/internal/cli/cloudprovider" + "github.com/apecloud/kubeblocks/internal/cli/printer" "github.com/apecloud/kubeblocks/internal/cli/util" "github.com/apecloud/kubeblocks/version" ) @@ -52,20 +57,20 @@ func cloudProviderRepoDir() (string, error) { return filepath.Join(dir, cpDir), err } -func initPlaygroundDir() error { +func initPlaygroundDir() (string, error) { dir, err := playgroundDir() if err != nil { - return err + return "", err } if _, err = os.Stat(dir); err != nil && os.IsNotExist(err) { - return os.MkdirAll(dir, 0750) + err = os.MkdirAll(dir, 0750) } - return nil + return dir, err } -// writeClusterInfoToFile writes the cluster info to a state file -func writeClusterInfoToFile(path string, info *cp.K8sClusterInfo) error { +// writeClusterInfo writes the cluster info to a state file +func writeClusterInfo(path string, info *cp.K8sClusterInfo) error { f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0640) if err != nil { return err @@ -111,10 +116,44 @@ func readClusterInfoFromFile(path string) (*cp.K8sClusterInfo, error) { return &info, nil } -func stateFilePath() (string, error) { - dir, err := playgroundDir() +func writeAndUseKubeConfig(kubeConfig string, kubeConfigPath string, out io.Writer) error { + spinner := printer.Spinner(out, fmt.Sprintf("%-50s", "Write kubeconfig to "+kubeConfigPath)) + defer spinner(false) + if err := kubeConfigWrite(kubeConfig, kubeConfigPath, writeKubeConfigOptions{ + UpdateExisting: true, + UpdateCurrentContext: true, + OverwriteExisting: true}); err != nil { + return err + } + + // use the new kubeconfig file + if err := util.SetKubeConfig(kubeConfigPath); err != nil { + return err + } + + spinner(true) + return nil +} + +// getKubeClient returns a kubernetes dynamic client and check if the cluster is reachable +func getKubeClient() (kubernetes.Interface, dynamic.Interface, error) { + f := util.NewFactory() + client, err := f.KubernetesClientSet() + errMsg := kubeClusterUnreachableErr.Error() + if err == genericclioptions.ErrEmptyConfig { + return nil, nil, kubeClusterUnreachableErr + } if err != nil { - return "", err + return nil, nil, errors.Wrap(err, errMsg) + } + + if _, err = client.ServerVersion(); err != nil { + return nil, nil, errors.Wrap(err, errMsg) + } + + dynamic, err := f.DynamicClient() + if err != nil { + return nil, nil, errors.Wrap(err, errMsg) } - return filepath.Join(dir, stateFileName), nil + return client, dynamic, nil } From b0afaaa10b013941f34a3a093eed2a55932ed416 Mon Sep 17 00:00:00 2001 From: shaojiang Date: Tue, 11 Apr 2023 14:48:20 +0800 Subject: [PATCH 74/80] fix: csi-hostpath-driver helm upgrade failed (#2508) --- deploy/csi-hostpath-driver/templates/statefulset.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy/csi-hostpath-driver/templates/statefulset.yaml b/deploy/csi-hostpath-driver/templates/statefulset.yaml index b9ffcfe19..c3d8284c0 100644 --- a/deploy/csi-hostpath-driver/templates/statefulset.yaml +++ b/deploy/csi-hostpath-driver/templates/statefulset.yaml @@ -12,11 +12,11 @@ spec: replicas: 1 selector: matchLabels: - {{ include "csi-hostpath-driver.labels" . | nindent 6 }} + {{ include "csi-hostpath-driver.selectorLabels" . | nindent 6 }} template: metadata: labels: - {{ include "csi-hostpath-driver.labels" . | nindent 8 }} + {{ include "csi-hostpath-driver.selectorLabels" . | nindent 8 }} spec: serviceAccountName: {{ include "csi-hostpath-driver.serviceAccountName" . }} {{ with .Values.imagePullSecrets }} From 395cc1901d111d8534279cedbbfb7c8bdfdd6796 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Tue, 11 Apr 2023 16:01:09 +0800 Subject: [PATCH 75/80] chore: add .github/utils/feature_triage.sh to auto. generate feature issues triage markdown table contents (#2504) --- .github/utils/feature_triage.sh | 47 +++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 .github/utils/feature_triage.sh diff --git a/.github/utils/feature_triage.sh b/.github/utils/feature_triage.sh new file mode 100755 index 000000000..41044eebd --- /dev/null +++ b/.github/utils/feature_triage.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash + +set -o errexit +set -o nounset +set -o pipefail + +REMOTE_URL=$(git config --get remote.origin.url) +OWNER=$(dirname ${REMOTE_URL} | awk -F ":" '{print $2}') +REPO=$(basename -s .git ${REMOTE_URL}) +MILESTONE_ID=${MILESTONE_ID:-5} + +# GH list issues API ref: https://docs.github.com/en/rest/issues/issues?apiVersion=2022-11-28#list-repository-issues +ISSUE_LIST=$(gh api \ + --header 'Accept: application/vnd.github+json' \ + --method GET \ + /repos/${OWNER}/${REPO}/issues \ + -F per_page=100 \ + -f milestone=${MILESTONE_ID} \ + -f labels=kind/feature \ + -f state=all) + +ROWS=$(echo ${ISSUE_LIST}| jq -r '. | sort_by(.state,.number)| .[].number') + + +printf "%s | %s | %s | %s | %s | %s\n" "Feature Title" "Assignees" "Issue State" "Code PR Merge Status" "Feature Doc. Status" "Extra Notes" +echo "---|---|---|---|---|---" +for ROW in $ROWS +do + ISSUE_ID=$(echo $ROW | awk -F "," '{print $1}') + # GH get issue API ref: https://docs.github.com/en/rest/issues/issues?apiVersion=2022-11-28#get-an-issue + ISSUE_BODY=$(gh api \ + --header 'Accept: application/vnd.github+json' \ + --method GET \ + /repos/${OWNER}/${REPO}/issues/${ISSUE_ID}) + URL=$(echo $ISSUE_BODY| jq -r '.url') + TITLE=$(echo $ISSUE_BODY| jq -r '.title') + ASSIGNEES=$(echo $ISSUE_BODY| jq -r '.assignees[]?.login') + ASSIGNEES_PRINTABLE= + for ASSIGNEE in $ASSIGNEES + do + ASSIGNEES_PRINTABLE="${ASSIGNEES_PRINTABLE},${ASSIGNEE}" + done + ASSIGNEES_PRINTABLE=${ASSIGNEES_PRINTABLE#,} + STATE=$(echo $ISSUE_BODY| jq -r '.state') + PR=$(echo $ISSUE_BODY| jq -r '.pull_request?.url') + printf "[%s](%s) #%s | %s | %s | | | \n" "$TITLE" $URL $ISSUE_ID "$ASSIGNEES_PRINTABLE" "$STATE" +done \ No newline at end of file From 9e38f4a55140f54766ec716847df31eaa4dd9e57 Mon Sep 17 00:00:00 2001 From: kubeJocker <102039539+kubeJocker@users.noreply.github.com> Date: Tue, 11 Apr 2023 17:04:49 +0800 Subject: [PATCH 76/80] test: improve test coverage to above 75% for internal/preflight (#2506) --- internal/cli/cmd/kubeblocks/preflight.go | 2 +- internal/preflight/analyze_test.go | 30 +----- internal/preflight/collect_test.go | 52 +++------- internal/preflight/testing/fake.go | 124 +++++++++++++++++++++++ internal/preflight/testing/fake_test.go | 43 ++++++++ internal/preflight/testing/suite_test.go | 29 ++++++ 6 files changed, 215 insertions(+), 65 deletions(-) create mode 100644 internal/preflight/testing/fake.go create mode 100644 internal/preflight/testing/fake_test.go create mode 100644 internal/preflight/testing/suite_test.go diff --git a/internal/cli/cmd/kubeblocks/preflight.go b/internal/cli/cmd/kubeblocks/preflight.go index bdd58fce6..a6452a1e1 100644 --- a/internal/cli/cmd/kubeblocks/preflight.go +++ b/internal/cli/cmd/kubeblocks/preflight.go @@ -195,7 +195,7 @@ func (p *PreflightOptions) run() error { fmt.Print(cursor.Hide()) defer fmt.Print(cursor.Show()) } - // set progress chain + // set progress chan progressCh := make(chan interface{}) defer close(progressCh) // make sure we shut down progress collection goroutines if an error occurs diff --git a/internal/preflight/analyze_test.go b/internal/preflight/analyze_test.go index 1e48f2fc6..4dda0200f 100644 --- a/internal/preflight/analyze_test.go +++ b/internal/preflight/analyze_test.go @@ -25,6 +25,7 @@ import ( troubleshoot "github.com/replicatedhq/troubleshoot/pkg/apis/troubleshoot/v1beta2" preflightv1beta2 "github.com/apecloud/kubeblocks/externalapis/preflight/v1beta2" + preflightTesting "github.com/apecloud/kubeblocks/internal/preflight/testing" ) var _ = Describe("analyze_test", func() { @@ -35,37 +36,12 @@ var _ = Describe("analyze_test", func() { kbAnalyzers []*preflightv1beta2.ExtendAnalyze hostAnalyzers []*troubleshoot.HostAnalyze kbhHostAnalyzers []*preflightv1beta2.ExtendHostAnalyze - clusterVersion = ` -{ - "info": { - "major": "1", - "minor": "23", - "gitVersion": "v1.23.15", - "gitCommit": "b84cb8ab29366daa1bba65bc67f54de2f6c34848", - "gitTreeState": "clean", - "buildDate": "2022-12-08T10:42:57Z", - "goVersion": "go1.17.13", - "compiler": "gc", - "platform": "linux/arm64" - }, - "string": "v1.23.15" -}` ) BeforeEach(func() { ctx = context.TODO() - allCollectedData = map[string][]byte{"cluster-info/cluster_version.json": []byte(clusterVersion)} - analyzers = []*troubleshoot.Analyze{ - {ClusterVersion: &troubleshoot.ClusterVersion{ - AnalyzeMeta: troubleshoot.AnalyzeMeta{ - CheckName: "ClusterVersionCheck", - }, - Outcomes: []*troubleshoot.Outcome{ - { - Pass: &troubleshoot.SingleOutcome{ - Message: "version is ok.", - }}}}}, - } + allCollectedData = preflightTesting.FakeCollectedData() + analyzers = preflightTesting.FakeAnalyzers() kbAnalyzers = []*preflightv1beta2.ExtendAnalyze{{}} hostAnalyzers = []*troubleshoot.HostAnalyze{{}} kbhHostAnalyzers = []*preflightv1beta2.ExtendHostAnalyze{{}} diff --git a/internal/preflight/collect_test.go b/internal/preflight/collect_test.go index f8c3c5a5e..cd1ac9717 100644 --- a/internal/preflight/collect_test.go +++ b/internal/preflight/collect_test.go @@ -28,7 +28,6 @@ import ( corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/util/yaml" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/resource" "k8s.io/client-go/kubernetes/scheme" @@ -38,16 +37,19 @@ import ( preflightv1beta2 "github.com/apecloud/kubeblocks/externalapis/preflight/v1beta2" "github.com/apecloud/kubeblocks/internal/cli/testing" "github.com/apecloud/kubeblocks/internal/cli/types" + preflightTesting "github.com/apecloud/kubeblocks/internal/preflight/testing" ) var _ = Describe("collect_test", func() { var ( - timeOut = 10 * time.Second - namespace = "test" - clusterName = "test" - tf *cmdtesting.TestFactory - cluster = testing.FakeCluster(clusterName, namespace) - pods = testing.FakePods(3, namespace, clusterName) + timeOut = 10 * time.Second + namespace = "test" + clusterName = "test" + tf *cmdtesting.TestFactory + cluster = testing.FakeCluster(clusterName, namespace) + pods = testing.FakePods(3, namespace, clusterName) + preflight *preflightv1beta2.Preflight + hostPreflight *preflightv1beta2.HostPreflight ) BeforeEach(func() { @@ -78,6 +80,9 @@ var _ = Describe("collect_test", func() { tf.Client = tf.UnstructuredClient tf.FakeDynamicClient = testing.FakeDynamicClient(cluster, testing.FakeClusterDef(), testing.FakeClusterVersion()) + + preflight = preflightTesting.FakeKbPreflight() + hostPreflight = preflightTesting.FakeKbHostPreflight() }) AfterEach(func() { @@ -85,55 +90,28 @@ var _ = Describe("collect_test", func() { }) It("CollectPreflight test, and expect success ", func() { - hostByte := ` -apiVersion: troubleshoot.sh/v1beta2 -kind: HostPreflight -metadata: - name: hostCheckTest -spec: - collectors: - - cpu: {} - extendCollectors: - - hostUtility : - collectorName: helmCheck - utilityName: helm -` - hostSpec := new(preflightv1beta2.HostPreflight) Eventually(func(g Gomega) { - g.Expect(yaml.Unmarshal([]byte(hostByte), hostSpec)).Should(Succeed()) progressCh := make(chan interface{}) go func() { for { g.Expect(<-progressCh).NotTo(BeNil()) } }() - results, err := CollectPreflight(context.TODO(), nil, hostSpec, progressCh) + results, err := CollectPreflight(context.TODO(), preflight, hostPreflight, progressCh) g.Expect(err).NotTo(HaveOccurred()) - g.Expect(len(results)).Should(Equal(1)) + g.Expect(len(results)).Should(BeNumerically(">=", 3)) }).WithTimeout(timeOut).Should(Succeed()) }) It("CollectHostData Test, and expect success", func() { - hostByte := ` -apiVersion: troubleshoot.sh/v1beta2 -kind: HostPreflight -metadata: -name: cpu -spec: -collectors: - - cpu: {} -analyzers: -` - hostSpec := new(preflightv1beta2.HostPreflight) Eventually(func(g Gomega) { - g.Expect(yaml.Unmarshal([]byte(hostByte), hostSpec)).Should(Succeed()) progressCh := make(chan interface{}) go func() { for { g.Expect(<-progressCh).NotTo(BeNil()) } }() - results, err := CollectHostData(context.TODO(), hostSpec, progressCh) + results, err := CollectHostData(context.TODO(), hostPreflight, progressCh) g.Expect(err).NotTo(HaveOccurred()) _, ok := (*results).(KBHostCollectResult) g.Expect(ok).Should(BeTrue()) diff --git a/internal/preflight/testing/fake.go b/internal/preflight/testing/fake.go new file mode 100644 index 000000000..f74af46cc --- /dev/null +++ b/internal/preflight/testing/fake.go @@ -0,0 +1,124 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testing + +import ( + troubleshoot "github.com/replicatedhq/troubleshoot/pkg/apis/troubleshoot/v1beta2" + + preflightv1beta2 "github.com/apecloud/kubeblocks/externalapis/preflight/v1beta2" +) + +const ( + clusterVersion = ` + { + "info": { + "major": "1", + "minor": "23", + "gitVersion": "v1.23.15", + "gitCommit": "b84cb8ab29366daa1bba65bc67f54de2f6c34848", + "gitTreeState": "clean", + "buildDate": "2022-12-08T10:42:57Z", + "goVersion": "go1.17.13", + "compiler": "gc", + "platform": "linux/arm64" + }, + "string": "v1.23.15" + }` + deploymentStatus = ` + { + "string": "1" + }` +) + +func FakeAnalyzers() []*troubleshoot.Analyze { + return []*troubleshoot.Analyze{ + { + ClusterVersion: &troubleshoot.ClusterVersion{ + AnalyzeMeta: troubleshoot.AnalyzeMeta{ + CheckName: "ClusterVersionCheck", + }, + Outcomes: []*troubleshoot.Outcome{ + { + Pass: &troubleshoot.SingleOutcome{ + Message: "version is ok.", + }, + }, + }, + }, + }, + { + DeploymentStatus: &troubleshoot.DeploymentStatus{ + AnalyzeMeta: troubleshoot.AnalyzeMeta{ + CheckName: "DeploymentStatusCheck", + }, + Outcomes: []*troubleshoot.Outcome{ + { + Warn: &troubleshoot.SingleOutcome{ + When: "absent", + Message: "The deployment is not present.", + }, + Pass: &troubleshoot.SingleOutcome{ + Message: "There are multiple replicas of the deployment ready", + }, + }, + }, + }, + }, + } +} + +func FakeCollectedData() map[string][]byte { + return map[string][]byte{ + "cluster-info/cluster_version.json": []byte(clusterVersion), + "cluster-info/deployment_status.json": []byte(deploymentStatus), + } +} + +func FakeKbPreflight() *preflightv1beta2.Preflight { + var collectList []*troubleshoot.Collect + collect := &troubleshoot.Collect{ + ClusterInfo: &troubleshoot.ClusterInfo{}, + ClusterResources: &troubleshoot.ClusterResources{}, + } + collectList = append(collectList, collect) + + return &preflightv1beta2.Preflight{ + Spec: preflightv1beta2.PreflightSpec{ + PreflightSpec: troubleshoot.PreflightSpec{ + Collectors: collectList, + }, + }, + } +} + +func FakeKbHostPreflight() *preflightv1beta2.HostPreflight { + var extendCollectList []*preflightv1beta2.ExtendHostCollect + var remoteCollectList []*troubleshoot.RemoteCollect + extendCollectList = append( + extendCollectList, + &preflightv1beta2.ExtendHostCollect{HostUtility: &preflightv1beta2.HostUtility{}}, + ) + remoteCollectList = append(remoteCollectList, &troubleshoot.RemoteCollect{CPU: &troubleshoot.RemoteCPU{}}) + + hostPreflight := &preflightv1beta2.HostPreflight{ + Spec: preflightv1beta2.HostPreflightSpec{ + ExtendCollectors: extendCollectList, + }, + } + hostPreflight.Spec.RemoteCollectors = remoteCollectList + return hostPreflight +} diff --git a/internal/preflight/testing/fake_test.go b/internal/preflight/testing/fake_test.go new file mode 100644 index 000000000..488b89542 --- /dev/null +++ b/internal/preflight/testing/fake_test.go @@ -0,0 +1,43 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testing + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var _ = Describe("test fake in internal/preflight", func() { + It("test FakeKbPreflight", func() { + kbPreflight := FakeKbPreflight() + Expect(kbPreflight).ShouldNot(BeNil()) + Expect(len(kbPreflight.Spec.Collectors)).Should(BeNumerically(">", 0)) + }) + + It("test FakeKbHostPreflight", func() { + hostKbPreflight := FakeKbHostPreflight() + Expect(hostKbPreflight).ShouldNot(BeNil()) + Expect(len(hostKbPreflight.Spec.RemoteCollectors)).Should(BeNumerically(">", 0)) + Expect(len(hostKbPreflight.Spec.ExtendCollectors)).Should(BeNumerically(">", 0)) + }) + + It("test FakeAnalyzers", func() { + analyzers := FakeAnalyzers() + Expect(analyzers).ShouldNot(BeNil()) + Expect(len(analyzers)).Should(BeNumerically(">", 0)) + }) +}) diff --git a/internal/preflight/testing/suite_test.go b/internal/preflight/testing/suite_test.go new file mode 100644 index 000000000..ba4c7f25e --- /dev/null +++ b/internal/preflight/testing/suite_test.go @@ -0,0 +1,29 @@ +/* +Copyright ApeCloud, Inc. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testing + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestFake(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Fake Preflight") +} From 86300bd069a05a2db480821e4439dec55133cc75 Mon Sep 17 00:00:00 2001 From: diankuizhao Date: Tue, 11 Apr 2023 17:48:43 +0800 Subject: [PATCH 77/80] fix: mysql-scale remove keyspace and shard params (#2501) --- .../templates/clusterdefinition.yaml | 11 ----------- deploy/apecloud-mysql-scale/values.yaml | 4 +--- 2 files changed, 1 insertion(+), 14 deletions(-) diff --git a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml index 8e67682c6..e76bb9f15 100644 --- a/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml +++ b/deploy/apecloud-mysql-scale/templates/clusterdefinition.yaml @@ -181,10 +181,6 @@ spec: env: - name: CELL value: {{ .Values.wesqlscale.cell | default "zone1" | quote }} - - name: KEYSPACE - value: {{ .Values.wesqlscale.keySpace | default "commerce" | quote }} - - name: SHARD - value: {{ .Values.wesqlscale.shard | default "0" | quote }} - name: ETCD_SERVER value: "$(KB_CLUSTER_NAME)-etcd-headless" - name: ETCD_PORT @@ -214,8 +210,6 @@ spec: - -c - | cell=${CELL:-'zone1'} - keyspace=${KEYSPACE:-'commerce'} - shard=${SHARD:-'0'} uid="${KB_POD_NAME##*-}" mysql_root=${MYSQL_ROOT_USER:-'root'} mysql_root_passwd=${MYSQL_ROOT_PASSWORD:-'123456'} @@ -242,8 +236,6 @@ spec: --log_queries_to_file $VTDATAROOT/$tablet_logfile \ --tablet-path $alias \ --tablet_hostname "$tablet_hostname" \ - --init_keyspace $keyspace \ - --init_shard $shard \ --init_tablet_type $tablet_type \ --health_check_interval 1s \ --shard_sync_retry_delay 1s \ @@ -511,8 +503,6 @@ spec: - | echo "starting vtconsensus" cell=${CELL:-'zone1'} - keyspace=${KEYSPACE:-'commerce'} - shard=${SHARD:-'0'} vtconsensusport=${VTCONSENSUS_PORT:-'16000'} topology_fags=${TOPOLOGY_FLAGS:-'--topo_implementation etcd2 --topo_global_server_address 127.0.0.1:2379 --topo_global_root /vitess/global'} @@ -521,7 +511,6 @@ spec: exec vtconsensus \ $topology_fags \ --alsologtostderr \ - --clusters_to_watch "$keyspace/$shard" \ --refresh_interval 1s \ --scan_repair_timeout 1s \ --log_dir ${VTDATAROOT} \ diff --git a/deploy/apecloud-mysql-scale/values.yaml b/deploy/apecloud-mysql-scale/values.yaml index f8fad702f..9eab84d14 100644 --- a/deploy/apecloud-mysql-scale/values.yaml +++ b/deploy/apecloud-mysql-scale/values.yaml @@ -74,9 +74,7 @@ metrics: pullPolicy: IfNotPresent wesqlscale: - cell: "zone1" - keySpace: "commerce" - shard: "0" + cell: "zone1" image: registry: registry.cn-hangzhou.aliyuncs.com repository: apecloud/apecloud-mysql-scale From a3051afc95a3f4e739e86a4bf59502d1f814c896 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Wed, 12 Apr 2023 13:21:10 +0800 Subject: [PATCH 78/80] chore: fixed go dependencies security issues (#2528) --- go.mod | 6 +++--- go.sum | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/go.mod b/go.mod index e5774b932..ce66694f2 100644 --- a/go.mod +++ b/go.mod @@ -19,8 +19,8 @@ require ( github.com/dapr/dapr v1.9.5 github.com/dapr/go-sdk v1.7.0 github.com/dapr/kit v0.0.3 - github.com/docker/cli v20.10.21+incompatible - github.com/docker/docker v20.10.23+incompatible + github.com/docker/cli v20.10.24+incompatible + github.com/docker/docker v20.10.24+incompatible github.com/docker/go-connections v0.4.0 github.com/evanphx/json-patch v5.6.0+incompatible github.com/fatih/color v1.14.1 @@ -268,7 +268,7 @@ require ( github.com/oklog/ulid v1.3.1 // indirect github.com/olekukonko/tablewriter v0.0.5 // indirect github.com/opencontainers/go-digest v1.0.0 // indirect - github.com/opencontainers/runc v1.1.4 // indirect + github.com/opencontainers/runc v1.1.5 // indirect github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 // indirect github.com/opencontainers/selinux v1.10.2 // indirect github.com/openzipkin/zipkin-go v0.4.0 // indirect diff --git a/go.sum b/go.sum index 1e33980cc..1989c2b82 100644 --- a/go.sum +++ b/go.sum @@ -473,16 +473,16 @@ github.com/distribution/distribution/v3 v3.0.0-20221208165359-362910506bc2 h1:aB github.com/distribution/distribution/v3 v3.0.0-20221208165359-362910506bc2/go.mod h1:WHNsWjnIn2V1LYOrME7e8KxSeKunYHsxEm4am0BUtcI= github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E= github.com/docker/cli v0.0.0-20191017083524-a8ff7f821017/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8= -github.com/docker/cli v20.10.21+incompatible h1:qVkgyYUnOLQ98LtXBrwd/duVqPT2X4SHndOuGsfwyhU= -github.com/docker/cli v20.10.21+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8= +github.com/docker/cli v20.10.24+incompatible h1:vfV+1kv9yD0/cpL6wWY9cE+Y9J8hL/NqJDGob0B3RVw= +github.com/docker/cli v20.10.24+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8= github.com/docker/distribution v0.0.0-20190905152932-14b96e55d84c/go.mod h1:0+TTO4EOBfRPhZXAeF1Vu+W3hHZ8eLp8PgKVZlcvtFY= github.com/docker/distribution v2.7.1-0.20190205005809-0d3efadf0154+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= github.com/docker/distribution v2.7.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= github.com/docker/distribution v2.8.1+incompatible h1:Q50tZOPR6T/hjNsyc9g8/syEs6bk8XXApsHjKukMl68= github.com/docker/distribution v2.8.1+incompatible/go.mod h1:J2gT2udsDAN96Uj4KfcMRqY0/ypR+oyYUYmja8H+y+w= github.com/docker/docker v1.4.2-0.20190924003213-a8608b5b67c7/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= -github.com/docker/docker v20.10.23+incompatible h1:1ZQUUYAdh+oylOT85aA2ZcfRp22jmLhoaEcVEfK8dyA= -github.com/docker/docker v20.10.23+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/docker v20.10.24+incompatible h1:Ugvxm7a8+Gz6vqQYQQ2W7GYq5EUPaAiuPgIfVyI3dYE= +github.com/docker/docker v20.10.24+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= github.com/docker/docker-credential-helpers v0.6.3/go.mod h1:WRaJzqw3CTB9bk10avuGsjVBZsD05qeibJ1/TYlvc0Y= github.com/docker/docker-credential-helpers v0.7.0 h1:xtCHsjxogADNZcdv1pKUHXryefjlVRqWqIhk/uXJp0A= github.com/docker/docker-credential-helpers v0.7.0/go.mod h1:rETQfLdHNT3foU5kuNkFR1R1V12OJRRO5lzt2D1b5X0= @@ -1275,8 +1275,8 @@ github.com/opencontainers/runc v1.0.0-rc8.0.20190926000215-3e425f80a8c9/go.mod h github.com/opencontainers/runc v1.0.0-rc9/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U= github.com/opencontainers/runc v1.0.0-rc93/go.mod h1:3NOsor4w32B2tC0Zbl8Knk4Wg84SM2ImC1fxBuqJ/H0= github.com/opencontainers/runc v1.0.2/go.mod h1:aTaHFFwQXuA71CiyxOdFFIorAoemI04suvGRQFzWTD0= -github.com/opencontainers/runc v1.1.4 h1:nRCz/8sKg6K6jgYAFLDlXzPeITBZJyX28DBVhWD+5dg= -github.com/opencontainers/runc v1.1.4/go.mod h1:1J5XiS+vdZ3wCyZybsuxXZWGrgSr8fFJHLXuG2PsnNg= +github.com/opencontainers/runc v1.1.5 h1:L44KXEpKmfWDcS02aeGm8QNTFXTo2D+8MYGDIJ/GDEs= +github.com/opencontainers/runc v1.1.5/go.mod h1:1J5XiS+vdZ3wCyZybsuxXZWGrgSr8fFJHLXuG2PsnNg= github.com/opencontainers/runtime-spec v0.1.2-0.20190507144316-5b71a03e2700/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0= github.com/opencontainers/runtime-spec v1.0.1/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0= github.com/opencontainers/runtime-spec v1.0.2-0.20190207185410-29686dbc5559/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0= From 83ee0af9bf74c56bfa32901f9c8bad7fda4bd3e6 Mon Sep 17 00:00:00 2001 From: Nash Tsai Date: Wed, 12 Apr 2023 13:47:51 +0800 Subject: [PATCH 79/80] chore: fixed go dependencies moderate security issues (#2529) --- go.mod | 6 +- go.sum | 319 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 311 insertions(+), 14 deletions(-) diff --git a/go.mod b/go.mod index ce66694f2..1b2ec0e5f 100644 --- a/go.mod +++ b/go.mod @@ -132,7 +132,7 @@ require ( github.com/cloudflare/circl v1.1.0 // indirect github.com/cockroachdb/apd/v2 v2.0.1 // indirect github.com/containerd/cgroups v1.0.4 // indirect - github.com/containerd/containerd v1.6.15 // indirect + github.com/containerd/containerd v1.6.18 // indirect github.com/containers/image/v5 v5.24.0 // indirect github.com/containers/libtrust v0.0.0-20230121012942-c1716e8a8d01 // indirect github.com/containers/ocicrypt v1.1.7 // indirect @@ -204,7 +204,7 @@ require ( github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.0 // indirect github.com/hashicorp/errwrap v1.1.0 // indirect github.com/hashicorp/go-cleanhttp v0.5.2 // indirect - github.com/hashicorp/go-getter v1.6.2 // indirect + github.com/hashicorp/go-getter v1.7.0 // indirect github.com/hashicorp/go-multierror v1.1.1 // indirect github.com/hashicorp/go-safetemp v1.0.0 // indirect github.com/hashicorp/golang-lru v0.5.4 // indirect @@ -249,7 +249,7 @@ require ( github.com/mistifyio/go-zfs/v3 v3.0.0 // indirect github.com/mitchellh/copystructure v1.2.0 // indirect github.com/mitchellh/go-homedir v1.1.0 // indirect - github.com/mitchellh/go-testing-interface v1.14.0 // indirect + github.com/mitchellh/go-testing-interface v1.14.1 // indirect github.com/mitchellh/go-wordwrap v1.0.1 // indirect github.com/mitchellh/reflectwalk v1.0.2 // indirect github.com/moby/locker v1.0.1 // indirect diff --git a/go.sum b/go.sum index 1989c2b82..b1c5304f0 100644 --- a/go.sum +++ b/go.sum @@ -20,35 +20,173 @@ cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmW cloud.google.com/go v0.78.0/go.mod h1:QjdrLG0uq+YwhjoVOLsS1t7TW8fs36kLs4XO5R5ECHg= cloud.google.com/go v0.79.0/go.mod h1:3bzgcEeQlzbuEAYu4mrWhKqWjmpprinYgKJLgKHnbb8= cloud.google.com/go v0.81.0/go.mod h1:mk/AM35KwGk/Nm2YSeZbxXdrNK3KZOYHmLkOqC2V6E0= +cloud.google.com/go v0.83.0/go.mod h1:Z7MJUsANfY0pYPdw0lbnivPx4/vhy/e2FEkSkF7vAVY= +cloud.google.com/go v0.84.0/go.mod h1:RazrYuxIK6Kb7YrzzhPoLmCVzl7Sup4NrbKPg8KHSUM= +cloud.google.com/go v0.87.0/go.mod h1:TpDYlFy7vuLzZMMZ+B6iRiELaY7z/gJPaqbMx6mlWcY= +cloud.google.com/go v0.90.0/go.mod h1:kRX0mNRHe0e2rC6oNakvwQqzyDmg57xJ+SZU1eT2aDQ= +cloud.google.com/go v0.93.3/go.mod h1:8utlLll2EF5XMAV15woO4lSbWQlk8rer9aLOfLh7+YI= +cloud.google.com/go v0.94.1/go.mod h1:qAlAugsXlC+JWO+Bke5vCtc9ONxjQT3drlTTnAplMW4= +cloud.google.com/go v0.97.0/go.mod h1:GF7l59pYBVlXQIBLx3a761cZ41F9bBH3JUlihCt2Udc= +cloud.google.com/go v0.99.0/go.mod h1:w0Xx2nLzqWJPuozYQX+hFfCSI8WioryfRDzkoI/Y2ZA= +cloud.google.com/go v0.100.2/go.mod h1:4Xra9TjzAeYHrl5+oeLlzbM2k3mjVhZh4UqTZ//w99A= +cloud.google.com/go v0.102.0/go.mod h1:oWcCzKlqJ5zgHQt9YsaeTY9KzIvjyy0ArmiBUgpQ+nc= +cloud.google.com/go v0.102.1/go.mod h1:XZ77E9qnTEnrgEOvr4xzfdX5TRo7fB4T2F4O6+34hIU= +cloud.google.com/go v0.104.0/go.mod h1:OO6xxXdJyvuJPcEPBLN9BJPD+jep5G1+2U5B5gkRYtA= cloud.google.com/go v0.105.0 h1:DNtEKRBAAzeS4KyIory52wWHuClNaXJ5x1F7xa4q+5Y= cloud.google.com/go v0.105.0/go.mod h1:PrLgOJNe5nfE9UMxKxgXj4mD3voiP+YQ6gdt6KMFOKM= +cloud.google.com/go/aiplatform v1.22.0/go.mod h1:ig5Nct50bZlzV6NvKaTwmplLLddFx0YReh9WfTO5jKw= +cloud.google.com/go/aiplatform v1.24.0/go.mod h1:67UUvRBKG6GTayHKV8DBv2RtR1t93YRu5B1P3x99mYY= +cloud.google.com/go/analytics v0.11.0/go.mod h1:DjEWCu41bVbYcKyvlws9Er60YE4a//bK6mnhWvQeFNI= +cloud.google.com/go/analytics v0.12.0/go.mod h1:gkfj9h6XRf9+TS4bmuhPEShsh3hH8PAZzm/41OOhQd4= +cloud.google.com/go/area120 v0.5.0/go.mod h1:DE/n4mp+iqVyvxHN41Vf1CR602GiHQjFPusMFW6bGR4= +cloud.google.com/go/area120 v0.6.0/go.mod h1:39yFJqWVgm0UZqWTOdqkLhjoC7uFfgXRC8g/ZegeAh0= +cloud.google.com/go/artifactregistry v1.6.0/go.mod h1:IYt0oBPSAGYj/kprzsBjZ/4LnG/zOcHyFHjWPCi6SAQ= +cloud.google.com/go/artifactregistry v1.7.0/go.mod h1:mqTOFOnGZx8EtSqK/ZWcsm/4U8B77rbcLP6ruDU2Ixk= +cloud.google.com/go/asset v1.5.0/go.mod h1:5mfs8UvcM5wHhqtSv8J1CtxxaQq3AdBxxQi2jGW/K4o= +cloud.google.com/go/asset v1.7.0/go.mod h1:YbENsRK4+xTiL+Ofoj5Ckf+O17kJtgp3Y3nn4uzZz5s= +cloud.google.com/go/asset v1.8.0/go.mod h1:mUNGKhiqIdbr8X7KNayoYvyc4HbbFO9URsjbytpUaW0= +cloud.google.com/go/assuredworkloads v1.5.0/go.mod h1:n8HOZ6pff6re5KYfBXcFvSViQjDwxFkAkmUFffJRbbY= +cloud.google.com/go/assuredworkloads v1.6.0/go.mod h1:yo2YOk37Yc89Rsd5QMVECvjaMKymF9OP+QXWlKXUkXw= +cloud.google.com/go/assuredworkloads v1.7.0/go.mod h1:z/736/oNmtGAyU47reJgGN+KVoYoxeLBoj4XkKYscNI= +cloud.google.com/go/automl v1.5.0/go.mod h1:34EjfoFGMZ5sgJ9EoLsRtdPSNZLcfflJR39VbVNS2M0= +cloud.google.com/go/automl v1.6.0/go.mod h1:ugf8a6Fx+zP0D59WLhqgTDsQI9w07o64uf/Is3Nh5p8= cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o= cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE= cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc= cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg= cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc= cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ= +cloud.google.com/go/bigquery v1.42.0/go.mod h1:8dRTJxhtG+vwBKzE5OseQn/hiydoQN3EedCaOdYmxRA= +cloud.google.com/go/billing v1.4.0/go.mod h1:g9IdKBEFlItS8bTtlrZdVLWSSdSyFUZKXNS02zKMOZY= +cloud.google.com/go/billing v1.5.0/go.mod h1:mztb1tBc3QekhjSgmpf/CV4LzWXLzCArwpLmP2Gm88s= +cloud.google.com/go/binaryauthorization v1.1.0/go.mod h1:xwnoWu3Y84jbuHa0zd526MJYmtnVXn0syOjaJgy4+dM= +cloud.google.com/go/binaryauthorization v1.2.0/go.mod h1:86WKkJHtRcv5ViNABtYMhhNWRrD1Vpi//uKEy7aYEfI= +cloud.google.com/go/cloudtasks v1.5.0/go.mod h1:fD92REy1x5woxkKEkLdvavGnPJGEn8Uic9nWuLzqCpY= +cloud.google.com/go/cloudtasks v1.6.0/go.mod h1:C6Io+sxuke9/KNRkbQpihnW93SWDU3uXt92nu85HkYI= +cloud.google.com/go/compute v0.1.0/go.mod h1:GAesmwr110a34z04OlxYkATPBEfVhkymfTBXtfbBFow= +cloud.google.com/go/compute v1.3.0/go.mod h1:cCZiE1NHEtai4wiufUhW8I8S1JKkAnhnQJWM7YD99wM= +cloud.google.com/go/compute v1.5.0/go.mod h1:9SMHyhJlzhlkJqrPAc839t2BZFTSk6Jdj6mkzQJeu0M= +cloud.google.com/go/compute v1.6.0/go.mod h1:T29tfhtVbq1wvAPo0E3+7vhgmkOYeXjhFvz/FMzPu0s= +cloud.google.com/go/compute v1.6.1/go.mod h1:g85FgpzFvNULZ+S8AYq87axRKuf2Kh7deLqV/jJ3thU= +cloud.google.com/go/compute v1.7.0/go.mod h1:435lt8av5oL9P3fv1OEzSbSUe+ybHXGMPQHHZWZxy9U= +cloud.google.com/go/compute v1.10.0/go.mod h1:ER5CLbMxl90o2jtNbGSbtfOpQKR0t15FOtRsugnLrlU= cloud.google.com/go/compute v1.14.0 h1:hfm2+FfxVmnRlh6LpB7cg1ZNU+5edAHmW679JePztk0= cloud.google.com/go/compute v1.14.0/go.mod h1:YfLtxrj9sU4Yxv+sXzZkyPjEyPBZfXHUvjxega5vAdo= cloud.google.com/go/compute/metadata v0.2.3 h1:mg4jlk7mCAj6xXp9UJ4fjI9VUI5rubuGBW5aJ7UnBMY= cloud.google.com/go/compute/metadata v0.2.3/go.mod h1:VAV5nSsACxMJvgaAuX6Pk2AawlZn8kiOGuCv6gTkwuA= +cloud.google.com/go/containeranalysis v0.5.1/go.mod h1:1D92jd8gRR/c0fGMlymRgxWD3Qw9C1ff6/T7mLgVL8I= +cloud.google.com/go/containeranalysis v0.6.0/go.mod h1:HEJoiEIu+lEXM+k7+qLCci0h33lX3ZqoYFdmPcoO7s4= +cloud.google.com/go/datacatalog v1.3.0/go.mod h1:g9svFY6tuR+j+hrTw3J2dNcmI0dzmSiyOzm8kpLq0a0= +cloud.google.com/go/datacatalog v1.5.0/go.mod h1:M7GPLNQeLfWqeIm3iuiruhPzkt65+Bx8dAKvScX8jvs= +cloud.google.com/go/datacatalog v1.6.0/go.mod h1:+aEyF8JKg+uXcIdAmmaMUmZ3q1b/lKLtXCmXdnc0lbc= +cloud.google.com/go/dataflow v0.6.0/go.mod h1:9QwV89cGoxjjSR9/r7eFDqqjtvbKxAK2BaYU6PVk9UM= +cloud.google.com/go/dataflow v0.7.0/go.mod h1:PX526vb4ijFMesO1o202EaUmouZKBpjHsTlCtB4parQ= +cloud.google.com/go/dataform v0.3.0/go.mod h1:cj8uNliRlHpa6L3yVhDOBrUXH+BPAO1+KFMQQNSThKo= +cloud.google.com/go/dataform v0.4.0/go.mod h1:fwV6Y4Ty2yIFL89huYlEkwUPtS7YZinZbzzj5S9FzCE= +cloud.google.com/go/datalabeling v0.5.0/go.mod h1:TGcJ0G2NzcsXSE/97yWjIZO0bXj0KbVlINXMG9ud42I= +cloud.google.com/go/datalabeling v0.6.0/go.mod h1:WqdISuk/+WIGeMkpw/1q7bK/tFEZxsrFJOJdY2bXvTQ= +cloud.google.com/go/dataqna v0.5.0/go.mod h1:90Hyk596ft3zUQ8NkFfvICSIfHFh1Bc7C4cK3vbhkeo= +cloud.google.com/go/dataqna v0.6.0/go.mod h1:1lqNpM7rqNLVgWBJyk5NF6Uen2PHym0jtVJonplVsDA= cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE= cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk= +cloud.google.com/go/datastream v1.2.0/go.mod h1:i/uTP8/fZwgATHS/XFu0TcNUhuA0twZxxQ3EyCUQMwo= +cloud.google.com/go/datastream v1.3.0/go.mod h1:cqlOX8xlyYF/uxhiKn6Hbv6WjwPPuI9W2M9SAXwaLLQ= +cloud.google.com/go/dialogflow v1.15.0/go.mod h1:HbHDWs33WOGJgn6rfzBW1Kv807BE3O1+xGbn59zZWI4= +cloud.google.com/go/dialogflow v1.16.1/go.mod h1:po6LlzGfK+smoSmTBnbkIZY2w8ffjz/RcGSS+sh1el0= +cloud.google.com/go/dialogflow v1.17.0/go.mod h1:YNP09C/kXA1aZdBgC/VtXX74G/TKn7XVCcVumTflA+8= +cloud.google.com/go/documentai v1.7.0/go.mod h1:lJvftZB5NRiFSX4moiye1SMxHx0Bc3x1+p9e/RfXYiU= +cloud.google.com/go/documentai v1.8.0/go.mod h1:xGHNEB7CtsnySCNrCFdCyyMz44RhFEEX2Q7UD0c5IhU= +cloud.google.com/go/domains v0.6.0/go.mod h1:T9Rz3GasrpYk6mEGHh4rymIhjlnIuB4ofT1wTxDeT4Y= +cloud.google.com/go/domains v0.7.0/go.mod h1:PtZeqS1xjnXuRPKE/88Iru/LdfoRyEHYA9nFQf4UKpg= +cloud.google.com/go/edgecontainer v0.1.0/go.mod h1:WgkZ9tp10bFxqO8BLPqv2LlfmQF1X8lZqwW4r1BTajk= +cloud.google.com/go/edgecontainer v0.2.0/go.mod h1:RTmLijy+lGpQ7BXuTDa4C4ssxyXT34NIuHIgKuP4s5w= cloud.google.com/go/firestore v1.1.0/go.mod h1:ulACoGHTpvq5r8rxGJ4ddJZBZqakUQqClKRT5SZwBmk= +cloud.google.com/go/functions v1.6.0/go.mod h1:3H1UA3qiIPRWD7PeZKLvHZ9SaQhR26XIJcC0A5GbvAk= +cloud.google.com/go/functions v1.7.0/go.mod h1:+d+QBcWM+RsrgZfV9xo6KfA1GlzJfxcfZcRPEhDDfzg= +cloud.google.com/go/gaming v1.5.0/go.mod h1:ol7rGcxP/qHTRQE/RO4bxkXq+Fix0j6D4LFPzYTIrDM= +cloud.google.com/go/gaming v1.6.0/go.mod h1:YMU1GEvA39Qt3zWGyAVA9bpYz/yAhTvaQ1t2sK4KPUA= +cloud.google.com/go/gkeconnect v0.5.0/go.mod h1:c5lsNAg5EwAy7fkqX/+goqFsU1Da/jQFqArp+wGNr/o= +cloud.google.com/go/gkeconnect v0.6.0/go.mod h1:Mln67KyU/sHJEBY8kFZ0xTeyPtzbq9StAVvEULYK16A= +cloud.google.com/go/gkehub v0.9.0/go.mod h1:WYHN6WG8w9bXU0hqNxt8rm5uxnk8IH+lPY9J2TV7BK0= +cloud.google.com/go/gkehub v0.10.0/go.mod h1:UIPwxI0DsrpsVoWpLB0stwKCP+WFVG9+y977wO+hBH0= +cloud.google.com/go/grafeas v0.2.0/go.mod h1:KhxgtF2hb0P191HlY5besjYm6MqTSTj3LSI+M+ByZHc= +cloud.google.com/go/iam v0.3.0/go.mod h1:XzJPvDayI+9zsASAFO68Hk07u3z+f+JrT2xXNdp4bnY= +cloud.google.com/go/iam v0.5.0/go.mod h1:wPU9Vt0P4UmCux7mqtRu6jcpPAb74cP1fh50J3QpkUc= cloud.google.com/go/iam v0.8.0 h1:E2osAkZzxI/+8pZcxVLcDtAQx/u+hZXVryUaYQ5O0Kk= cloud.google.com/go/iam v0.8.0/go.mod h1:lga0/y3iH6CX7sYqypWJ33hf7kkfXJag67naqGESjkE= +cloud.google.com/go/language v1.4.0/go.mod h1:F9dRpNFQmJbkaop6g0JhSBXCNlO90e1KWx5iDdxbWic= +cloud.google.com/go/language v1.6.0/go.mod h1:6dJ8t3B+lUYfStgls25GusK04NLh3eDLQnWM3mdEbhI= +cloud.google.com/go/lifesciences v0.5.0/go.mod h1:3oIKy8ycWGPUyZDR/8RNnTOYevhaMLqh5vLUXs9zvT8= +cloud.google.com/go/lifesciences v0.6.0/go.mod h1:ddj6tSX/7BOnhxCSd3ZcETvtNr8NZ6t/iPhY2Tyfu08= cloud.google.com/go/longrunning v0.3.0 h1:NjljC+FYPV3uh5/OwWT6pVU+doBqMg2x/rZlE+CamDs= +cloud.google.com/go/mediatranslation v0.5.0/go.mod h1:jGPUhGTybqsPQn91pNXw0xVHfuJ3leR1wj37oU3y1f4= +cloud.google.com/go/mediatranslation v0.6.0/go.mod h1:hHdBCTYNigsBxshbznuIMFNe5QXEowAuNmmC7h8pu5w= +cloud.google.com/go/memcache v1.4.0/go.mod h1:rTOfiGZtJX1AaFUrOgsMHX5kAzaTQ8azHiuDoTPzNsE= +cloud.google.com/go/memcache v1.5.0/go.mod h1:dk3fCK7dVo0cUU2c36jKb4VqKPS22BTkf81Xq617aWM= +cloud.google.com/go/metastore v1.5.0/go.mod h1:2ZNrDcQwghfdtCwJ33nM0+GrBGlVuh8rakL3vdPY3XY= +cloud.google.com/go/metastore v1.6.0/go.mod h1:6cyQTls8CWXzk45G55x57DVQ9gWg7RiH65+YgPsNh9s= +cloud.google.com/go/networkconnectivity v1.4.0/go.mod h1:nOl7YL8odKyAOtzNX73/M5/mGZgqqMeryi6UPZTk/rA= +cloud.google.com/go/networkconnectivity v1.5.0/go.mod h1:3GzqJx7uhtlM3kln0+x5wyFvuVH1pIBJjhCpjzSt75o= +cloud.google.com/go/networksecurity v0.5.0/go.mod h1:xS6fOCoqpVC5zx15Z/MqkfDwH4+m/61A3ODiDV1xmiQ= +cloud.google.com/go/networksecurity v0.6.0/go.mod h1:Q5fjhTr9WMI5mbpRYEbiexTzROf7ZbDzvzCrNl14nyU= +cloud.google.com/go/notebooks v1.2.0/go.mod h1:9+wtppMfVPUeJ8fIWPOq1UnATHISkGXGqTkxeieQ6UY= +cloud.google.com/go/notebooks v1.3.0/go.mod h1:bFR5lj07DtCPC7YAAJ//vHskFBxA5JzYlH68kXVdk34= +cloud.google.com/go/osconfig v1.7.0/go.mod h1:oVHeCeZELfJP7XLxcBGTMBvRO+1nQ5tFG9VQTmYS2Fs= +cloud.google.com/go/osconfig v1.8.0/go.mod h1:EQqZLu5w5XA7eKizepumcvWx+m8mJUhEwiPqWiZeEdg= +cloud.google.com/go/oslogin v1.4.0/go.mod h1:YdgMXWRaElXz/lDk1Na6Fh5orF7gvmJ0FGLIs9LId4E= +cloud.google.com/go/oslogin v1.5.0/go.mod h1:D260Qj11W2qx/HVF29zBg+0fd6YCSjSqLUkY/qEenQU= +cloud.google.com/go/phishingprotection v0.5.0/go.mod h1:Y3HZknsK9bc9dMi+oE8Bim0lczMU6hrX0UpADuMefr0= +cloud.google.com/go/phishingprotection v0.6.0/go.mod h1:9Y3LBLgy0kDTcYET8ZH3bq/7qni15yVUoAxiFxnlSUA= +cloud.google.com/go/privatecatalog v0.5.0/go.mod h1:XgosMUvvPyxDjAVNDYxJ7wBW8//hLDDYmnsNcMGq1K0= +cloud.google.com/go/privatecatalog v0.6.0/go.mod h1:i/fbkZR0hLN29eEWiiwue8Pb+GforiEIBnV9yrRUOKI= cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I= cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw= cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA= cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU= +cloud.google.com/go/recaptchaenterprise v1.3.1/go.mod h1:OdD+q+y4XGeAlxRaMn1Y7/GveP6zmq76byL6tjPE7d4= +cloud.google.com/go/recaptchaenterprise/v2 v2.1.0/go.mod h1:w9yVqajwroDNTfGuhmOjPDN//rZGySaf6PtFVcSCa7o= +cloud.google.com/go/recaptchaenterprise/v2 v2.2.0/go.mod h1:/Zu5jisWGeERrd5HnlS3EUGb/D335f9k51B/FVil0jk= +cloud.google.com/go/recaptchaenterprise/v2 v2.3.0/go.mod h1:O9LwGCjrhGHBQET5CA7dd5NwwNQUErSgEDit1DLNTdo= +cloud.google.com/go/recommendationengine v0.5.0/go.mod h1:E5756pJcVFeVgaQv3WNpImkFP8a+RptV6dDLGPILjvg= +cloud.google.com/go/recommendationengine v0.6.0/go.mod h1:08mq2umu9oIqc7tDy8sx+MNJdLG0fUi3vaSVbztHgJ4= +cloud.google.com/go/recommender v1.5.0/go.mod h1:jdoeiBIVrJe9gQjwd759ecLJbxCDED4A6p+mqoqDvTg= +cloud.google.com/go/recommender v1.6.0/go.mod h1:+yETpm25mcoiECKh9DEScGzIRyDKpZ0cEhWGo+8bo+c= +cloud.google.com/go/redis v1.7.0/go.mod h1:V3x5Jq1jzUcg+UNsRvdmsfuFnit1cfe3Z/PGyq/lm4Y= +cloud.google.com/go/redis v1.8.0/go.mod h1:Fm2szCDavWzBk2cDKxrkmWBqoCiL1+Ctwq7EyqBCA/A= +cloud.google.com/go/retail v1.8.0/go.mod h1:QblKS8waDmNUhghY2TI9O3JLlFk8jybHeV4BF19FrE4= +cloud.google.com/go/retail v1.9.0/go.mod h1:g6jb6mKuCS1QKnH/dpu7isX253absFl6iE92nHwlBUY= +cloud.google.com/go/scheduler v1.4.0/go.mod h1:drcJBmxF3aqZJRhmkHQ9b3uSSpQoltBPGPxGAWROx6s= +cloud.google.com/go/scheduler v1.5.0/go.mod h1:ri073ym49NW3AfT6DZi21vLZrG07GXr5p3H1KxN5QlI= +cloud.google.com/go/secretmanager v1.6.0/go.mod h1:awVa/OXF6IiyaU1wQ34inzQNc4ISIDIrId8qE5QGgKA= +cloud.google.com/go/security v1.5.0/go.mod h1:lgxGdyOKKjHL4YG3/YwIL2zLqMFCKs0UbQwgyZmfJl4= +cloud.google.com/go/security v1.7.0/go.mod h1:mZklORHl6Bg7CNnnjLH//0UlAlaXqiG7Lb9PsPXLfD0= +cloud.google.com/go/security v1.8.0/go.mod h1:hAQOwgmaHhztFhiQ41CjDODdWP0+AE1B3sX4OFlq+GU= +cloud.google.com/go/securitycenter v1.13.0/go.mod h1:cv5qNAqjY84FCN6Y9z28WlkKXyWsgLO832YiWwkCWcU= +cloud.google.com/go/securitycenter v1.14.0/go.mod h1:gZLAhtyKv85n52XYWt6RmeBdydyxfPeTrpToDPw4Auc= +cloud.google.com/go/servicedirectory v1.4.0/go.mod h1:gH1MUaZCgtP7qQiI+F+A+OpeKF/HQWgtAddhTbhL2bs= +cloud.google.com/go/servicedirectory v1.5.0/go.mod h1:QMKFL0NUySbpZJ1UZs3oFAmdvVxhhxB6eJ/Vlp73dfg= +cloud.google.com/go/speech v1.6.0/go.mod h1:79tcr4FHCimOp56lwC01xnt/WPJZc4v3gzyT7FoBkCM= +cloud.google.com/go/speech v1.7.0/go.mod h1:KptqL+BAQIhMsj1kOP2la5DSEEerPDuOP/2mmkhHhZQ= cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw= cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos= cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk= cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs= cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0= +cloud.google.com/go/storage v1.22.1/go.mod h1:S8N1cAStu7BOeFfE8KAQzmyyLkK8p/vmRq6kuBTW58Y= +cloud.google.com/go/storage v1.23.0/go.mod h1:vOEEDNFnciUMhBeT6hsJIn3ieU5cFRmzeLgDvXzfIXc= cloud.google.com/go/storage v1.27.0 h1:YOO045NZI9RKfCj1c5A/ZtuuENUc8OAW+gHdGnDgyMQ= cloud.google.com/go/storage v1.27.0/go.mod h1:x9DOL8TK/ygDUMieqwfhdpQryTeEkhGKMi80i/iqR2s= +cloud.google.com/go/talent v1.1.0/go.mod h1:Vl4pt9jiHKvOgF9KoZo6Kob9oV4lwd/ZD5Cto54zDRw= +cloud.google.com/go/talent v1.2.0/go.mod h1:MoNF9bhFQbiJ6eFD3uSsg0uBALw4n4gaCaEjBw9zo8g= +cloud.google.com/go/videointelligence v1.6.0/go.mod h1:w0DIDlVRKtwPCn/C4iwZIJdvC69yInhW0cfi+p546uU= +cloud.google.com/go/videointelligence v1.7.0/go.mod h1:k8pI/1wAhjznARtVT9U1llUaFNPh7muw8QyOUpavru4= +cloud.google.com/go/vision v1.2.0/go.mod h1:SmNwgObm5DpFBme2xpyOyasvBc1aPdjvMk2bBk0tKD0= +cloud.google.com/go/vision/v2 v2.2.0/go.mod h1:uCdV4PpN1S0jyCyq8sIM42v2Y6zOLkZs+4R9LrGYwFo= +cloud.google.com/go/vision/v2 v2.3.0/go.mod h1:UO61abBx9QRMFkNBbf1D8B1LXdS2cGiiCRx0vSpZoUo= +cloud.google.com/go/webrisk v1.4.0/go.mod h1:Hn8X6Zr+ziE2aNd8SliSDWpEnSS1u4R9+xXZmFiHmGE= +cloud.google.com/go/webrisk v1.5.0/go.mod h1:iPG6fr52Tv7sGk0H6qUFzmL3HHZev1htXuWDEEsqMTg= +cloud.google.com/go/workflows v1.6.0/go.mod h1:6t9F5h/unJz41YqfBmqSASJSXccBLtD1Vwf+KmJENM0= +cloud.google.com/go/workflows v1.7.0/go.mod h1:JhSrZuVZWuiDfKEFxU0/F1PQjmpnpcoISEXH2bcHC3M= contrib.go.opencensus.io/exporter/prometheus v0.4.1 h1:oObVeKo2NxpdF/fIfrPsNj6K0Prg0R0mHM+uANlYMiM= contrib.go.opencensus.io/exporter/prometheus v0.4.1/go.mod h1:t9wvfitlUjGXG2IXAZsuFq26mDGid/JwCEXp+gTG/9U= cuelang.org/go v0.4.3 h1:W3oBBjDTm7+IZfCKZAmC8uDG0eYfJL4Pp/xbbCMKaVo= @@ -200,7 +338,6 @@ github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d/go.mod h1:W github.com/authzed/controller-idioms v0.7.0 h1:HhNMUBb8hJzYqY3mhen3B2AC5nsIem3fBe0tC/AAOHo= github.com/authzed/controller-idioms v0.7.0/go.mod h1:0B/PmqCguKv8b3azSMF+HdyKpKr2o3UAZ5eo12Ze8Fo= github.com/aws/aws-sdk-go v1.15.11/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= -github.com/aws/aws-sdk-go v1.15.78/go.mod h1:E3/ieXAlvM0XWO57iftYVDLLvQ824smPP3ATZkfNZeM= github.com/aws/aws-sdk-go v1.44.122 h1:p6mw01WBaNpbdP2xrisz5tIkcNwzj/HysobNoaAHjgo= github.com/aws/aws-sdk-go v1.44.122/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo= github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= @@ -346,8 +483,8 @@ github.com/containerd/containerd v1.5.0-beta.4/go.mod h1:GmdgZd2zA2GYIBZ0w09Zvgq github.com/containerd/containerd v1.5.0-rc.0/go.mod h1:V/IXoMqNGgBlabz3tHD2TWDoTJseu1FGOKuoA4nNb2s= github.com/containerd/containerd v1.5.1/go.mod h1:0DOxVqwDy2iZvrZp2JUx/E+hS0UNTVn7dJnIOwtYR4g= github.com/containerd/containerd v1.5.7/go.mod h1:gyvv6+ugqY25TiXxcZC3L5yOeYgEw0QMhscqVp1AR9c= -github.com/containerd/containerd v1.6.15 h1:4wWexxzLNHNE46aIETc6ge4TofO550v+BlLoANrbses= -github.com/containerd/containerd v1.6.15/go.mod h1:U2NnBPIhzJDm59xF7xB2MMHnKtggpZ+phKg8o2TKj2c= +github.com/containerd/containerd v1.6.18 h1:qZbsLvmyu+Vlty0/Ex5xc0z2YtKpIsb5n45mAMI+2Ns= +github.com/containerd/containerd v1.6.18/go.mod h1:1RdCUu95+gc2v9t3IL+zIlpClSmew7/0YS8O5eQZrOw= github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= github.com/containerd/continuity v0.0.0-20190815185530-f2a389ac0a02/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= github.com/containerd/continuity v0.0.0-20191127005431-f65d91d395eb/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= @@ -739,6 +876,7 @@ github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM= github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golangci/check v0.0.0-20180506172741-cfe4005ccda2/go.mod h1:k9Qvh+8juN+UKMCS/3jFtGICgW8O96FVaZsaxdzDkR4= @@ -797,6 +935,7 @@ github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXi github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0= github.com/google/martian/v3 v3.2.1 h1:d8MncMlErDFTwQGBK1xhv026j9kqhvw1Qv9IbWT1VLQ= +github.com/google/martian/v3 v3.2.1/go.mod h1:oBOf6HBosgwRXnUGWUB05QECsc6uvmMiJ3+6W4l/CUk= github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM= @@ -808,6 +947,9 @@ github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLe github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210122040257-d980be63207e/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/pprof v0.0.0-20210226084205-cbba55b83ad5/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20210601050228-01bbb1931b22/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20210609004039-a478d1d731e9/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= +github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE= github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4= github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ= @@ -817,13 +959,24 @@ github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+ github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/googleapis/enterprise-certificate-proxy v0.0.0-20220520183353-fd19c99a87aa/go.mod h1:17drOmN3MwGY7t0e+Ei9b45FFGA3fBs3x36SsCg1hq8= +github.com/googleapis/enterprise-certificate-proxy v0.1.0/go.mod h1:17drOmN3MwGY7t0e+Ei9b45FFGA3fBs3x36SsCg1hq8= +github.com/googleapis/enterprise-certificate-proxy v0.2.0/go.mod h1:8C0jb7/mgJe/9KK8Lm7X9ctZC2t60YyIpYEI16jx0Qg= github.com/googleapis/enterprise-certificate-proxy v0.2.1 h1:RY7tHKZcRlk788d5WSo/e83gOyyy742E8GSs771ySpg= github.com/googleapis/enterprise-certificate-proxy v0.2.1/go.mod h1:AwSRAtLfXpU5Nm3pW+v7rGDHp09LsPtGY9MduiEsR9k= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk= +github.com/googleapis/gax-go/v2 v2.1.0/go.mod h1:Q3nei7sK6ybPYH7twZdmQpAd1MKb7pfu6SK+H1/DsU0= +github.com/googleapis/gax-go/v2 v2.1.1/go.mod h1:hddJymUZASv3XPyGkUpKj8pPO47Rmb0eJc8R6ouapiM= +github.com/googleapis/gax-go/v2 v2.2.0/go.mod h1:as02EH8zWkzwUoLbBaFeQ+arQaj/OthfcblKl4IGNaM= +github.com/googleapis/gax-go/v2 v2.3.0/go.mod h1:b8LNqSzNabLiUpXKkY7HAR5jr6bIT99EXz9pXxye9YM= +github.com/googleapis/gax-go/v2 v2.4.0/go.mod h1:XOTVJ59hdnfJLIP/dh8n5CGryZR2LxK9wbMD5+iXC6c= +github.com/googleapis/gax-go/v2 v2.5.1/go.mod h1:h6B0KMMFNtI2ddbGJn3T3ZbwkeT6yqEF02fYlzkUCyo= +github.com/googleapis/gax-go/v2 v2.6.0/go.mod h1:1mjbznJAPHFpesgE5ucqfYEscaz5kMdcIDwU/6+DDoY= github.com/googleapis/gax-go/v2 v2.7.0 h1:IcsPKeInNvYi7eqSaDjiZqDDKu5rsmunY0Y1YupQSSQ= github.com/googleapis/gax-go/v2 v2.7.0/go.mod h1:TEop28CZZQ2y+c0VxMUmu1lV+fQx57QpBWsYpwqHJx8= github.com/googleapis/gnostic v0.4.1/go.mod h1:LRhVm6pbyptWbWbuZ38d1eyptfvIytN3ir6b65WBswg= +github.com/googleapis/go-type-adapters v1.0.0/go.mod h1:zHW75FOG2aur7gAO2B+MLby+cLsWGBF62rFAi7WjWO4= github.com/gookit/color v1.2.5/go.mod h1:AhIE+pS6D4Ql0SQWbBeXPHw7gY0/sjHoA4s/n1KB7xg= github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= github.com/gorilla/handlers v0.0.0-20150720190736-60c7bfde3e33/go.mod h1:Qkdc/uu4tH4g6mTK6auzZ766c4CA0Ng8+o/OAirnOIQ= @@ -874,8 +1027,8 @@ github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtng github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80= github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= -github.com/hashicorp/go-getter v1.6.2 h1:7jX7xcB+uVCliddZgeKyNxv0xoT7qL5KDtH7rU4IqIk= -github.com/hashicorp/go-getter v1.6.2/go.mod h1:IZCrswsZPeWv9IkVnLElzRU/gz/QPi6pZHn4tv6vbwA= +github.com/hashicorp/go-getter v1.7.0 h1:bzrYP+qu/gMrL1au7/aDvkoOVGUJpeKBgbqRHACAFDY= +github.com/hashicorp/go-getter v1.7.0/go.mod h1:W7TalhMmbPmsSMdNjD0ZskARur/9GJ17cfHTRtXV744= github.com/hashicorp/go-hclog v1.3.1 h1:vDwF1DFNZhntP4DAjuTpOw3uEgMUpXh1pB5fW9DqHpo= github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60= github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM= @@ -892,7 +1045,6 @@ github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdv github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= github.com/hashicorp/go-uuid v1.0.2/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro= -github.com/hashicorp/go-version v1.1.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= github.com/hashicorp/go-version v1.6.0 h1:feTTfFNnjP967rlCxM/I9g701jU+RN74YKx2mOkIeek= github.com/hashicorp/go-version v1.6.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90= @@ -1011,12 +1163,12 @@ github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.10.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.10.10/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= -github.com/klauspost/compress v1.11.2/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= github.com/klauspost/compress v1.15.0/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU= +github.com/klauspost/compress v1.15.11/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM= github.com/klauspost/compress v1.15.12/go.mod h1:QPwzmACJjUTFsnSHH934V6woptycfrDDJnH7hvFVbGM= github.com/klauspost/compress v1.15.15 h1:EF27CXIuDsYJ6mmvtBRlEuB2UVOqHG1tAXgZ7yIO+lw= github.com/klauspost/compress v1.15.15/go.mod h1:ZcK2JAFqKOpnBlxcLsJzYfrS9X1akm9fHZNnD9+Vo/4= @@ -1159,8 +1311,8 @@ github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= github.com/mitchellh/go-ps v1.0.0/go.mod h1:J4lOc8z8yJs6vUwklHw2XEIiT4z4C40KtWVN3nvg8Pg= github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI= -github.com/mitchellh/go-testing-interface v1.14.0 h1:/x0XQ6h+3U3nAyk1yx+bHPURrKa9sVVvYbuqZ7pIAtI= -github.com/mitchellh/go-testing-interface v1.14.0/go.mod h1:gfgS7OtZj6MA4U1UrDRp04twqAjfvlZyCfX3sDjEym8= +github.com/mitchellh/go-testing-interface v1.14.1 h1:jrgshOhYAUVNMAJiKbEu7EqAwgJJ2JqpQmpLJOu07cU= +github.com/mitchellh/go-testing-interface v1.14.1/go.mod h1:gfgS7OtZj6MA4U1UrDRp04twqAjfvlZyCfX3sDjEym8= github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/go-wordwrap v1.0.1 h1:TLuKupo69TCn6TQSyGxwI1EblZZEsQ0vMlAFQflz0v0= github.com/mitchellh/go-wordwrap v1.0.1/go.mod h1:R62XHJLzvMFRBbcrT7m7WgmE1eOyTSsCt+hzestvNj0= @@ -1553,7 +1705,7 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 h1:uruHq4 github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tommy-muehle/go-mnd v1.3.1-0.20200224220436-e6f9a994e8fa/go.mod h1:dSUh0FtTP8VhvkL1S+gUR1OKd9ZnSaozuI6r3m6wOig= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= -github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= +github.com/ulikunitz/xz v0.5.10/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8= github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/ultraware/funlen v0.0.3/go.mod h1:Dp4UiAus7Wdb9KUZsYWZEWiRzGuM2kXM1lPbfaF6xhA= @@ -1861,6 +2013,7 @@ golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLd golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k= golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= +golang.org/x/net v0.0.0-20210503060351-7fd8e65b6420/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210726213435-c6fcb2dbf985/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= @@ -1869,9 +2022,18 @@ golang.org/x/net v0.0.0-20210917221730-978cfadd31cf/go.mod h1:9nx3DQGgdP8bBQD5qx golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220325170049-de3da57026de/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220412020605-290c469a71a5/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk= +golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.0.0-20220617184016-355a448f1bc9/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.0.0-20220624214902-1bab6f366d9e/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= golang.org/x/net v0.0.0-20220906165146-f3363e06e74c/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= +golang.org/x/net v0.0.0-20220909164309-bea034e7d591/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= +golang.org/x/net v0.0.0-20221014081412-f15817d10f9b/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk= +golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco= golang.org/x/net v0.2.0/go.mod h1:KqCZLdyyvdV855qA2rE3GC2aiw5xGR5TEjj8smXukLY= golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws= golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= @@ -1889,8 +2051,19 @@ golang.org/x/oauth2 v0.0.0-20210220000619-9bb904979d93/go.mod h1:KelEdhl1UZF7XfJ golang.org/x/oauth2 v0.0.0-20210313182246-cd4f82c27b84/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20210514164344-f6687ab2804c/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.0.0-20210628180205-a41e5a781914/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.0.0-20210805134026-6f1e6394065a/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= +golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A= golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= +golang.org/x/oauth2 v0.0.0-20220309155454-6242fa91716a/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= +golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5/go.mod h1:DAh4E804XQdzx2j+YRIaUnCqCV2RuMz24cGBJ5QYIrc= +golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE= +golang.org/x/oauth2 v0.0.0-20220622183110-fd043fe589d2/go.mod h1:jaDAt6Dkxork7LmZnYtzbRWj0W47D86a3TGe0YHBvmE= +golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.0.0-20220909003341-f21342109be1/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783/go.mod h1:h4gKUeWbJ4rQPri7E0u6Gs4e9Ri2zaLxzw5DI5XGrYg= +golang.org/x/oauth2 v0.1.0/go.mod h1:G9FE4dLTsbXUu90h/Pf85g4w1D+SSAgR+q46nJZ8M4A= golang.org/x/oauth2 v0.4.0 h1:NF0gk8LVPg1Ml7SSbGyySuoxdsXitj7TvgvuRxIMc/M= golang.org/x/oauth2 v0.4.0/go.mod h1:RznEsdpjGAINPTOF0UH/t+xJ75L18YO3Ho6Pyn+uRec= golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -1904,7 +2077,9 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220601150217-0de741cfad7f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220929204114-8fcdb60fdcc0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o= golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -2000,27 +2175,43 @@ golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210426230700-d19ff857e887/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210502180810-71e4cd670f79/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210823070655-63515b42dcdf/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210908233432-aa78b53d3365/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211116061358-0a5406a5449c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211124211545-fe61309f8881/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20211210111614-af8b64212486/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220310020820-b874c991c1a5/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220517195934-5e4e11fc645e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220502124256-b6088ccd6cba/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220610221304-9f5ed59c137d/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220624220833-87e55d714810/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220825204002-c680a09ffe64/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= @@ -2030,6 +2221,7 @@ golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXR golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.0.0-20220722155259-a9ba230a4035/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.2.0/go.mod h1:TVmDHMZPmdnySmBfhjOoOdhjzdE1h4u1VwSiw2l1Nuc= golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ= golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY= @@ -2139,6 +2331,9 @@ golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4f golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0= golang.org/x/tools v0.1.1/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= +golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.6-0.20210726203631-07bc1bf47fb2/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo= golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= @@ -2148,6 +2343,9 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20220411194840-2f41105eb62f/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= +golang.org/x/xerrors v0.0.0-20220609144429-65e65417b02f/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 h1:H2TDz8ibqkAF6YGhCdN3jS9O0/s90v0rJh3X/OLHEUk= golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2/go.mod h1:K8+ghG5WaK9qNqU5K3HdILfMLy1f3aNYFI/wnl100a8= gomodules.xyz/jsonpatch/v2 v2.2.0 h1:4pT439QV83L+G9FkcCriY6EkpcK6r6bK+A5FBUMI7qY= @@ -2175,6 +2373,33 @@ google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjR google.golang.org/api v0.41.0/go.mod h1:RkxM5lITDfTzmyKFPt+wGrCJbVfniCr2ool8kTBzRTU= google.golang.org/api v0.43.0/go.mod h1:nQsDGjRXMo4lvh5hP0TKqF244gqhGcr/YSIykhUk/94= google.golang.org/api v0.44.0/go.mod h1:EBOGZqzyhtvMDoxwS97ctnh0zUmYY6CxqXsc1AvkYD8= +google.golang.org/api v0.47.0/go.mod h1:Wbvgpq1HddcWVtzsVLyfLp8lDg6AA241LmgIL59tHXo= +google.golang.org/api v0.48.0/go.mod h1:71Pr1vy+TAZRPkPs/xlCf5SsU8WjuAWv1Pfjbtukyy4= +google.golang.org/api v0.50.0/go.mod h1:4bNT5pAuq5ji4SRZm+5QIkjny9JAyVD/3gaSihNefaw= +google.golang.org/api v0.51.0/go.mod h1:t4HdrdoNgyN5cbEfm7Lum0lcLDLiise1F8qDKX00sOU= +google.golang.org/api v0.54.0/go.mod h1:7C4bFFOvVDGXjfDTAsgGwDgAxRDeQ4X8NvUedIt6z3k= +google.golang.org/api v0.55.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= +google.golang.org/api v0.56.0/go.mod h1:38yMfeP1kfjsl8isn0tliTjIb1rJXcQi4UXlbqivdVE= +google.golang.org/api v0.57.0/go.mod h1:dVPlbZyBo2/OjBpmvNdpn2GRm6rPy75jyU7bmhdrMgI= +google.golang.org/api v0.61.0/go.mod h1:xQRti5UdCmoCEqFxcz93fTl338AVqDgyaDRuOZ3hg9I= +google.golang.org/api v0.63.0/go.mod h1:gs4ij2ffTRXwuzzgJl/56BdwJaA194ijkfn++9tDuPo= +google.golang.org/api v0.67.0/go.mod h1:ShHKP8E60yPsKNw/w8w+VYaj9H6buA5UqDp8dhbQZ6g= +google.golang.org/api v0.70.0/go.mod h1:Bs4ZM2HGifEvXwd50TtW70ovgJffJYw2oRCOFU/SkfA= +google.golang.org/api v0.71.0/go.mod h1:4PyU6e6JogV1f9eA4voyrTY2batOLdgZ5qZ5HOCc4j8= +google.golang.org/api v0.74.0/go.mod h1:ZpfMZOVRMywNyvJFeqL9HRWBgAuRfSjJFpe9QtRRyDs= +google.golang.org/api v0.75.0/go.mod h1:pU9QmyHLnzlpar1Mjt4IbapUCy8J+6HD6GeELN69ljA= +google.golang.org/api v0.77.0/go.mod h1:pU9QmyHLnzlpar1Mjt4IbapUCy8J+6HD6GeELN69ljA= +google.golang.org/api v0.78.0/go.mod h1:1Sg78yoMLOhlQTeF+ARBoytAcH1NNyyl390YMy6rKmw= +google.golang.org/api v0.80.0/go.mod h1:xY3nI94gbvBrE0J6NHXhxOmW97HG7Khjkku6AFB3Hyg= +google.golang.org/api v0.84.0/go.mod h1:NTsGnUFJMYROtiquksZHBWtHfeMC7iYthki7Eq3pa8o= +google.golang.org/api v0.85.0/go.mod h1:AqZf8Ep9uZ2pyTvgL+x0D3Zt0eoT9b5E8fmzfu6FO2g= +google.golang.org/api v0.90.0/go.mod h1:+Sem1dnrKlrXMR/X0bPnMWyluQe4RsNoYfmNLhOIkzw= +google.golang.org/api v0.93.0/go.mod h1:+Sem1dnrKlrXMR/X0bPnMWyluQe4RsNoYfmNLhOIkzw= +google.golang.org/api v0.95.0/go.mod h1:eADj+UBuxkh5zlrSntJghuNeg8HwQ1w5lTKkuqaETEI= +google.golang.org/api v0.96.0/go.mod h1:w7wJQLTM+wvQpNf5JyEcBoxK0RH7EDrh/L4qfsuJ13s= +google.golang.org/api v0.97.0/go.mod h1:w7wJQLTM+wvQpNf5JyEcBoxK0RH7EDrh/L4qfsuJ13s= +google.golang.org/api v0.98.0/go.mod h1:w7wJQLTM+wvQpNf5JyEcBoxK0RH7EDrh/L4qfsuJ13s= +google.golang.org/api v0.100.0/go.mod h1:ZE3Z2+ZOr87Rx7dqFsdRQkRBk36kDtp/h+QpHbB7a70= google.golang.org/api v0.107.0 h1:I2SlFjD8ZWabaIFOfeEDg3pf0BHJDh6iYQ1ic3Yu/UU= google.golang.org/api v0.107.0/go.mod h1:2Ts0XTHNVWxypznxWOYUeI4g3WdP9Pk2Qk58+a/O9MY= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= @@ -2230,10 +2455,69 @@ google.golang.org/genproto v0.0.0-20210222152913-aa3ee6e6a81c/go.mod h1:FWY/as6D google.golang.org/genproto v0.0.0-20210303154014-9728d6b83eeb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210310155132-4ce2db91004e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= google.golang.org/genproto v0.0.0-20210319143718-93e7006c17a6/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no= +google.golang.org/genproto v0.0.0-20210329143202-679c6ae281ee/go.mod h1:9lPAdzaEmUacj36I+k7YKbEc5CXzPIeORRgDAUOu28A= google.golang.org/genproto v0.0.0-20210402141018-6c239bbf2bb1/go.mod h1:9lPAdzaEmUacj36I+k7YKbEc5CXzPIeORRgDAUOu28A= +google.golang.org/genproto v0.0.0-20210513213006-bf773b8c8384/go.mod h1:P3QM42oQyzQSnHPnZ/vqoCdDmzH28fzWByN9asMeM8A= google.golang.org/genproto v0.0.0-20210602131652-f16073e35f0c/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= +google.golang.org/genproto v0.0.0-20210604141403-392c879c8b08/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= +google.golang.org/genproto v0.0.0-20210608205507-b6d2f5bf0d7d/go.mod h1:UODoCrxHCcBojKKwX1terBiRUaqAsFqJiF615XL43r0= +google.golang.org/genproto v0.0.0-20210624195500-8bfb893ecb84/go.mod h1:SzzZ/N+nwJDaO1kznhnlzqS8ocJICar6hYhVyhi++24= +google.golang.org/genproto v0.0.0-20210713002101-d411969a0d9a/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k= +google.golang.org/genproto v0.0.0-20210716133855-ce7ef5c701ea/go.mod h1:AxrInvYm1dci+enl5hChSFPOmmUF1+uAa/UsgNRWd7k= +google.golang.org/genproto v0.0.0-20210728212813-7823e685a01f/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= +google.golang.org/genproto v0.0.0-20210805201207-89edb61ffb67/go.mod h1:ob2IJxKrgPT52GcgX759i1sleT07tiKowYBGbczaW48= +google.golang.org/genproto v0.0.0-20210813162853-db860fec028c/go.mod h1:cFeNkxwySK631ADgubI+/XFU/xp8FD5KIVV4rj8UC5w= +google.golang.org/genproto v0.0.0-20210821163610-241b8fcbd6c8/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210828152312-66f60bf46e71/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210831024726-fe130286e0e2/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af/go.mod h1:eFjDcFEctNawg4eG61bRv87N7iHBWyVhJu7u1kqDUXY= +google.golang.org/genproto v0.0.0-20210924002016-3dee208752a0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20211118181313-81c1377c94b1/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211206160659-862468c7d6e0/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20211221195035-429b39de9b1c/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= google.golang.org/genproto v0.0.0-20220107163113-42d7afdf6368/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20220126215142-9970aeb2e350/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20220207164111-0872dc986b00/go.mod h1:5CzLGKJ67TSI2B9POpiiyGha0AjJvZIUgRMt1dSmuhc= +google.golang.org/genproto v0.0.0-20220218161850-94dd64e39d7c/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= +google.golang.org/genproto v0.0.0-20220222213610-43724f9ea8cf/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= +google.golang.org/genproto v0.0.0-20220304144024-325a89244dc8/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= +google.golang.org/genproto v0.0.0-20220310185008-1973136f34c6/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI= +google.golang.org/genproto v0.0.0-20220324131243-acbaeb5b85eb/go.mod h1:hAL49I2IFola2sVEjAn7MEwsja0xp51I0tlGAf9hz4E= +google.golang.org/genproto v0.0.0-20220407144326-9054f6ed7bac/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220414192740-2d67ff6cf2b4/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220421151946-72621c1f0bd3/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220429170224-98d788798c3e/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo= +google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4= +google.golang.org/genproto v0.0.0-20220505152158-f39f71e6c8f3/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4= +google.golang.org/genproto v0.0.0-20220518221133-4f43b3371335/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4= +google.golang.org/genproto v0.0.0-20220523171625-347a074981d8/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4= +google.golang.org/genproto v0.0.0-20220608133413-ed9918b62aac/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA= +google.golang.org/genproto v0.0.0-20220616135557-88e70c0c3a90/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA= +google.golang.org/genproto v0.0.0-20220617124728-180714bec0ad/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA= +google.golang.org/genproto v0.0.0-20220624142145-8cd45d7dbd1f/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA= +google.golang.org/genproto v0.0.0-20220628213854-d9e0b6570c03/go.mod h1:KEWEmljWE5zPzLBa/oHl6DaEt9LmfH6WtH1OHIvleBA= +google.golang.org/genproto v0.0.0-20220722212130-b98a9ff5e252/go.mod h1:GkXuJDJ6aQ7lnJcRF+SJVgFdQhypqgl3LB1C9vabdRE= +google.golang.org/genproto v0.0.0-20220801145646-83ce21fca29f/go.mod h1:iHe1svFLAZg9VWz891+QbRMwUv9O/1Ww+/mngYeThbc= +google.golang.org/genproto v0.0.0-20220815135757-37a418bb8959/go.mod h1:dbqgFATTzChvnt+ujMdZwITVAJHFtfyN1qUhDqEiIlk= +google.golang.org/genproto v0.0.0-20220817144833-d7fd3f11b9b1/go.mod h1:dbqgFATTzChvnt+ujMdZwITVAJHFtfyN1qUhDqEiIlk= +google.golang.org/genproto v0.0.0-20220822174746-9e6da59bd2fc/go.mod h1:dbqgFATTzChvnt+ujMdZwITVAJHFtfyN1qUhDqEiIlk= +google.golang.org/genproto v0.0.0-20220829144015-23454907ede3/go.mod h1:dbqgFATTzChvnt+ujMdZwITVAJHFtfyN1qUhDqEiIlk= +google.golang.org/genproto v0.0.0-20220829175752-36a9c930ecbf/go.mod h1:dbqgFATTzChvnt+ujMdZwITVAJHFtfyN1qUhDqEiIlk= +google.golang.org/genproto v0.0.0-20220913154956-18f8339a66a5/go.mod h1:0Nb8Qy+Sk5eDzHnzlStwW3itdNaWoZA5XeSG+R3JHSo= +google.golang.org/genproto v0.0.0-20220914142337-ca0e39ece12f/go.mod h1:0Nb8Qy+Sk5eDzHnzlStwW3itdNaWoZA5XeSG+R3JHSo= +google.golang.org/genproto v0.0.0-20220915135415-7fd63a7952de/go.mod h1:0Nb8Qy+Sk5eDzHnzlStwW3itdNaWoZA5XeSG+R3JHSo= +google.golang.org/genproto v0.0.0-20220916172020-2692e8806bfa/go.mod h1:0Nb8Qy+Sk5eDzHnzlStwW3itdNaWoZA5XeSG+R3JHSo= +google.golang.org/genproto v0.0.0-20220919141832-68c03719ef51/go.mod h1:0Nb8Qy+Sk5eDzHnzlStwW3itdNaWoZA5XeSG+R3JHSo= +google.golang.org/genproto v0.0.0-20220920201722-2b89144ce006/go.mod h1:ht8XFiar2npT/g4vkk7O0WYS1sHOHbdujxbEp7CJWbw= +google.golang.org/genproto v0.0.0-20220926165614-551eb538f295/go.mod h1:woMGP53BroOrRY3xTxlbr8Y3eB/nzAvvFM83q7kG2OI= +google.golang.org/genproto v0.0.0-20220926220553-6981cbe3cfce/go.mod h1:woMGP53BroOrRY3xTxlbr8Y3eB/nzAvvFM83q7kG2OI= +google.golang.org/genproto v0.0.0-20221010155953-15ba04fc1c0e/go.mod h1:3526vdqwhZAwq4wsRUaVG555sVgsNmIjRtO7t/JH29U= +google.golang.org/genproto v0.0.0-20221014173430-6e2ab493f96b/go.mod h1:1vXfmgAz9N9Jx0QA82PqRVauvCz1SGSz739p0f183jM= +google.golang.org/genproto v0.0.0-20221014213838-99cd37c6964a/go.mod h1:1vXfmgAz9N9Jx0QA82PqRVauvCz1SGSz739p0f183jM= +google.golang.org/genproto v0.0.0-20221025140454-527a21cfbd71/go.mod h1:9qHF0xnpdSfF6knlcsnpzUu5y+rpwgbvsyGAZPBMg4s= google.golang.org/genproto v0.0.0-20230104163317-caabf589fcbf h1:/JqRexUvugu6JURQ0O7RfV1EnvgrOxUV4tSjuAv0Sr0= google.golang.org/genproto v0.0.0-20230104163317-caabf589fcbf/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM= google.golang.org/grpc v0.0.0-20160317175043-d3ddb4469d5a/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= @@ -2260,14 +2544,27 @@ google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA5 google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= google.golang.org/grpc v1.36.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= google.golang.org/grpc v1.36.1/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU= +google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= google.golang.org/grpc v1.37.1/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM= +google.golang.org/grpc v1.39.0/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= +google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnDzfrE= google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= +google.golang.org/grpc v1.40.1/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34= google.golang.org/grpc v1.41.0/go.mod h1:U3l9uK9J0sini8mHphKoXyaqDA/8VyGnDee1zzIUK6k= google.golang.org/grpc v1.42.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= +google.golang.org/grpc v1.44.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU= +google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ= google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= +google.golang.org/grpc v1.46.2/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= +google.golang.org/grpc v1.47.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= +google.golang.org/grpc v1.48.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk= +google.golang.org/grpc v1.49.0/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI= +google.golang.org/grpc v1.50.0/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI= +google.golang.org/grpc v1.50.1/go.mod h1:ZgQEeidpAuNRZ8iRrlBKXZQP1ghovWIVhdJRyCDK+GI= google.golang.org/grpc v1.52.0 h1:kd48UiU7EHsV4rnLyOJRuP/Il/UHE7gdDAQ+SZI7nZk= google.golang.org/grpc v1.52.0/go.mod h1:pu6fVzoFb+NBYNAvQL08ic+lvB2IojljRYuun5vorUY= +google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw= google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= From f126115c805b538ac010385f91271cf14f3a49fc Mon Sep 17 00:00:00 2001 From: zhangtao <111836083+sophon-zt@users.noreply.github.com> Date: Wed, 12 Apr 2023 17:17:41 +0800 Subject: [PATCH 80/80] feat: update pg14/15 parameters constraint (#2160) (#2322) --- .../apps/configuration/sync_upgrade_policy.go | 6 +- .../config/pg14-config-constraint.cue | 66 +++++++++++++++---- .../config/pg14-config-effect-scope.yaml | 42 +++++++++++- .../templates/configconstraint.yaml | 6 ++ 4 files changed, 103 insertions(+), 17 deletions(-) diff --git a/controllers/apps/configuration/sync_upgrade_policy.go b/controllers/apps/configuration/sync_upgrade_policy.go index 6cfed37b7..bbf8c98f9 100644 --- a/controllers/apps/configuration/sync_upgrade_policy.go +++ b/controllers/apps/configuration/sync_upgrade_policy.go @@ -17,6 +17,8 @@ limitations under the License. package configuration import ( + "fmt" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" @@ -104,13 +106,13 @@ func sync(params reconfigureParams, updatedParameters map[string]string, pods [] return makeReturnedStatus(ESAndRetryFailed), err } if len(pods) == 0 { - params.Ctx.Log.Info("no pods to update, and retry, selector: %v, current all pod: %v", params.ConfigConstraint.Selector) + params.Ctx.Log.Info(fmt.Sprintf("no pods to update, and retry, selector: %s", params.ConfigConstraint.Selector.String())) return makeReturnedStatus(ESRetry), nil } requireUpdatedCount := int32(len(pods)) for _, pod := range pods { - params.Ctx.Log.V(1).Info("sync pod: %s", pod.Name) + params.Ctx.Log.V(1).Info(fmt.Sprintf("sync pod: %s", pod.Name)) if podutil.IsMatchConfigVersion(&pod, configKey, versionHash) { progress++ continue diff --git a/deploy/postgresql/config/pg14-config-constraint.cue b/deploy/postgresql/config/pg14-config-constraint.cue index 31f7f924e..1d735c145 100644 --- a/deploy/postgresql/config/pg14-config-constraint.cue +++ b/deploy/postgresql/config/pg14-config-constraint.cue @@ -14,18 +14,26 @@ // PostgreSQL parameters: https://postgresqlco.nf/doc/en/param/ #PGParameter: { + // Allows tablespaces directly inside pg_tblspc, for testing, pg version: 15 + allow_in_place_tablespaces?: bool + // Allows modification of the structure of system tables as well as certain other risky actions on system tables. This is otherwise not allowed even for superusers. Ill-advised use of this setting can cause irretrievable data loss or seriously corrupt the database system. + allow_system_table_mods?: bool // Sets the application name to be reported in statistics and logs. application_name?: string // Sets the shell command that will be called to archive a WAL file. archive_command?: string + // The library to use for archiving completed WAL file segments. If set to an empty string (the default), archiving via shell is enabled, and archive_command is used. Otherwise, the specified shared library is used for archiving. The WAL archiver process is restarted by the postmaster when this parameter changes. For more information, see backup-archiving-wal and archive-modules. + archive_library?: string + // When archive_mode is enabled, completed WAL segments are sent to archive storage by setting archive_command or guc-archive-library. In addition to off, to disable, there are two modes: on, and always. During normal operation, there is no difference between the two modes, but when set to always the WAL archiver is enabled also during archive recovery or standby mode. In always mode, all files restored from the archive or streamed with streaming replication will be archived (again). See continuous-archiving-in-standby for details. + archive_mode: string & "always" | "on" | "off" // (s) Forces a switch to the next xlog file if a new file has not been started within N seconds. archive_timeout: int & >=0 & <=2147483647 | *300 @timeDurationResource(1s) // Enable input of NULL elements in arrays. - array_nulls?: bool & false | true + array_nulls?: bool // (s) Sets the maximum allowed time to complete client authentication. - authentication_timeout?: int & >=1 & <=600 @timeDurationResource() + authentication_timeout?: int & >=1 & <=600 @timeDurationResource(1s) // Use EXPLAIN ANALYZE for plan logging. - "auto_explain.log_analyze"?: bool & false | true + "auto_explain.log_analyze"?: bool // Log buffers usage. "auto_explain.log_buffers"?: bool & false | true // EXPLAIN format to be used for plan logging. @@ -50,7 +58,7 @@ "auto_explain.sample_rate"?: float & >=0 & <=1 // Starts the autovacuum subprocess. - autovacuum?: bool & false | true + autovacuum?: bool // Number of tuple inserts, updates or deletes prior to analyze as a fraction of reltuples. autovacuum_analyze_scale_factor: float & >=0 & <=100 | *0.05 @@ -59,7 +67,7 @@ autovacuum_analyze_threshold?: int & >=0 & <=2147483647 // Age at which to autovacuum a table to prevent transaction ID wraparound. - autovacuum_freeze_max_age?: int & >=100000000 & <=750000000 + autovacuum_freeze_max_age?: int & >=100000 & <=2000000000 // Sets the maximum number of simultaneously running autovacuum worker processes. autovacuum_max_workers?: int & >=1 & <=8388607 @@ -71,7 +79,7 @@ autovacuum_naptime: int & >=1 & <=2147483 | *15 @timeDurationResource(1s) // (ms) Vacuum cost delay in milliseconds, for autovacuum. - autovacuum_vacuum_cost_delay?: int & >=-1 & <=100 + autovacuum_vacuum_cost_delay?: int & >=-1 & <=100 @timeDurationResource() // Vacuum cost amount available before napping, for autovacuum. autovacuum_vacuum_cost_limit?: int & >=-1 & <=10000 @@ -92,9 +100,9 @@ autovacuum_work_mem?: int & >=-1 & <=2147483647 @storeResource(1KB) // (8Kb) Number of pages after which previously performed writes are flushed to disk. - backend_flush_after?: int & >=0 & <=256 + backend_flush_after?: int & >=0 & <=256 @storeResource(8KB) - // Sets whether \ is allowed in string literals. + // Sets whether "\" is allowed in string literals. backslash_quote?: string & "safe_encoding" | "on" | "off" // Log backtrace for errors in these functions. @@ -104,7 +112,7 @@ bgwriter_delay?: int & >=10 & <=10000 @timeDurationResource() // (8Kb) Number of pages after which previously performed writes are flushed to disk. - bgwriter_flush_after?: int & >=0 & <=256 + bgwriter_flush_after?: int & >=0 & <=256 @storeResource(8KB) // Background writer maximum number of LRU pages to flush per round. bgwriter_lru_maxpages?: int & >=0 & <=1000 @@ -352,6 +360,9 @@ // Use of huge pages on Linux. huge_pages?: string & "on" | "off" | "try" + // The size of huge page that should be requested. Controls the size of huge pages, when they are enabled with huge_pages. The default is zero (0). When set to 0, the default huge page size on the system will be used. This parameter can only be set at server start. + huge_page_size?: int & >=0 & <=2147483647 @storeResource(1KB) + // Sets the servers ident configuration file. ident_file?: string @@ -368,7 +379,7 @@ intervalstyle?: string & "postgres" | "postgres_verbose" | "sql_standard" | "iso_8601" // Allow JIT compilation. - jit: bool & false | true | *false + jit: bool // Perform JIT compilation if query is more expensive. jit_above_cost?: float & >=-1 & <=1.79769e+308 @@ -484,6 +495,9 @@ // (kB) Automatic log file rotation will occur after N kilobytes. log_rotation_size?: int & >=0 & <=2097151 @storeResource(1KB) + // Time between progress updates for long-running startup operations. Sets the amount of time after which the startup process will log a message about a long-running operation that is still in progress, as well as the interval between further progress messages for that operation. The default is 10 seconds. A setting of 0 disables the feature. If this value is specified without units, it is taken as milliseconds. This setting is applied separately to each operation. This parameter can only be set in the postgresql.conf file or on the server command line. + log_startup_progress_interval: int & >=0 & <=2147483647 @timeDurationResource() + // Sets the type of statements logged. log_statement?: string & "none" | "ddl" | "mod" | "all" @@ -491,7 +505,7 @@ log_statement_sample_rate?: float & >=0 & <=1 // Writes cumulative performance statistics to the server log. - log_statement_stats?: bool & false | true + log_statement_stats?: bool // (kB) Log the use of temporary files larger than this number of kilobytes. log_temp_files?: int & >=-1 & <=2147483647 @storeResource(1KB) @@ -577,6 +591,9 @@ // (8kB) Sets the minimum amount of index data for a parallel scan. min_parallel_index_scan_size?: int & >=0 & <=715827882 @storeResource(8KB) + // Sets the minimum size of relations to be considered for parallel scan. Sets the minimum size of relations to be considered for parallel scan. + min_parallel_relation_size?: int & >=0 & <=715827882 @storeResource(8KB) + // (8kB) Sets the minimum amount of table data for a parallel scan. min_parallel_table_scan_size?: int & >=0 & <=715827882 @storeResource(8KB) @@ -584,7 +601,7 @@ min_wal_size: int & >=128 & <=201326592 | *192 @storeResource(1MB) // (min) Time before a snapshot is too old to read pages changed after the snapshot was taken. - old_snapshot_threshold?: int & >=-1 & <=86400 + old_snapshot_threshold?: int & >=-1 & <=86400 @timeDurationResource(1min) // Emulate oracle's date output behaviour. "orafce.nls_date_format"?: string @@ -838,6 +855,12 @@ // Sets the TCP port the server listens on. port?: int & >=1 & <=65535 + // Sets the amount of time to wait after authentication on connection startup. The amount of time to delay when a new server process is started, after it conducts the authentication procedure. This is intended to give developers an opportunity to attach to the server process with a debugger. If this value is specified without units, it is taken as seconds. A value of zero (the default) disables the delay. This parameter cannot be changed after session start. + post_auth_delay?: int & >=0 & <=2147 @timeDurationResource(1s) + + // Sets the amount of time to wait before authentication on connection startup. The amount of time to delay just after a new server process is forked, before it conducts the authentication procedure. This is intended to give developers an opportunity to attach to the server process with a debugger to trace down misbehavior in authentication. If this value is specified without units, it is taken as seconds. A value of zero (the default) disables the delay. This parameter can only be set in the postgresql.conf file or on the server command line. + pre_auth_delay?: int & >=0 & <=60 @timeDurationResource(1s) + // Enable for disable GDAL drivers used with PostGIS in Postgres 9.3.5 and above. "postgis.gdal_enabled_drivers"?: string & "ENABLE_ALL" | "DISABLE_ALL" @@ -938,11 +961,14 @@ // (s) Time between TCP keepalive retransmits. tcp_keepalives_interval?: int & >=0 & <=2147483647 @timeDurationResource(1s) + // TCP user timeout. Specifies the amount of time that transmitted data may remain unacknowledged before the TCP connection is forcibly closed. If this value is specified without units, it is taken as milliseconds. A value of 0 (the default) selects the operating system's default. This parameter is supported only on systems that support TCP_USER_TIMEOUT; on other systems, it must be zero. In sessions connected via a Unix-domain socket, this parameter is ignored and always reads as zero. + tcp_user_timeout?: int & >=0 & <=2147483647 @timeDurationResource() + // (8kB) Sets the maximum number of temporary buffers used by each session. temp_buffers?: int & >=100 & <=1073741823 @storeResource(8KB) // (kB) Limits the total size of all temporary files used by each process. - temp_file_limit?: int & >=-1 & <=2147483647 @storeResource(8KB) + temp_file_limit?: int & >=-1 & <=2147483647 @storeResource(1KB) // Sets the tablespace(s) to use for temporary tables and sort files. temp_tablespaces?: string @@ -954,7 +980,7 @@ track_activities?: bool & false | true // Sets the size reserved for pg_stat_activity.current_query, in bytes. - track_activity_query_size: int & >=100 & <=1048576 | *4096 + track_activity_query_size: int & >=100 & <=1048576 | *4096 @storeResource() // Collects transaction commit time. track_commit_timestamp?: bool & false | true @@ -1028,6 +1054,12 @@ // Compresses full-page writes written in WAL file. wal_compression: bool & false | true | *true + // Sets the WAL resource managers for which WAL consistency checks are done. + wal_consistency_checking?: string + + // Buffer size for reading ahead in the WAL during recovery. + wal_decode_buffer_size: int & >=65536 & <=1073741823 | *524288 @storeResource() + // (MB) Sets the size of WAL files held for standby servers. wal_keep_size: int & >=0 & <=2147483647 | *2048 @storeResource(1MB) @@ -1040,6 +1072,12 @@ // (ms) Sets the maximum wait time to receive data from the primary. wal_receiver_timeout: int & >=0 & <=3600000 | *30000 @timeDurationResource() + // Recycles WAL files by renaming them. If set to on (the default), this option causes WAL files to be recycled by renaming them, avoiding the need to create new ones. On COW file systems, it may be faster to create new ones, so the option is given to disable this behavior. + wal_recycle?: bool + + // Sets the time to wait before retrying to retrieve WAL after a failed attempt. Specifies how long the standby server should wait when WAL data is not available from any sources (streaming replication, local pg_wal or WAL archive) before trying again to retrieve WAL data. If this value is specified without units, it is taken as milliseconds. The default value is 5 seconds. This parameter can only be set in the postgresql.conf file or on the server command line. + wal_retrieve_retry_interval: int & >=1 & <=2147483647 | *5000 @timeDurationResource() + // (ms) Sets the maximum time to wait for WAL replication. wal_sender_timeout: int & >=0 & <=3600000 | *30000 @timeDurationResource() diff --git a/deploy/postgresql/config/pg14-config-effect-scope.yaml b/deploy/postgresql/config/pg14-config-effect-scope.yaml index 6e4d69046..e103dfaa9 100644 --- a/deploy/postgresql/config/pg14-config-effect-scope.yaml +++ b/deploy/postgresql/config/pg14-config-effect-scope.yaml @@ -19,4 +19,44 @@ staticParameters: - bg_mon.history_buckets - pg_stat_statements.track_utility - extwlist.extensions - - extwlist.custom_path \ No newline at end of file + - extwlist.custom_path + +immutableParameters: + - archive_command + - archive_timeout + - backtrace_functions + - config_file + - cron.use_background_workers + - data_directory + - db_user_namespace + - exit_on_error + - fsync + - full_page_writes + - hba_file + - ident_file + - ignore_invalid_pages + - listen_addresses + - lo_compat_privileges + - log_directory + - log_file_mode + - logging_collector + - log_line_prefix + - log_timezone + - log_truncate_on_rotation + - port + - rds.max_tcp_buffers + - recovery_init_sync_method + - restart_after_crash + - ssl + - ssl_ca_file + - ssl_cert_file + - ssl_ciphers + - ssl_key_file + - stats_temp_directory + - superuser_reserved_connections + - unix_socket_directories + - unix_socket_group + - unix_socket_permissions + - update_process_title + - wal_receiver_create_temp_slot + - wal_sync_method diff --git a/deploy/postgresql/templates/configconstraint.yaml b/deploy/postgresql/templates/configconstraint.yaml index d5da741fa..97c1787e4 100644 --- a/deploy/postgresql/templates/configconstraint.yaml +++ b/deploy/postgresql/templates/configconstraint.yaml @@ -8,9 +8,15 @@ metadata: spec: reloadOptions: tplScriptTrigger: + sync: true scriptConfigMapRef: patroni-reload-script namespace: {{ .Release.Namespace }} + # update patroni master + selector: + matchLabels: + "apps.kubeblocks.postgres.patroni/role": "master" + # top level mysql configuration type cfgSchemaTopLevelName: PGParameter