feat: replica cluster bootstrap via VolumeSnapshot (cloudnative-pg#2647)

The codebase already supported creating a replica cluster from a volume snapshot. This patch adds the documentation and the E2e tests for it. Signed-off-by: Niccolò Fei <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Leonardo Cecchi <[email protected]>
EnterpriseDB · Aug 31, 2023 · c2ca044 · c2ca044
1 parent b5f594a
commit c2ca044
Show file tree

Hide file tree

Showing 15 changed files with 617 additions and 107 deletions.
diff --git a/docs/src/replica_cluster.md b/docs/src/replica_cluster.md
@@ -92,6 +92,8 @@ file and define the following parts accordingly:
   we need to do is to enable the replica mode through option `spec.replica.enabled`
   and set the `externalClusters` name in option `spec.replica.source`
 
+#### Example using pg_basebackup
+
 This **first example** defines a replica cluster using streaming replication in
 both bootstrap and continuous recovery. The replica cluster connects to the
 source cluster using TLS authentication.
@@ -135,6 +137,8 @@ in case the replica cluster is in a separate namespace.
       key: ca.crt
 ```
 
+#### Example using a Backup from an object store
+
 The **second example** defines a replica cluster that bootstraps from an object
 store using the `recovery` section and continuous recovery using both streaming
 replication and the given object store. For streaming replication, the replica
@@ -183,6 +187,21 @@ a backup of the source cluster has been created already.
     clusters, and that all the necessary secrets which hold passwords or
     certificates are properly created in advance.
 
+#### Example using a Volume Snapshot
+
+If you use volume snapshots and your storage class provides
+snapshots cross-cluster availability, you can leverage that to
+bootstrap a replica cluster through a volume snapshot of the
+source cluster.
+
+The **third example** defines a replica cluster that bootstraps
+from a volume snapshot using the `recovery` section. It uses
+streaming replication (via basic authentication) and the object
+store to fetch the WAL files.
+
+You can check the [sample YAML](samples/cluster-example-replica-from-volume-snapshot.yaml)
+for it in the `samples/` subdirectory.
+
 ## Promoting the designated primary in the replica cluster
 
 To promote the **designated primary** to **primary**, all we need to do is to

diff --git a/docs/src/samples.md b/docs/src/samples.md
@@ -45,7 +45,7 @@ PostGIS example
 : [`postgis-example.yaml`](samples/postgis-example.yaml):
    an example of "PostGIS cluster" (see the [PostGIS section](postgis.md) for details.)
 
-Replica cluster via streaming
+Replica cluster via streaming (pg_basebackup)
 :   **Prerequisites**: [`cluster-example.yaml`](samples/cluster-example.yaml)
     applied and Healthy
 :   [`cluster-example-replica-streaming.yaml`](samples/cluster-example-replica-streaming.yaml): a replica cluster following `cluster-example` with streaming replication.
@@ -56,7 +56,7 @@ Simple cluster with backup configured
 :  [`cluster-example-with-backup.yaml`](samples/cluster-example-with-backup.yaml)
    a basic cluster with backups configured.
 
-Replica cluster via backup
+Replica cluster via Backup from an object store
 :   **Prerequisites**:
     [`cluster-storage-class-with-backup.yaml`](samples/cluster-storage-class-with-backup.yaml) applied and Healthy.
     And a backup
@@ -65,6 +65,15 @@ Replica cluster via backup
 : [`cluster-example-replica-from-backup-simple.yaml`](samples/cluster-example-replica-from-backup-simple.yaml):
    a replica cluster following a cluster with backup configured.
 
+Replica cluster via Volume Snapshot
+:   **Prerequisites**:
+    [`cluster-example-with-volume-snapshot.yaml`](samples/cluster-example-with-volume-snapshot.yaml) applied and Healthy.
+    And a volume snapshot
+    [`backup-with-volume-snapshot.yaml`](samples/backup-with-volume-snapshot.yaml)
+    applied and Completed.
+: [`cluster-example-replica-from-volume-snapshot.yaml`](samples/cluster-example-replica-from-volume-snapshot.yaml):
+   a replica cluster following a cluster with volume snapshot configured.
+
 Bootstrap cluster with SQL files
 : [`cluster-example-initdb-sql-refs.yaml`](samples/cluster-example-initdb-sql-refs.yaml):
    a cluster example that will execute a set of queries defined in a Secret and a ConfigMap right after the database is created.

diff --git a/docs/src/samples/backup-with-volume-snapshot.yaml b/docs/src/samples/backup-with-volume-snapshot.yaml
@@ -0,0 +1,8 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Backup
+metadata:
+  name: backup-with-volume-snapshot
+spec:
+  method: volumeSnapshot
+  cluster:
+    name: cluster-example-with-volume-snapshot
diff --git a/docs/src/samples/cluster-example-replica-from-volume-snapshot.yaml b/docs/src/samples/cluster-example-replica-from-volume-snapshot.yaml
@@ -0,0 +1,54 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: cluster-example-replica-from-snapshot
+spec:
+  instances: 1
+
+  storage:
+    storageClass: csi-hostpath-sc
+    size: 1Gi
+  walStorage:
+    storageClass: csi-hostpath-sc
+    size: 1Gi
+
+  bootstrap:
+    recovery:
+      source: cluster-example-with-volume-snapshot
+      volumeSnapshots:
+        storage:
+          name: cluster-example-with-volume-snapshot-2-1692618163
+          kind: VolumeSnapshot
+          apiGroup: snapshot.storage.k8s.io
+        walStorage:
+          name: cluster-example-with-volume-snapshot-2-wal-1692618163
+          kind: VolumeSnapshot
+          apiGroup: snapshot.storage.k8s.io
+
+  replica:
+    enabled: true
+    source: cluster-example-with-volume-snapshot
+
+  externalClusters:
+    - name: cluster-example-with-volume-snapshot
+
+      connectionParameters:
+        host: cluster-example-with-volume-snapshot-rw.default.svc
+        user: postgres
+        dbname: postgres
+      password:
+        name: cluster-example-with-volume-snapshot-superuser
+        key: password
+
+      barmanObjectStore:
+        destinationPath: s3://backups/
+        endpointURL: http://minio:9000
+        s3Credentials:
+          accessKeyId:
+            name: minio
+            key: ACCESS_KEY_ID
+          secretAccessKey:
+            name: minio
+            key: ACCESS_SECRET_KEY
+        wal:
+          maxParallel: 8
diff --git a/docs/src/samples/cluster-example-replica-streaming.yaml b/docs/src/samples/cluster-example-replica-streaming.yaml
@@ -10,7 +10,7 @@ spec:
       source: cluster-example
 
   replica:
-    enabled: false
+    enabled: true
     source: cluster-example
 
   storage:

diff --git a/docs/src/samples/cluster-example-with-volume-snapshot.yaml b/docs/src/samples/cluster-example-with-volume-snapshot.yaml
@@ -0,0 +1,32 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: cluster-example-with-volume-snapshot
+spec:
+  instances: 3
+  primaryUpdateStrategy: unsupervised
+
+  # Persistent storage configuration
+  storage:
+    storageClass: csi-hostpath-sc
+    size: 1Gi
+  walStorage:
+    storageClass: csi-hostpath-sc
+    size: 1Gi
+
+  # Backup properties
+  backup:
+    volumeSnapshot:
+       className: csi-hostpath-snapclass
+    barmanObjectStore:
+      destinationPath: s3://backups/
+      endpointURL: http://minio:9000
+      s3Credentials:
+        accessKeyId:
+          name: minio
+          key: ACCESS_KEY_ID
+        secretAccessKey:
+          name: minio
+          key: ACCESS_SECRET_KEY
+      wal:
+        compression: gzip
diff --git a/tests/e2e/asserts_test.go b/tests/e2e/asserts_test.go
@@ -785,22 +785,18 @@ func getScheduledBackupCompleteBackupsCount(namespace string, scheduledBackupNam
 	return completed, nil
 }
 
+// AssertReplicaModeCluster checks that, after inserting some data in a source cluster,
+// a replica cluster can be bootstrapped using pg_basebackup and is properly replicating
+// from the source cluster
 func AssertReplicaModeCluster(
 	namespace,
 	srcClusterName,
-	srcClusterSample,
-	replicaClusterName,
 	replicaClusterSample,
 	checkQuery string,
 	pod *corev1.Pod,
 ) {
 	var primaryReplicaCluster *corev1.Pod
-	var err error
 	commandTimeout := time.Second * 10
-	By("creating source cluster", func() {
-		// Create replica source cluster
-		AssertCreateCluster(namespace, srcClusterName, srcClusterSample, env)
-	})
 
 	By("creating test data in source cluster", func() {
 		cmd := "CREATE TABLE IF NOT EXISTS test_replica AS VALUES (1),(2);"
@@ -821,6 +817,8 @@ func AssertReplicaModeCluster(
 	})
 
 	By("creating replica cluster", func() {
+		replicaClusterName, err := env.GetResourceNameFromYAML(replicaClusterSample)
+		Expect(err).ToNot(HaveOccurred())
 		AssertCreateCluster(namespace, replicaClusterName, replicaClusterSample, env)
 		// Get primary from replica cluster
 		Eventually(func() error {
@@ -868,6 +866,60 @@ func AssertReplicaModeCluster(
 	})
 }
 
+// AssertDetachReplicaModeCluster verifies that a replica cluster can be detached from the
+// source cluster, and its target primary can be promoted. As such, new write operation
+// on the source cluster shouldn't be received anymore by the detached replica cluster.
+// Also, make sure the boostrap fields database and owner of the replica cluster are
+// properly ignored
+func AssertDetachReplicaModeCluster(
+	namespace,
+	srcClusterName,
+	replicaClusterName,
+	srcDatabaseName,
+	replicaDatabaseName,
+	srcTableName string,
+) {
+	var primaryReplicaCluster *corev1.Pod
+	replicaCommandTimeout := time.Second * 10
+
+	By("disabling the replica mode", func() {
+		Eventually(func(g Gomega) {
+			_, _, err := testsUtils.RunUnchecked(fmt.Sprintf(
+				"kubectl patch cluster %v -n %v  -p '{\"spec\":{\"replica\":{\"enabled\":false}}}'"+
+					" --type='merge'",
+				replicaClusterName, namespace))
+			g.Expect(err).ToNot(HaveOccurred())
+		}, 60, 5).Should(Succeed())
+	})
+
+	By("verifying write operation on the replica cluster primary pod", func() {
+		query := "CREATE TABLE IF NOT EXISTS replica_cluster_primary AS VALUES (1),(2);"
+		// Expect write operation to succeed
+		Eventually(func(g Gomega) {
+			var err error
+
+			// Get primary from replica cluster
+			primaryReplicaCluster, err = env.GetClusterPrimary(namespace, replicaClusterName)
+			g.Expect(err).ToNot(HaveOccurred())
+			_, _, err = env.EventuallyExecCommand(env.Ctx, *primaryReplicaCluster, specs.PostgresContainerName,
+				&replicaCommandTimeout, "psql", "-U", "postgres", "appSrc", "-tAc", query)
+			g.Expect(err).ToNot(HaveOccurred())
+		}, 300, 15).Should(Succeed())
+	})
+
+	By("verifying the replica database doesn't exist in the replica cluster", func() {
+		AssertDatabaseExists(namespace, primaryReplicaCluster.Name, replicaDatabaseName, false)
+	})
+
+	By("writing some new data to the source cluster", func() {
+		insertRecordIntoTableWithDatabaseName(namespace, srcClusterName, srcDatabaseName, srcTableName, 4, psqlClientPod)
+	})
+
+	By("verifying that replica cluster was not modified", func() {
+		AssertDataExpectedCountWithDatabaseName(namespace, primaryReplicaCluster.Name, srcDatabaseName, srcTableName, 3)
+	})
+}
+
 func AssertWritesToReplicaFails(
 	connectingPod *corev1.Pod,
 	service string,

diff --git a/tests/e2e/fixtures/replica_mode_cluster/cluster-replica-from-backup.yaml.template b/tests/e2e/fixtures/replica_mode_cluster/cluster-replica-from-backup.yaml.template
@@ -0,0 +1,47 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: cluster-replica-from-backup
+spec:
+  instances: 1
+
+  # Persistent storage configuration
+  storage:
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+    size: 1Gi
+  walStorage:
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+    size: 1Gi
+
+  replica:
+    enabled: true
+    source: cluster-replica-src
+
+  bootstrap:
+    recovery:
+      source: cluster-replica-src
+
+  externalClusters:
+    - name: cluster-replica-src
+
+      connectionParameters:
+        host: cluster-replica-src-rw
+        user: postgres
+        dbname: postgres
+        port: "5432"
+      password:
+        name: cluster-replica-src-superuser
+        key: password
+
+      barmanObjectStore:
+        destinationPath: s3://cluster-backups/
+        endpointURL: http://minio-service:9000
+        s3Credentials:
+          accessKeyId:
+            name: backup-storage-creds
+            key: ID
+          secretAccessKey:
+            name: backup-storage-creds
+            key: KEY
+        wal:
+          compression: gzip
diff --git a/tests/e2e/fixtures/replica_mode_cluster/cluster-replica-from-snapshot.yaml.template b/tests/e2e/fixtures/replica_mode_cluster/cluster-replica-from-snapshot.yaml.template
@@ -0,0 +1,56 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: cluster-replica-from-snapshot
+spec:
+  instances: 1
+
+  # Persistent storage configuration
+  storage:
+    storageClass: ${E2E_CSI_STORAGE_CLASS}
+    size: 1Gi
+  walStorage:
+    storageClass: ${E2E_CSI_STORAGE_CLASS}
+    size: 1Gi
+
+  replica:
+    enabled: true
+    source: cluster-replica-src
+
+  bootstrap:
+    recovery:
+      source: cluster-replica-src
+      volumeSnapshots:
+        storage:
+          name: ${REPLICA_CLUSTER_SNAPSHOT_NAME_PGDATA}
+          kind: VolumeSnapshot
+          apiGroup: snapshot.storage.k8s.io
+        walStorage:
+          name: ${REPLICA_CLUSTER_SNAPSHOT_NAME_PGWAL}
+          kind: VolumeSnapshot
+          apiGroup: snapshot.storage.k8s.io
+
+  externalClusters:
+    - name: cluster-replica-src
+
+      connectionParameters:
+        host: cluster-replica-src-rw
+        user: postgres
+        dbname: postgres
+        port: "5432"
+      password:
+        name: cluster-replica-src-superuser
+        key: password
+
+      barmanObjectStore:
+        destinationPath: s3://cluster-backups/
+        endpointURL: http://minio-service:9000
+        s3Credentials:
+          accessKeyId:
+            name: backup-storage-creds
+            key: ID
+          secretAccessKey:
+            name: backup-storage-creds
+            key: KEY
+        wal:
+          compression: gzip