Merge pull request #6098 from EnterpriseDB/release/2024-09-25b

Release: 2024-09-25b

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/apply_constraints.mdx

@@ -0,0 +1,21 @@
+---
+title: "Applying constraints"
+---
+
+At the beginning of your data migration journey with EDB Data Migration Service (EDB DMS), you [prepared and imported the schema](prepare_schema) of your source database. Now, connect to the target database and re-apply the constraints that were excluded from the schema and data migration.
+
+## Primary key and unique constraints
+
+For primary key and unique constraints, you have already created the tables and constraints in the target Postgres database. This allowed EDB DMS to map them to the source objects and migrate data successfuly. You don't need do to anything else. 
+
+The same applies to not null constraints if you included them in your schema import.
+
+## Foreign key, check, and exclusion constraints
+
+You can now re-apply any foreign key, check, or exclusion constraints you excluded during the [schema preparation and import](prepare_schema).
+
+## Ensuring data integrity
+
+Rows in tables that don't have primary key or unique constraints were migrated with at-least-once delivery, therefore, it is possible that these rows are duplicate. 
+
+Deduplication can be performed as part of the [verification](verify_migration).

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/config_reader.mdx

@@ -0,0 +1,173 @@
+---
+title: "Configuring and running the EDB DMS Reader"
+deepToC: true
+redirects: 
+  - /purl/dms/configure_source
+---
+
+## Getting credentials
+
+1.  Access the [EDB Postgres AI® Console](https://portal.biganimal.com) and log in with your EDB Postgres AI Database Cloud Service credentials.
+
+1.  Select the project where you created the database cluster.
+
+1.  Within your project, select **Migrate** > **Credentials**.
+
+1.  Select **Create Migration Credential** > **Download Credential**.
+
+1.  Unzip the credentials folder and copy it to the host where the reader is installed.
+
+## Configuring the reader
+
+1.  Open the EDB DMS reader located in `/opt/cdcreader/run-cdcreader.sh` and ensure you have write permissions. 
+
+1.  Set the variables according to your environment and uncomment the edited lines. See [parameters](#parameters) for further guidance.
+
+```shell
+### set the following environment variables:
+
+##############################################
+# Data Migration Service Cloud Configuration #
+##############################################
+
+# This ID is used to identify the cdcreader.
+#export DBZ_ID=
+
+# Now we only support aws
+#export CLOUD_PROVIDER=
+
+# No need to change about this field
+#export RW_SERVICE_HOST=https://transporter-rw-service.biganimal.com
+
+# You need to create migration credentials in EDB postgresAI platform and set these fields with the path of credential files
+#export TLS_PRIVATE_KEY_PATH=$MY_CREDENTIALS_PATH/client-key.pem
+#export TLS_CERTIFICATE_PATH=$MY_CREDENTIALS_PATH/client-cert.pem
+#export TLS_CA_PATH=$MY_CREDENTIALS_PATH/int.crt
+#export APICURIOREQUEST_CLIENT_KEYSTORE_LOCATION=$MY_CREDENTIALS_PATH/client.keystore.p12
+#export APICURIOREQUEST_TRUSTSTORE_LOCATION=$MY_CREDENTIALS_PATH/int.truststore.p12
+#export KAFKASECURITY_CLIENT_KEYSTORE_LOCATION=$MY_CREDENTIALS_PATH/client.keystore.p12
+#export KAFKASECURITY_TRUSTSTORE_LOCATION=$MY_CREDENTIALS_PATH/int.truststore.p12
+
+################################################## 
+# Data Migration Service Source DB Configuration #
+##################################################
+
+# A sample configuration to create a single postgres database connection:
+#export DBZ_DATABASES_0__TYPE=POSTGRES
+#export DBZ_DATABASES_0__HOSTNAME=localhost
+#export DBZ_DATABASES_0__PORT=5432
+#export DBZ_DATABASES_0__CATALOG=source
+#export DBZ_DATABASES_0__USERNAME=postgres
+#export DBZ_DATABASES_0__PASSWORD=password
+
+# You can increase the index to config more database for the reader
+#export DBZ_DATABASES_1__TYPE=ORACLE
+#export DBZ_DATABASES_1__HOSTNAME=localhost
+#export DBZ_DATABASES_1__PORT=1521
+#export DBZ_DATABASES_1__CATALOG=ORCLCDB/ORCLPDB1
+#export DBZ_DATABASES_1__USERNAME=oracle
+#export DBZ_DATABASES_1__PASSWORD=password
+
+##########################################
+# Optional Parameters Below              #
+##########################################
+
+# Configure logging
+# Generic loglevel
+#export QUARKUS_LOG_LEVEL=DEBUG
+# Loglevel for a single package
+#export QUARKUS_LOG_CATEGORY__COM_ENTERPRISEDB__LEVEL=DEBUG
+```
+
+## Parameters
+
+### `DBZ_ID`
+
+This is the name you assign to identify a source. This name will later appear as a _source_ in the **Migrate** > **Sources** section of the EDB Postgres AI Console. 
+
+Consider the following ID guidelines:
+
+- The maximum character length for the ID is 255 characters.
+- You can use lowercase and uppercase characters, numbers, underscores(_) and hyphens(-) for the ID. Other special characters are not supported.
+- The ID must be unique. The source instances cannot have the same ID.
+
+### `RW_SERVICE_HOST`
+
+Specifies the URL of the service that will host the migration. `transporter-rw-service` is always https://transporter-rw-service.biganimal.com.
+
+### `TLS_PRIVATE_KEY_PATH`
+
+Directory path to the `client-key.pem` private key you downloaded from the EDB Postgres AI Console. 
+
+The HTTP client of the EDB DMS Reader uses it to perform mTLS authentication with the `transporter-rw-service`.
+
+### `TLS_CERTIFICATE_PATH`
+
+Directory path to the X509 `client-cert.pem` certificate you downloaded from the EDB Postgres AI Console.
+
+The HTTP client of the EDB DMS Reader uses it to perform mTLS authentication with the `transporter-rw-service`.
+
+### `TLS_CA_PATH`
+
+Directory path to the `int.cert` Certificate Authority you downloaded from the EDB Postgres AI Console.
+
+It signs the certificate configured in `TLS_CERTIFICATE_PATH`.
+
+### `APICURIOREQUEST_CLIENT_KEYSTORE_LOCATION`
+
+Directory path to the `client-keystore.p12` keystore location file you downloaded from the EDB Postgres AI Console.
+It is created from the private key and certificate configured in `TLS_PRIVATE_KEY_PATH` and `TLS_CERTIFICATE_PATH`.
+
+The Apicurio client uses it to perform mTLS authentication with the `transporter-rw-service`.
+
+### `APICURIOREQUEST_TRUSTSTORE_LOCATION`
+
+Created from the Certificate Authority configured in `TLS_CA_PATH`.
+
+The Apicurio client uses it to perform mTLS authentication with the `transporter-rw-service`.
+
+### `DBZ_DATABASES`
+
+This is a list of source database information you require for the EDB DMS Reader be able to read the correct source database information for the migration. 
+
+You can configure the EDB DMS Reader to migrate multiple databases. The `DBZ_DATABASES_0__TYPE` section delimits the information for the first database. You can use `DBZ_DATABASES_1__TYPE` to provide data for a second database. Add more sections to the EDB DMS Reader (`DBZ_DATABASES_2__TYPE`, `DBZ_DATABASES_3__TYPE`) by increasing the index manully.
+
+#### `DBZ_DATABASES_0__TYPE`
+
+This is the source database type. EDB DMS reader supports `ORACLE` and `POSTGRES`.
+
+#### `DBZ_DATABASES_0__HOSTNAME`
+
+The hostname of the source database.
+
+#### `DBZ_DATABASES_0__PORT`
+
+The port of the source database.
+
+#### `DBZ_DATABASES_0__CATALOG`
+
+The database name in the source database server.
+
+#### `DBZ_DATABASES_0__USERNAME`
+
+The database username of the source database.
+
+#### `DBZ_DATABASES_0__PASSWORD`
+
+The password for the database username of the source database.
+
+
+## Running the EDB DMS Reader
+
+1.  Start the migration: 
+
+    ```shell
+    cd /opt/cdcreader
+    ./run-cdcreader.sh
+    ```
+
+1.  Go to the [EDB Postgres AI Console](https://portal.biganimal.com), and verify that a source with the `DBZ_ID` name is displayed in **Migrate** > **Sources**.
+
+You can select this source for your [migration](create_migration).
+
+

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/create_database.mdx

@@ -0,0 +1,23 @@
+---
+title: "Creating a database cluster"
+---
+
+You can use an existing EDB Postgres® AI cluster or create a new cluster for the target of the database migration. 
+
+To use an existing cluster as a target for the migration, ensure the tables you migrate and the load generated on target don't interfere with existing workloads.
+
+1.  Access the [EDB Postgres AI Console](https://portal.biganimal.com) and log in with your EDB Postgres AI Cloud Service credentials.
+
+1.  Select the project where you want to create the database cluster.
+
+    See [Creating a project](/biganimal/latest/administering_cluster/projects/#creating-a-project) if you want to create one.
+
+1.  Within your project, select **Create New** and **Database Cluster** to create an instance that will serve as target for the EDB Data Migration Service (EDB DMS).
+
+    See [Creating a cluster](/biganimal/release/getting_started/creating_a_cluster/) for detailed instructions on how to create a single-node or a primary/standby high availability cluster.
+
+    See [Creating a distributed high-availability cluster](/biganimal/latest/getting_started/creating_a_cluster/creating_a_dha_cluster/) for detailed instructions on how to create a distributed high availibility cluster.
+
+1.  In the **Clusters** page, select your cluster, and use the **Quick Connect** option to access your instance from your terminal.
+
+1.  Create a new empty database that you will use as a target for the migration. Alternatively, you can use the default database `edb_admin`. 

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/create_migration.mdx

@@ -0,0 +1,30 @@
+---
+title: "Creating a migration"
+---
+
+After you use the EDB DMS Reader to read the source database, create a new migration in the EDB Postgres® AI Console. 
+This establishes a sync between the source database and a target cluster in the EDB Postgres AI Console. 
+
+1.  Access the [EDB Postgres AI Console](https://portal.biganimal.com) and log in with your EDB Postgres AI Database Cloud Service credentials.
+
+1.  Select the project where you created the database cluster. 
+
+1.  Within your project, select **Migrate** > **Migrations**. 
+
+1.  In the **Migrations** page, select **Create New Migration** > **To Managed Postgres**.
+
+1.  In the **Create Migration** page, assign a **Name** to the migration. 
+
+1.  Select the **Source** of the migration. The ID for the EDB DMS Reader is listed in the drop-down menu.
+
+1.  Under **Destination**, select a target cluster for the migration and enter the name of the database where you want the migration to copy data and select **Next**. 
+
+1.  Select the tables and columns to migrate. Modify the table and column names if needed.
+
+1.  Select **Create Migration**.
+
+The EDB Postgres AI Console now displays a new migration. The EDB DMS Reader is constantly streaming data when the migration displays the **Running** state. Changes to data are replicated from the source to the target database as long as the migration is running. 
+
+!!!note 
+   The EDB DMS Reader streams data changes. It does not stream changes in the DDL objects.
+!!!

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/index.mdx

@@ -0,0 +1,35 @@
+---
+title: "Getting started"
+description: Understand how to create a migration from planning to execution.
+indexCards: none
+navigation:
+  - create_database
+  - prepare_schema
+  - installing
+  - preparing_db
+  - config_reader
+  - create_migration
+  - mark_completed
+  - apply_constraints
+  - verify_migration
+---
+
+Creating a migration from an Oracle or Postgres database to EDB Postgres AI involves several steps.
+
+1.  **[Create a target Postgres database cluster](create_database)**: In the EDB Postgres® AI Console, ensure you have created a database cluster. Connect to the cluster and create a database that will serve as a target for the migration.
+
+1.  **[Prepare the source database schema](prepare_schema)**: In your source machine, prepare the source database by exporting it and excluding unsupported constraints. Then, import the adapted schema to the target database.
+
+1.  **[Install the EDB DMS Reader](installing)**: In your source machine, install the EDB DMS Reader from the EDB repository.
+
+1.  **[Prepare your source Oracle or Postgres database](preparing_db)**: In your source machine, prepare the source database by altering settings and creating users that are required for the migration. Ensure your source database can accept SSL connections.
+
+1.  **[Configure the EDB DMS Reader](config_reader)**: In the EDB Postgres AI Console, download dedicated migration credentials. In your source machine, configure the EDB DMS Reader by exporting environment variables that allow the Reader to connect to the source. Execute the Reader.
+
+1.  **[Create a new migration](create_migration)**: In the EDB Postgres AI Console, create a new migration by selecting the source generated by the Reader in the Console, and selecting the target database you created for this purpose.
+
+1.  **[Mark the Migration as completed](mark_completed)**: In the EDB Postgres AI Console, mark the migration as completed to stop the streaming process.
+
+1.  **[Reapply any excluded constraints](apply_constraints)**: Apply the constraints you excluded from the schema migration in the new database.
+
+1.  **[Verify the migration completed successfully](verify_migration)**: Use LiveCompare to ensure the target database has the same data as the source database.

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/index.mdx

@@ -0,0 +1,31 @@
+---
+navTitle: Installing
+title: Installing EDB Data Migration Service Reader on Linux
+
+navigation:
+  - linux_x86_64
+---
+
+Select a link to access the applicable installation instructions:
+
+## Linux [x86-64 (amd64)](linux_x86_64)
+
+### Red Hat Enterprise Linux (RHEL) and derivatives
+
+- [RHEL 9](linux_x86_64/edb-dms-reader_rhel_9), [RHEL 8](linux_x86_64/edb-dms-reader_rhel_8)
+
+- [Oracle Linux (OL) 9](linux_x86_64/edb-dms-reader_rhel_9), [Oracle Linux (OL) 8](linux_x86_64/edb-dms-reader_rhel_8)
+
+- [Rocky Linux 9](linux_x86_64/edb-dms-reader_other_linux_9)
+
+- [AlmaLinux 9](linux_x86_64/edb-dms-reader_other_linux_9)
+
+### SUSE Linux Enterprise (SLES)
+
+- [SLES 15](linux_x86_64/edb-dms-reader_sles_15), [SLES 12](linux_x86_64/edb-dms-reader_sles_12)
+
+### Debian and derivatives
+
+- [Ubuntu 22.04](linux_x86_64/edb-dms-reader_ubuntu_22), [Ubuntu 20.04](linux_x86_64/edb-dms-reader_ubuntu_20)
+
+- [Debian 12](linux_x86_64/edb-dms-reader_debian_12), [Debian 11](linux_x86_64/edb-dms-reader_debian_11)

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/linux_x86_64/edb-dms-reader_debian_11.mdx

@@ -0,0 +1,36 @@
+---
+navTitle: Debian 11
+title: Installing EDB Data Migration Service Reader on Debian 11 x86_64
+---
+
+## Prerequisites
+
+Before you begin the installation process:
+
+- Set up the EDB repository.
+
+  Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step.
+
+  To determine if your repository exists, enter:
+
+  `apt-cache search enterprisedb`
+
+  If no output is generated, the repository isn't installed.
+
+  To set up the EDB repository:
+
+  1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads).
+
+  1. Select the button that provides access to the EDB repository.
+
+  1. Select the platform and software that you want to download.
+
+  1. Follow the instructions for setting up the EDB repository.
+
+## Install the package
+
+Install the EDB DMS Reader (packaged as `cdcreader`):
+
+```shell
+sudo apt-get install cdcreader
+```

advocacy_docs/edb-postgres-ai/migration-etl/data-migration-service/getting_started/installing/linux_x86_64/edb-dms-reader_debian_12.mdx

@@ -0,0 +1,36 @@
+---
+navTitle: Debian 12
+title: Installing EDB Data Migration Service Reader on Debian 12 x86_64
+---
+
+## Prerequisites
+
+Before you begin the installation process:
+
+- Set up the EDB repository.
+
+  Setting up the repository is a one-time task. If you have already set up your repository, you don't need to perform this step.
+
+  To determine if your repository exists, enter:
+
+  `apt-cache search enterprisedb`
+
+  If no output is generated, the repository isn't installed.
+
+  To set up the EDB repository:
+
+  1. Go to [EDB repositories](https://www.enterprisedb.com/repos-downloads).
+
+  1. Select the button that provides access to the EDB repository.
+
+  1. Select the platform and software that you want to download.
+
+  1. Follow the instructions for setting up the EDB repository.
+
+## Install the package
+
+Install the EDB DMS Reader (packaged as `cdcreader`):
+
+```shell
+sudo apt-get install cdcreader
+```