updated the architecture doc #2786

yaronp68 · 2023-08-23T09:55:23Z

No description provided.

dwdougherty

Lots of small changes.

dwdougherty · 2023-08-23T13:48:34Z

content/rdi/architecture.md

@@ -9,59 +9,89 @@ headerRange: "[2]"
 aliases: 
 ---

-Redis Data Integration is a product for data ingestion from existing systems of record (mainly databases) into Redis Enterprise.
-You can think of Redis Data Integration as a streaming ELT process:
+Redis Data Integration (RDI) is a product that helps Redis Enterprise users ingest and export data in near real time


Suggested change

Redis Data Integration (RDI) is a product that helps Redis Enterprise users ingest and export data in near real time

Redis Data Integration (RDI) is a product that helps Redis Enterprise users ingest and export data in near real time. RDI provides the following:

dwdougherty · 2023-08-23T13:48:50Z

content/rdi/architecture.md

- Data is **E**xtracted from the source database using [Debezium](https://debezium.io/) connectors.
+- End to end solution - No need for additional tools and integrations
+- Capture Data Change (CDC) included
+- Covers most popular databases as sources & targets


Suggested change

- Covers most popular databases as sources & targets

- Covers most popular databases as sources and targets

dwdougherty · 2023-08-23T13:49:35Z

content/rdi/architecture.md

+- Data delivery guaranteed (at least once)
+- Zero downtime day 2 operations (Reconfigure & Upgrade)
+
+Redis Data Integration currently supports 2 use cases:


Suggested change

Redis Data Integration currently supports 2 use cases:

Redis Data Integration currently supports two use cases:

dwdougherty · 2023-08-23T13:50:40Z

content/rdi/architecture.md

+- Covers most popular databases as sources & targets
+- Declarative data mapping and transformations - no custom code needed
+- Data delivery guaranteed (at least once)
+- Zero downtime day 2 operations (Reconfigure & Upgrade)


Suggested change

- Zero downtime day 2 operations (Reconfigure & Upgrade)

- Zero downtime day 2 operations (reconfigure & upgrade)

dwdougherty · 2023-08-23T13:50:59Z

content/rdi/architecture.md

+- Ingest (Cache prefetch).
+- Write Behind.


Suggested change

- Ingest (Cache prefetch).

- Write Behind.

- Ingest (cache prefetch)

- Write behind

dwdougherty · 2023-08-23T14:00:42Z

content/rdi/architecture.md

+
+### Write behind topology
+
+RDI CLI and engine comes in one edition that can run both ingest and write behind. However, the topology for write behind is different.


Suggested change

RDI CLI and engine comes in one edition that can run both ingest and write behind. However, the topology for write behind is different.

RDI CLI and engine come in one edition that can run both ingest and write behind. However, the topology for write behind is different.

dwdougherty · 2023-08-23T14:02:07Z

content/rdi/architecture.md

+### Write behind topology
+
+RDI CLI and engine comes in one edition that can run both ingest and write behind. However, the topology for write behind is different.
+RDI engine is installed on Redis database containing the application data and not on a separate staging Redis database. RDI streams and control plain adds a small footprint of few hundreds of MBs to the Redis database. In the write behind topology, RDI process in parallel on each shard and establishes a single connection from each shard to the downstream database.


Suggested change

RDI engine is installed on Redis database containing the application data and not on a separate staging Redis database. RDI streams and control plain adds a small footprint of few hundreds of MBs to the Redis database. In the write behind topology, RDI process in parallel on each shard and establishes a single connection from each shard to the downstream database.

RDI engine is installed on a Redis database containing the application data and not on a separate staging Redis database. RDI streams and control plane add a small footprint of few hundreds of MBs to the Redis database. In the write behind topology, RDI processed in parallel on each shard and establishes a single connection from each shard to the downstream database.

dwdougherty · 2023-08-23T14:02:30Z

content/rdi/architecture.md

+- Hash
+- Json
+- Set
+- Sorted Set


Suggested change

- Hash

- Json

- Set

- Sorted Set

- Hash

- JSON

- Set

- Sorted set

dwdougherty · 2023-08-23T14:03:08Z

content/rdi/architecture.md

+- Set
+- Sorted Set
+
+Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis key and target database record.


Suggested change

Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis key and target database record.

Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis keys and target database's records.

dwdougherty · 2023-08-23T14:03:26Z

content/rdi/architecture.md

+- Sorted Set
+
+Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis key and target database record.
+The job `keys` and `mapping` sections help making this an easy task.


Suggested change

The job `keys` and `mapping` sections help making this an easy task.

The job `keys` and `mapping` sections help make this an easy task.

dwdougherty · 2023-08-23T15:41:12Z

content/rdi/architecture.md

+RDI captures the changes to a selected set of key-patterns in a Redis keyspace and asynchronously writes them in small batches to the downstream database, so the application doesn't need to take care of remodeling the data or managing the connection with the downstream database.
+
+RDI write behind can normalize a key in Redis to several records in one or more tables at the target.
+To learn more about write behind declarative jobs and renormalization read the [write behind quick start guide]({{<relref="/rdi/quickstart/write-behind-guide.md">}})


One more fix... the relref shortcode syntax was incorrect (no = sign).

Suggested change

To learn more about write behind declarative jobs and renormalization read the [write behind quick start guide]({{<relref="/rdi/quickstart/write-behind-guide.md">}})

To learn more about write behind declarative jobs and renormalization read the [write behind quick start guide]({{< relref "/rdi/quickstart/write-behind-guide.md" >}})

updated the architecture doc

405c288

yaronp68 requested a review from dwdougherty August 23, 2023 09:55

yaronp68 self-assigned this Aug 23, 2023

yaronp68 added the rdi Redis Data Integration label Aug 23, 2023

dwdougherty requested changes Aug 23, 2023

View reviewed changes

fixed PR comments

34fd1d6

yaronp68 requested a review from dwdougherty August 23, 2023 15:12

dwdougherty approved these changes Aug 23, 2023

View reviewed changes

dwdougherty reviewed Aug 23, 2023

View reviewed changes

removed =

c933819

dwdougherty merged commit 57a942b into rdi-preview-build Aug 23, 2023
4 checks passed

dwdougherty deleted the update-rdi-architecture branch August 23, 2023 16:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updated the architecture doc #2786

updated the architecture doc #2786

yaronp68 commented Aug 23, 2023

dwdougherty left a comment

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

dwdougherty Aug 23, 2023

yaronp68 Aug 23, 2023

	Redis Data Integration (RDI) is a product that helps Redis Enterprise users ingest and export data in near real time
	Redis Data Integration (RDI) is a product that helps Redis Enterprise users ingest and export data in near real time. RDI provides the following:

	- Covers most popular databases as sources & targets
	- Covers most popular databases as sources and targets

	Redis Data Integration currently supports 2 use cases:
	Redis Data Integration currently supports two use cases:

	- Zero downtime day 2 operations (Reconfigure & Upgrade)
	- Zero downtime day 2 operations (reconfigure & upgrade)


		### Write behind topology

		RDI CLI and engine comes in one edition that can run both ingest and write behind. However, the topology for write behind is different.

	RDI CLI and engine comes in one edition that can run both ingest and write behind. However, the topology for write behind is different.
	RDI CLI and engine come in one edition that can run both ingest and write behind. However, the topology for write behind is different.

	RDI engine is installed on Redis database containing the application data and not on a separate staging Redis database. RDI streams and control plain adds a small footprint of few hundreds of MBs to the Redis database. In the write behind topology, RDI process in parallel on each shard and establishes a single connection from each shard to the downstream database.
	RDI engine is installed on a Redis database containing the application data and not on a separate staging Redis database. RDI streams and control plane add a small footprint of few hundreds of MBs to the Redis database. In the write behind topology, RDI processed in parallel on each shard and establishes a single connection from each shard to the downstream database.

	Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis key and target database record.
	Unlike the ingest scenario, write behind has no default behavior for model translation. It is up to the user to create a declarative job, specifying the mapping between Redis keys and target database's records.

	The job `keys` and `mapping` sections help making this an easy task.
	The job `keys` and `mapping` sections help make this an easy task.

	To learn more about write behind declarative jobs and renormalization read the [write behind quick start guide]({{<relref="/rdi/quickstart/write-behind-guide.md">}})
	To learn more about write behind declarative jobs and renormalization read the [write behind quick start guide]({{< relref "/rdi/quickstart/write-behind-guide.md" >}})

updated the architecture doc #2786

updated the architecture doc #2786

Conversation

yaronp68 commented Aug 23, 2023

dwdougherty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment