Skip to content

Commit

Permalink
RDSC-1456 Add default job information
Browse files Browse the repository at this point in the history
  • Loading branch information
eizners committed Aug 22, 2023
1 parent cbebdf4 commit 1af31e8
Show file tree
Hide file tree
Showing 2 changed files with 33 additions and 1 deletion.
31 changes: 31 additions & 0 deletions content/rdi/data-transformation/data-transformation-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,37 @@ Each job describes the transformation logic to perform on data from a single sou

![Data transformation pipeline, high level](/images/rdi/data-transformation-pipeline.png)

## Default Job
In situations where there is a need to perform a default data transformation, the default job is used. The transformation associated with this job will be applied to all tables that lack their own explicitly defined jobs. The default job must have a table name of "*", and only one instance of this type of job is permitted.

For example, the default job can be used to implement common things like adding a prefix/postfix to all Redis keys or injecting a new field for each new hash or JSON written to the target without the need to define such transformations individually for each source table.

At this moment the default job is supported for 'ingest' pipelines only.

### Example
This example demonstrates the process of adding an app_code field with a value of 'rdi' using the `add_field`` block to all tables that lack explicitly defined jobs. Additionally, it appends an 'aws' prefix and a 'gcp' postfix to every generated hash key.

default.yaml
```yaml
source:
table: "*"
row_format: full
transform:
- uses: add_field
with:
fields:
- field: after.app_code
expression: "`rdi`"
language: jmespath
output:
- uses: redis.write
with:
data_type: hash
key:
expression: concat(['aws', '#', table, '#', keys(key)[0], '#', values(key)[0], '#gcp'])
language: jmespath
```
## Jobs
Each job will be in a separate YAML file. All of these files will be uploaded to Redis Data Integration using the `deploy` command (see [Deploy configuration](#deploy-configuration)). If you are using the [scaffold]({{<relref "/rdi/quickstart/ingest-guide.md">}}) command, place the job files in the `jobs` folder.
Expand Down
3 changes: 2 additions & 1 deletion content/rdi/quickstart/ingest-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ aliases:

---

This guide will take you through the creation of a write-behind pipeline.
This guide will take you through the creation of a ingest pipeline.


## Prerequisites
Expand Down Expand Up @@ -48,6 +48,7 @@ The following files will be created in the provided directory:
│   └── application.properties
├── jobs
│   └── README.md
│ └── default.yaml
└── config.yaml

```
Expand Down

0 comments on commit 1af31e8

Please sign in to comment.