Skip to content

Commit

Permalink
profile overrides in docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Nico Gelders committed Oct 28, 2023
1 parent d1dca54 commit f2fc41d
Show file tree
Hide file tree
Showing 2 changed files with 71 additions and 0 deletions.
70 changes: 70 additions & 0 deletions docs/profile_overrides.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Profile overrides

When using dbt, you might want to dynamically change the profile configuration based for example on the environment you are running in. Also some profile configurations might be sensitive and you might not want to store them in your repository. Profile overrides allow you to inject profile configurations via python code, at runtime. In this section we will show you how to use profile overrides.

## Basic usage

To override a profile you must past the `DbtProfile` dataclass to the dbt_flow function.

Profile overrides handle the following cases:

- When a `profile.yml` is detected in the configured `profiles_dir` it will read in the profile and override the values with the values in the `overrides` dict. The values that are not overridden will be kept as is.
- When no `profile.yml` is detected it will create a new `profile.yml` with the values in the `overrides` dict. In this case you need to provide all the values needed for your dbt adapter.

## Example


Contents of `path_to/jaffle_shop/profile.yml` BEFORE override:
```yaml
example_jaffle_shop:
target: dev
outputs:
dev:
type: postgres
host: data-db
port: 5432
dbname: data
```
Configuring the profile overrides:
```python
import os

from prefect import variables
from prefect.blocks.system import Secret

postgres_user = variables.get("POSTGRES_USER")
postgres_pw = Secret.load("POSTGRES_PASSWORD").get()

my_flow = dbt_flow(
project=DbtProject(
...,
profiles_dir="path_to/jaffle_shop",
),
profile=DbtProfile(
target="dev",
overrides={
"user": postgres_user,
"password": postgres_pw,
"schema": os.environ["POSTGRES_SCHEMA"],
"connect_timeout": 30,
},
),
)
```

Contents of `path_to/jaffle_shop/profile.yml` AFTER override:
```yaml
example_jaffle_shop:
target: dev
outputs:
dev:
type: postgres
host: data-db
port: 5432
dbname: data
user: admin
password: super-secret-password
schema: example
connect_timeout: 30
```
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ nav:
- Getting started: getting_started.md
- How it works: how_it_works.md
- DAG integration: dag_integration.md
- Profile overrides: profile_overrides.md
- Changelog: https://github.com/datarootsio/prefect-dbt-flow/releases
- License: license.md
- API:
Expand Down

0 comments on commit f2fc41d

Please sign in to comment.