Skip to content

Commit

Permalink
Fixed relative links
Browse files Browse the repository at this point in the history
  • Loading branch information
dat-a-man committed Oct 16, 2024
1 parent 66846ce commit bf375b1
Showing 1 changed file with 4 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -486,7 +486,7 @@ with **chunk_size** 1000. **chat_message** table will load data incrementally us
database = sql_database()
```

You can configure all the arguments this way (except adapter callback function). [Standard dlt rules apply](https://dlthub.com/docs/general-usage/credentials/configuration#configure-dlt-sources-and-resources). You can use environment variables [by translating the names properly](https://dlthub.com/docs/general-usage/credentials/config_providers#toml-vs-environment-variables) ie.
You can configure all the arguments this way (except adapter callback function). [Standard dlt rules apply](../../general-usage/credentials/setup.md). You can use environment variables [by translating the names properly](../../general-usage/credentials/setup.md#environment-variables) ie.
```sh
SOURCES__SQL_DATABASE__CREDENTIALS="mssql+pyodbc://loader.database.windows.net/dlt_data?trusted_connection=yes&driver=ODBC+Driver+17+for+SQL+Server"
SOURCES__SQL_DATABASE__BACKEND=pandas
Expand All @@ -495,7 +495,7 @@ SOURCES__SQL_DATABASE__CHAT_MESSAGE__INCREMENTAL__CURSOR_PATH=updated_at
```

### Configuring incremental loading
`dlt.sources.incremental` class is a [config spec](https://dlthub.com/docs/general-usage/credentials/config_specs) and can be configured like any other spec, here's an example that sets all possible options:
`dlt.sources.incremental` class is a [config spec](../../walkthroughs/add_credentials.md) and can be configured like any other spec, here's an example that sets all possible options:
```toml
[sources.sql_database.chat_message.incremental]
cursor_path="updated_at"
Expand Down Expand Up @@ -611,7 +611,7 @@ resource. Below we show you an example on how to pseudonymize the data before it
```

1. To pseudonymize columns and hide personally identifiable information (PII), refer to the
[documentation](https://dlthub.com/docs/general-usage/customising-pipelines/pseudonymizing_columns).
[documentation](../../general-usage/customising-pipelines/pseudonymizing_columns.md).
As an example, here's how to pseudonymize the "rfam_acc" column in the "family" table:

```py
Expand Down Expand Up @@ -663,6 +663,6 @@ resource. Below we show you an example on how to pseudonymize the data before it
print(info)
```

1. Remember to keep the pipeline name and destination dataset name consistent. The pipeline name is crucial for retrieving the [state](https://dlthub.com/docs/general-usage/state) from the last run, which is essential for incremental loading. Altering these names could initiate a "[dev_mode](https://dlthub.com/docs/general-usage/pipeline#do-experiments-with-dev-mode)", interfering with the metadata tracking necessary for [incremental loads](https://dlthub.com/docs/general-usage/incremental-loading).
1. Remember to keep the pipeline name and destination dataset name consistent. The pipeline name is crucial for retrieving the [state](../../general-usage/state.md) from the last run, which is essential for incremental loading. Altering these names could initiate a "[dev_mode](../../general-usage/pipeline#do-experiments-with-dev-mode)", interfering with the metadata tracking necessary for [incremental loads](../../general-usage/incremental-loading)).

<!--@@@DLT_TUBA sql_database-->

0 comments on commit bf375b1

Please sign in to comment.