Skip to content

Commit

Permalink
Added a note on incremental behaviour
Browse files Browse the repository at this point in the history
  • Loading branch information
dat-a-man committed Mar 6, 2024
1 parent 1957384 commit 0ca4f6b
Showing 1 changed file with 11 additions and 0 deletions.
11 changes: 11 additions & 0 deletions docs/website/docs/general-usage/incremental-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,6 +296,17 @@ We just yield all the events and `dlt` does the filtering (using `id` column dec

Github returns events ordered from newest to oldest so we declare the `rows_order` as **descending** to [stop requesting more pages once the incremental value is out of range](#declare-row-order-to-not-request-unnecessary-data). We stop requesting more data from the API after finding first event with `created_at` earlier than `initial_value`.

:::note
**Note on Incremental Cursor Behavior:**
When using incremental cursors for loading data, it's important to understand how dlt handles records in relation to the cursor's
last value. dlt will load only those records for which the incremental cursor value is higher than the last known value of the cursor.
This means that any records with a cursor value lower than or equal to the last recorded value will be ignored during the loading process.
This behavior ensures efficiency by avoiding the reprocessing of records that have already been loaded, but it can lead to confusion if
there are expectations of loading older records that fall below the current cursor threshold. If your use case requires the inclusion of
such records, consider adjusting your data extraction logic or using a full refresh strategy where appropriate.
:::


### max, min or custom `last_value_func`

`dlt.sources.incremental` allows to choose a function that orders (compares) cursor values to current `last_value`.
Expand Down

0 comments on commit 0ca4f6b

Please sign in to comment.