Document incremental definition via config.toml

[sultaniman]

Documentation description

Dlt user from community asked a question on initializing incremental from configuration

Is it a limitation of dlt that it cannot read configs when they are put inside the dlt.sources.incremental(...) block?

# resource for pipeline `talon_one_referrals_pipeline`
@dlt.resource(write_disposition="merge", table_name="referrals", primary_key="id")
def referrals_resource(
    api_secret_key=dlt.secrets.value,
    createdAfter=dlt.sources.incremental(
        "created",
        # BUG: cannot read this initial_value from config file
        initial_value=dlt.config.value,
        last_value_func=last_value_func,
    ),
    campaign_ids=dlt.config.value,
    schema_contract=SCHEMA_CONTRACT
):


[talon_one_referrals_pipeline]
campaign_ids = ["17", "14"]
initial_value = "2020-01-01T00:00:00.000000Z"

In fact we support loading incremental parameters from configuration below is the working example

[incs.sources.pipeline.inc_res.id_after]
cursor_path="idAfter"
initial_value=10

@dlt.resource(table_name="incs")
def inc_res(id_after: dlt.sources.incremental = dlt.config.value):
    for i in range(100):
        yield {"id": i, "idAfter": i, "name": "name-" + str(i)}


pipeline = dlt.pipeline(
    pipeline_name="incs",
    destination="duckdb",
)

pipeline.run(inc_res)

Are you a dlt user?

Yes, I'm already a dlt user.

[sultaniman]

@dat-a-man do you need some kind of detailed explanation to document this?

[dat-a-man]

Thank you, @sultaniman , It looks good and I will manage. Will add it to incremental loading docs

[sultaniman]

Awesome @dat-a-man, thank you as well!

[dat-a-man]

Created PR: 1118