dlt pipeline show results in ConfigFieldMissingException when using dynamic config

[trymzet]

dlt version

1.4.0

Describe the problem

I'm using Google Secret Manager to store all credentials except the ones to the GSM itself (those are stored in secrets.toml). AFAIK the only way to integrate secret managers with dlt currently is to use dynamic configuration:

def setup_mssql_credentials():
    dlt.secrets["sources.sql_database.credentials"] = get_secret_from_gcsm(
        "dlt-mssql-credentials"
    )

def setup_gcs_credentials():
    dlt.secrets["destination.filesystem.credentials"] = get_secret_from_gcsm(
        "dlt-gcs-credentials"
    )

(I use these functions at runtime, before the dlt code)

However, likely this is the cause for dlt pipeline show to then show the ConfigFieldMissingException exception when opening the browser, which means the Streamlit app doesn't work at all.

Expected behavior

No response

Steps to reproduce

1. Define and run a pipeline using dynamic config
2. Run dlt pipeline my_pipeline show
3. Open the browser

Operating system

Linux

Runtime environment

Local

Python version

3.11

[rudolfix]

@trymzet we will convert this error into warning to support schema migrations you did

[trymzet]

@rudolfix Thanks. Since creating this issue I've also learned that you can implement a custom config provider - I'm guessing if I did that, things would work out of the box.