materializes table schemas for empty tables

[rudolfix]

Description

fixes #1116

Here we allow to create empty tables / files at the destination. The standard dlt behavior is to delay creation of tables until first data arrives. This is beneficial when schema is determined from the data (fully or partially). Specifically yielding empty list will not create empty tables / files. We do not want to change this behavior.

In some cases ie. when full schema is known upfront or generated at runtime, empty tables and files should be created without waiting for data.

This PR introduces a new marker in dlt.mark that, when yielded, will materialize empty tables. Example (dynamic schema)

    @dlt.resource
    def users():
        yield dlt.mark.with_hints(
            # this is a special empty item which will materialize table schema
            dlt.mark.materialize_table_schema(),
            # emit table schema with the item
            dlt.mark.make_hints(
                columns=[
                    {"name": "id", "data_type": "bigint", "precision": 4, "nullable": False},
                    {"name": "name", "data_type": "text", "nullable": False},
                ]
            ),
        )

example (defined schema)

    @dlt.resource(columns=UsersPydanticModel)
    def users():
        yield dlt.mark.materialize_table_schema()

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	88f0029
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/65fb802bf16b7a0008eba98b
😎 Deploy Preview	https://deploy-preview-1122--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[sh-rp]

yes, this is probably a good idea

[sh-rp]

Looks good, I'm assuming there are existing tests in the code that test that no table will be created in the destination for empty resources if this marker is not emitted, right?

[sh-rp]

PS: What about docs?

[rudolfix]

@sh-rp where in the docs we should add this trick? I want to rewrite the Pipeline doc. it is outdated. and there I can mention all the possible markers for items

[sultaniman]

Should skip_write: bool default to True?

[rudolfix]

Looks good, I'm assuming there are existing tests in the code that test that no table will be created in the destination for empty resources if this marker is not emitted, right?

you wrote this tests when we were adding empty files to replace resources