Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No schema evolution with delta table format on filesystem destination with merge write disposition #1739

Closed
jorritsandbrink opened this issue Aug 24, 2024 · 0 comments · Fixed by #1742
Closed

No schema evolution with delta table format on filesystem destination with merge write disposition #1739

jorritsandbrink opened this issue Aug 24, 2024 · 0 comments · Fixed by #1742
Assignees
@jorritsandbrink
Labels
bug Something isn't working support This issue is monitored by Solution Engineer

Comments

@jorritsandbrink
Copy link
Collaborator

jorritsandbrink commented Aug 24, 2024 •
edited
Loading

dlt version

0.5.4a0

Describe the problem

Schema evolution doesn't work for the delta table format on the filesystem destination when using the merge write disposition. New columns (that emerge after initial creation of the table) are disregarded—they don't end up in the table.

The append and replace dispositions don't have this issue.

Expected behavior

No response

Operating system

Windows

Runtime environment

Local

Python version

3.8

dlt data source

No response

dlt destination

Filesystem & buckets

Other deployment details

No response

Additional information

No response
@jorritsandbrink jorritsandbrink added the bug Something isn't working label Aug 24, 2024
@jorritsandbrink jorritsandbrink self-assigned this Aug 24, 2024
@jorritsandbrink jorritsandbrink moved this from Todo to In Progress in dlt core library Aug 24, 2024
@jorritsandbrink jorritsandbrink linked a pull request Aug 25, 2024 that will close this issue
Enable schema evolution for merge write disposition with delta table format #1742
Merged
@jorritsandbrink jorritsandbrink mentioned this issue Aug 25, 2024
Merged
@rudolfix rudolfix added the support This issue is monitored by Solution Engineer label Aug 26, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jorritsandbrink jorritsandbrink

Labels
bug Something isn't working support This issue is monitored by Solution Engineer
Projects
dlt core library
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable schema evolution for merge write disposition with delta table format dlt-hub/dlt
2 participants
@rudolfix @jorritsandbrink