-
Notifications
You must be signed in to change notification settings - Fork 186
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
New "refresh" mode and "dev_mode" (#1063)
* Rename full_refresh -> dev_mode, add deprecation warning * Replace some full_refresh usage in code and docs * Replace full_refresh usage in tests * Init experimental refresh = full with drop command * Refresh modes with dropped_tables file * Init separate local/write drop command * Use load_package_state instead of drop tables file * Use drop schema in init_client (TODO: error) * Separate cli drop command instructions/execute * drop tables in init_client * dropped_tables field on load package state * Fix import * test/fix truncate mode * Save truncated tables in load package state * Remove load package state copying * cleanup * Drop cmd use package state, refactoring * Don't drop tables without data * Validate literals in configspec * Match stored schema by version+version_hash solves detecting when dropped tables need to be recreated * Cleanup * Fix dlt version test * Cleanup * Remove dropped_tables_filename * Fix snippet * Pipeline refresh docs * refresh argument docstring * Restore and update commented drop cmd tests * Cleanup refresh tests and test whether table dropped vs truncated * Test existing schema hash * Revert "Match stored schema by version+version_hash" This reverts commit 689b3ca. * Use replace_schema flag * Change drop_tables replace_schema to delete_schema * Refresh drop only selected sources * Rename refresh modes, update docs * pipeline.run/extract refresh argument * Don't modify schema when refresh='drop_data' * Move refresh tests to load, add filesystem truncate test * Fix duck import * Remove generated doc sections * Default full_refresh=None * Cleanup unused imports * Close caution blocks * Update config field docstring * Add filesystem drop_tables method * Run all refresh tests on local filesystem destination * Fix test drop * Fix iter filesystem schemas * Fix drop_resources * Default config.full_refresh also None * Fix filesystem + test
- Loading branch information
Showing
67 changed files
with
1,263 additions
and
380 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.