explicitly adding docs for destination item size control #2118
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
There was a problem hiding this comment.
the whole section looks confusing for me, I would make it as a note or info admonition in the Load section.
I would rephrase it as follows:
By enabling file rotation and adjusting the
file_max_itemsandfile_max_bytessettings, you can control the size and number of data chunks sent to the destination database. This can be particularly useful when dealing with large tables with millions of records, as it allows you to manage memory constraints on the database server and enhance performance.
| @@ -50,6 +50,8 @@ Below, we set files to rotate after 100,000 items written or when the filesize e | |||
|
|
|||
| <!--@@@DLT_SNIPPET ./performance_snippets/toml-snippets.toml::file_size_toml--> | |||
|
|
|||
| ### Controlling destination items sizes | |||
| As mentioned above, `dlt` uses the the same files created between **normalize** and **load** for loading to the destination. You can therefore control the destination items' sizes by controlling the maximum single file size or enabling rotation at the **normalize** stage as demonstrated in the snippet above. | |||
There was a problem hiding this comment.
Suggested change
| As mentioned above, `dlt` uses the the same files created between **normalize** and **load** for loading to the destination. You can therefore control the destination items' sizes by controlling the maximum single file size or enabling rotation at the **normalize** stage as demonstrated in the snippet above. | |
| As mentioned above, `dlt` uses the same files created between **normalize** and **load** steps for loading data to the destination. You can therefore control the destination items' sizes by controlling the maximum single file size or enabling rotation at the **normalize** stage as demonstrated in the snippet above. |
AstrakhantsevaAA
approved these changes
Dec 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adding explicit documentation on how to control the destination item sizes. The files created in normalize stage are used for loading as well so controlling normalize file sizes (by setting max file size or enabling rotation) should control destination items sizes too. Currently, the bot does not pick it up as it's not explicit.
Related Issues
Additional Context