Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for partitioning layout based on metadata in destination configuration #1044

Closed
wants to merge 12 commits into from
Closed

Add support for partitioning layout based on metadata in destination configuration #1044

wants to merge 12 commits into from

Conversation

Pipboyguy
Copy link
Collaborator

@Pipboyguy Pipboyguy commented Mar 4, 2024 •
edited
Loading

Description

Extends the functionality of the destination configuration to support partitioning based on metadata, rather than just the data itself.

  • Support for resource parameters in partition config: Parameters supplied to resources should be available to use as placeholders in the partition layout.
  • Support for resource metadata in partition config: Resource metadata such as object path or creation date should also be supported.

Related Issues

@Pipboyguy
Refactor filesystem and path utilities methods #930
7246de5 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge remote-tracking branch 'origin/devel' into 930-extended-filesys…
24788b6 
…tem-layout-placeholders-for-destinations
@Pipboyguy
Change anonymous capture group to named
8f0793f 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' into 930-extended-filesystem-layout-placeholders…
09e8ae8 
…-for-destinations
@Pipboyguy
Minor refactoring: grammar, spelling.
5ef0e94 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Merge branch 'devel' into 930-extended-filesystem-layout-placeholders…
d9fa17c 
…-for-destinations
@Pipboyguy
Merge branch 'devel' of github.com:dlt-hub/dlt into 930-extended-file…
e187896 
…system-layout-placeholders-for-destinations
@Pipboyguy
Merge branch 'devel' into 930-extended-filesystem-layout-placeholders…
48e68b2 
…-for-destinations
@Pipboyguy
Merge and minor refactors
335db81 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Add filesystem custom partitioning testing
e74cd02 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Add integration tests #930
5acedf3 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy
Format
2ecba0b 
Signed-off-by: Marcel Coetzee <[email protected]>
@Pipboyguy Pipboyguy linked an issue Mar 4, 2024 that may be closed by this pull request
Extended filesystem layout placeholders for destinations #930
Closed
@netlify Netlify
Copy link

netlify bot commented Mar 4, 2024 •
edited
Loading

✅ Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit 2ecba0b
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/65e59ba7d402b80008050d63
😎 Deploy Preview https://deploy-preview-1044--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@Pipboyguy Pipboyguy self-assigned this Mar 4, 2024
@Pipboyguy Pipboyguy changed the title Add support for partitioning based on metadata in destination configuration Add support for partitioning layout based on metadata in destination configuration Mar 4, 2024
@rudolfix rudolfix assigned sh-rp and unassigned Pipboyguy Mar 18, 2024
@rudolfix rudolfix closed this Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@sultaniman sultaniman

@sh-rp sh-rp

Labels
None yet
Projects
dlt core library
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Extended filesystem layout placeholders for destinations
4 participants
@Pipboyguy @sultaniman @sh-rp @rudolfix