Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend layout placeholder params for filesystem destinations #1182

Merged
merged 137 commits into from
Apr 16, 2024

Conversation

sultaniman
Copy link
Contributor

@sultaniman sultaniman commented Apr 3, 2024

This PR addresses #930

Requirements:

  • Make sure you can place something after the extension,
  • Should be able to pass custom placeholders dictionary to fs and merge w/ existing placeholders,
    • if placeholder is not known then show a warning and skip,
    • add validation for placeholders if they are defined with user friendly messages,
    • support custom current date formatting placeholders (interpolation options on top of pendulum or ISO standard)
  • Placeholders in dictionary can be values or callbacks,
  • Make configurable via config toml,
  • We would like to have overridable current date to filesystem arguments
  • Timestamp should be the timestamp from load_package
    @sh-rp recommends to test with staging destination
    you’ll have to test it, write a test where you run the destination settings that use staging and add a map function for the filename

Newly introduced placeholders

## Years
YYYY - 2024, 2025
Y - 2024, 2025

## Months
MMMM - January, February, March
MMM - Jan, Feb, Mar
MM - 01, 02, 03
M - 1, 2, 3

## Days of month
DD - 01, 02
D - 1, 2

## Hours 24h format
HH - 00, 01, 02...23
H - 0, 1, 2...23

## Minutes
mm - 00, 01, 02...59
m - 0, 1, 2...59

## Days of week
dddd - Monday, Tuesday, Wednesday
ddd - Mon, Tue, Wed
dd - Mo, Tu, We
d - 0-6

# Year quarter
Q - quarters Q1, Q2, Q3, Q4

timestamp

TODO

  • Tests,
  • Docs update

@sultaniman sultaniman added enhancement New feature or request destination Issue related to new destinations labels Apr 3, 2024
@sultaniman sultaniman self-assigned this Apr 3, 2024
Copy link

netlify bot commented Apr 3, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 362e34d
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/661d52b6117a930008da1bb5

@sultaniman sultaniman force-pushed the feat/filesystem-layout-extension branch from 8d86605 to ad9bba4 Compare April 4, 2024 15:56
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code got a little messy. let's review it together

dlt/common/storages/fsspec_filesystem.py Outdated Show resolved Hide resolved
dlt/destinations/impl/filesystem/factory.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
@sultaniman sultaniman requested a review from rudolfix April 8, 2024 13:09
@sultaniman sultaniman force-pushed the feat/filesystem-layout-extension branch from 44cfe5e to ba83add Compare April 9, 2024 07:46
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good! a few comments and let's deal with the replace

dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
rudolfix
rudolfix previously approved these changes Apr 10, 2024
dlt/destinations/impl/filesystem/configuration.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
dlt/destinations/path_utils.py Outdated Show resolved Hide resolved
tests/destinations/test_path_utils.py Show resolved Hide resolved
tests/load/filesystem/test_filesystem_common.py Outdated Show resolved Hide resolved
@sultaniman sultaniman requested a review from rudolfix April 15, 2024 10:51
@sultaniman sultaniman mentioned this pull request Apr 15, 2024
14 tasks
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@rudolfix rudolfix merged commit 2ce906c into devel Apr 16, 2024
44 of 46 checks passed
@rudolfix rudolfix deleted the feat/filesystem-layout-extension branch April 16, 2024 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
destination Issue related to new destinations enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants