Code

SCD2 support#1168

Code

Merged

rudolfix merged 45 commits intodeveldlt-hub/dlt:develfrom 828-scd2dlt-hub/dlt:828-scd2Copy head branch name to clipboardApr 14, 2024

+1,663-314

Commits

Commits on Mar 31, 2024

• format examples

  

  Jorrit Sandbrink

  committed
  720b115

  Copy full SHA for 720b115

• add core functionality for scd2 merge strategy

  

  Jorrit Sandbrink

  committed
  115b4c9

  Copy full SHA for 115b4c9

Commits on Apr 1, 2024

• make scd2 validity column names configurable

  

  Jorrit Sandbrink

  committed
  37befbc

  Copy full SHA for 37befbc

Commits on Apr 2, 2024

• make alias descriptive

  

  Jorrit Sandbrink

  committed
  7726d98

  Copy full SHA for 7726d98

• add validity column name conflict checking

  

  Jorrit Sandbrink

  committed
  30bb2e0

  Copy full SHA for 30bb2e0

Commits on Apr 7, 2024

• Merge branch 'devel' of https://github.com/dlt-hub/dlt into 828-scd2

  

  Jorrit Sandbrink

  committed
  765d652

  Copy full SHA for 765d652

• extend write disposition with dictionary configuration option

  

  Jorrit Sandbrink

  committed
  8f4d4ce

  Copy full SHA for 8f4d4ce

• add default delete-insert merge strategy

  

  Jorrit Sandbrink

  committed
  396ec59

  Copy full SHA for 396ec59

• update write_disposition type hints

  

  Jorrit Sandbrink

  committed
  c8d84d8

  Copy full SHA for c8d84d8

• extend tested destinations

  

  Jorrit Sandbrink

  committed
  11748a6

  Copy full SHA for 11748a6

Commits on Apr 9, 2024

• 2nd time setup (#1202)

  adrianbr

  authored
  e9c8f61

  Copy full SHA for e9c8f61

• remove obsolete deepcopy

  

  Jorrit Sandbrink

  committed
  1f399bc

  Copy full SHA for 1f399bc

• Merge branch 'devel' of https://github.com/dlt-hub/dlt into 828-scd2

  

  Jorrit Sandbrink

  committed
  c8f4173

  Copy full SHA for c8f4173

• Merge pull request #1200 from dlt-hub/devel

  Show description for c99d612
  rudolfix

  authored
  c99d612

  Copy full SHA for c99d612

• add scd2 docs

  

  Jorrit Sandbrink

  committed
  0fa603b

  Copy full SHA for 0fa603b

• add write_disposition existence condition

  

  Jorrit Sandbrink

  committed
  c110aae

  Copy full SHA for c110aae

• add nullability hints to validity columns

  

  Jorrit Sandbrink

  committed
  55df900

  Copy full SHA for 55df900

Commits on Apr 10, 2024

• cache functions to limit schema lookups

  

  Jorrit Sandbrink

  committed
  6b24378

  Copy full SHA for 6b24378

• add row_hash_column_name config option

  

  Jorrit Sandbrink

  committed
  4124d61

  Copy full SHA for 4124d61

Commits on Apr 11, 2024

• default to default merge strategy

  

  Jorrit Sandbrink

  committed
  4236f20

  Copy full SHA for 4236f20

• replace hardcoded column name with variable to fix test

  

  Jorrit Sandbrink

  committed
  0e7f8c0

  Copy full SHA for 0e7f8c0

• fix doc snippets

  

  Jorrit Sandbrink

  committed
  93e7f45

  Copy full SHA for 93e7f45

Commits on Apr 13, 2024

• compares records without order and with caps timestamps precision in scd2 tests

  rudolfix

  committed
  36da1f2

  Copy full SHA for 36da1f2

• defines create load id, stores package state typed, allows package state to be passed on, uses load_id as created_at if possible

  rudolfix

  committed
  0d6919a

  Copy full SHA for 0d6919a

• creates new package to normalize from extracted package so state is carried on

  rudolfix

  committed
  2195ebf

  Copy full SHA for 2195ebf

• bans direct pendulum import

  rudolfix

  committed
  52f0d7b

  Copy full SHA for 52f0d7b

• uses timestamps with properly reduced precision in scd2

  rudolfix

  committed
  caa9ae7

  Copy full SHA for caa9ae7

• selects newest state by load_id, not created_at. this will not affect execution as long as packages are processed in order

  rudolfix

  committed
  64baf2d

  Copy full SHA for 64baf2d

• adds formating datetime literal to escape

  rudolfix

  committed
  8cb24af

  Copy full SHA for 8cb24af

• renames x-row-hash to x-row-version

  rudolfix

  committed
  6039f1c

  Copy full SHA for 6039f1c

• corrects json and pendulum imports

  rudolfix

  committed
  dee8e08

  Copy full SHA for dee8e08

• uses unique column in scd2 sql generation

  rudolfix

  committed
  e63ffe1

  Copy full SHA for e63ffe1

• renames arrow items literal

  rudolfix

  committed
  12bdf2b

  Copy full SHA for 12bdf2b

• adds limitations to docs

  rudolfix

  committed
  c1614b9

  Copy full SHA for c1614b9

• passes only complete columns to arrow normalize

  rudolfix

  committed
  a3f47fc

  Copy full SHA for a3f47fc

• renames mode to disposition

  rudolfix

  committed
  e1c53b8

  Copy full SHA for e1c53b8

• Merge branch 'master' into 828-scd2

  rudolfix

  committed
  c815792

  Copy full SHA for c815792

• Merge branch 'devel' into 828-scd2

  rudolfix

  committed
  900cf06

  Copy full SHA for 900cf06

• saves parquet with timestamp precision corresponding to the destination and updates schema in the normalizer

  rudolfix

  committed
  6ed8000

  Copy full SHA for 6ed8000

• adds transform that computes hashes of tables

  rudolfix

  committed
  4fde3cc

  Copy full SHA for 4fde3cc

• tests arrow/pandas + scd2

  rudolfix

  committed
  7796b00

  Copy full SHA for 7796b00

• allows scd2 columns to be added to arrow items

  rudolfix

  committed
  feca9cd

  Copy full SHA for feca9cd

• various renames

  rudolfix

  committed
  e5c78fd

  Copy full SHA for e5c78fd

Commits on Apr 14, 2024

• uses generic caps when writing parquet if no destination context

  rudolfix

  committed
  d462a5b

  Copy full SHA for d462a5b

• disables coercing timestamps in parquet arrow writer

  rudolfix

  committed
  6944828

  Copy full SHA for 6944828