Fix/1571 Incremental: Optionally load or ignore records with cursor_path missing or None value#1576

Merged

willi-mueller merged 33 commits intodevelfrom fix/1571-tolerate-record-without-incremental-cursor-path

Aug 30, 2024

+624-35

Commits on Aug 2, 2024

adds failing test
willi-mueller
committed
incremental extract does not crash if one record has cursor_path = None
willi-mueller
committed
implements ability to raise on cursor_path = None or to include the row. Fails for pandas & pyarrow
willi-mueller
committed
renames start_out_of_range and end_out_of_range to is_below_initial_value and is_above_end_value
willi-mueller
committed
test that add_map can be used to transform items before the incremental function is called
willi-mueller
committed
Reverts renaming of start_out_of_range and end_out_of_range to is_below_initial_value and is_above_end_value
willi-mueller
committed
Documents handling NULL values at incremental cursor
willi-mueller
committed
tests that even records with None at cursor path are loaded
willi-mueller
committed
tests with full pipeline run instead of only extraction with table metrics
willi-mueller
committed
Allows exclusion of None values for python Objects, not Pandas or Arrows
willi-mueller
committed
refactors for speed and readability
willi-mueller
committed
shortens implementation
willi-mueller
committed
suggestions from code review
willi-mueller
committed
fixes exclusion of rows with None cursor values
willi-mueller
committed
include includes also rows where cursor_path is missing
willi-mueller
committed
Clarifies that value at cursor_path and cursor_path not in records are treated equally
willi-mueller
committed
makes supplied cursor_path pop out in error messages
willi-mueller
committed
code formatting & docs update
willi-mueller
committed

Commits on Aug 5, 2024

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

drops temp arrow index before concat and any returns
rudolfix
committed

Commits on Aug 30, 2024