Code

parquet-compression for lgbm#15

Code

Draft

YYYasin19 wants to merge 42 commits intomainQuantco/slim-trees:mainfrom model-parsingQuantco/slim-trees:model-parsingCopy head branch name to clipboard

+260-100

Commits

Commits on Mar 6, 2023

• add: first try at model parsing

  YYYasin19

  committed
  f10b51e

  Copy full SHA for f10b51e

• try pyarrow

  YYYasin19

  committed
  bd952cd

  Copy full SHA for bd952cd

• start with pyarrow encoding

  YYYasin19

  committed
  0069ca0

  Copy full SHA for 0069ca0

• somewhat optimize i guess

  YYYasin19

  committed
  8e6177e

  Copy full SHA for 8e6177e

• encode & decode pyarrow table to parquet

  YYYasin19

  committed
  cc0cffb

  Copy full SHA for cc0cffb

• cherry-pick result

  YYYasin19

  committed
  2fb8b84

  Copy full SHA for 2fb8b84

• save

  YYYasin19

  committed
  3594a41

  Copy full SHA for 3594a41

• update name

  YYYasin19

  committed
  92178e1

  Copy full SHA for 92178e1

• remove explicit schema selection (temp?)

  YYYasin19

  committed
  439af86

  Copy full SHA for 439af86

• rename compression_utils to compression

  YYYasin19

  committed
  78449aa

  Copy full SHA for 78449aa

• refactor: utils component

  YYYasin19

  committed
  21071de

  Copy full SHA for 21071de

• store leaf_values seperately

  YYYasin19

  committed
  43b9f0c

  Copy full SHA for 43b9f0c

• add: uncompress tree

  YYYasin19

  committed
  91be951

  Copy full SHA for 91be951

• commit

  YYYasin19

  committed
  5b44aba

  Copy full SHA for 5b44aba

• fix: deps

  YYYasin19

  committed
  e3b9217

  Copy full SHA for e3b9217

• all models pls

  YYYasin19

  committed
  37ed31f

  Copy full SHA for 37ed31f

• suppress warnings

  YYYasin19

  committed
  cc8228b

  Copy full SHA for cc8228b

• cleanup

  YYYasin19

  committed
  660450b

  Copy full SHA for 660450b

• update benchmark :(

  YYYasin19

  committed
  7209db2

  Copy full SHA for 7209db2

• add: benchmark file for optimizing performance of handle generation

  YYYasin19

  committed
  54e2889

  Copy full SHA for 54e2889

Commits on Mar 7, 2023

• push comments

  YYYasin19

  committed
  fc09de9

  Copy full SHA for fc09de9

Commits on Mar 17, 2023

• add linear features parsing and adapt tests

  YYYasin19

  committed
  0372263

  Copy full SHA for 0372263

• add linear features parsing and adapt tests

  YYYasin19

  committed
  5dbf70e

  Copy full SHA for 5dbf70e

Commits on Mar 19, 2023

• Merge branch 'main' into model-parsing

  pavelzw

  committed
  002de1d

  Copy full SHA for 002de1d

• Merge branch 'main' into model-parsing

  pavelzw

  committed
  62e20c7

  Copy full SHA for 62e20c7

• Merge branch 'main' into model-parsing

  pavelzw

  committed
  b28176d

  Copy full SHA for b28176d

Commits on Mar 20, 2023

• Merge branch 'main' into model-parsing

  pavelzw

  committed
  2eb48fa

  Copy full SHA for 2eb48fa

Commits on Mar 24, 2023

• it works (tm)

  YYYasin19

  committed
  45c61f2

  Copy full SHA for 45c61f2

• review changes

  YYYasin19

  committed
  e76045e

  Copy full SHA for e76045e

• review changes part 2

  YYYasin19

  committed
  3fd5a45

  Copy full SHA for 3fd5a45

• apparently needed bc. CI is slow af

  YYYasin19

  committed
  c78c807

  Copy full SHA for c78c807

• merge

  YYYasin19

  committed
  c94c29c

  Copy full SHA for c94c29c

• re-rename

  YYYasin19

  committed
  9052d40

  Copy full SHA for 9052d40

• still too slow

  YYYasin19

  committed
  9b9f146

  Copy full SHA for 9b9f146

Commits on Mar 25, 2023

• Fix typo

  pavelzw

  authored
  7cfbd3c

  Copy full SHA for 7cfbd3c

• add pandas dependency

  YYYasin19

  committed
  2aabb6a

  Copy full SHA for 2aabb6a

• add lenght validation

  YYYasin19

  committed
  6149ba4

  Copy full SHA for 6149ba4

• try higher compression level for lz4 (16)

  YYYasin19

  committed
  b8fc383

  Copy full SHA for b8fc383

• fix: int arg

  YYYasin19

  committed
  fcfa7b3

  Copy full SHA for fcfa7b3

Commits on Mar 26, 2023

• switch to zstd, seems better here?

  YYYasin19

  committed
  db30b19

  Copy full SHA for db30b19

• remove linear tree

  YYYasin19

  committed
  a7266c9

  Copy full SHA for a7266c9

Commits on Mar 27, 2023

• Merge branch 'main' into model-parsing

  pavelzw

  committed
  49834ac

  Copy full SHA for 49834ac