Code

Fix/large num chunks error#381

Code

Merged

tchaton merged 15 commits intoLightning-AI:mainLightning-AI/litdata:mainfrom bhimrazy:fix/large-num-chunks-errorbhimrazy/litdata:fix/large-num-chunks-errorCopy head branch name to clipboardOct 9, 2024

+36-2

Commits

Commits on Sep 22, 2024

• adds close method for tokens loader

  bhimrazy

  committed
  6c9f6a2

  Copy full SHA for 6c9f6a2

• add close memap for given index

  bhimrazy

  committed
  483750f

  Copy full SHA for 483750f

• Add test for dataset with large number of chunks

  bhimrazy

  committed
  46461ff

  Copy full SHA for 46461ff

• Add test for generating dataset with large number of chunks

  bhimrazy

  committed
  c9fddd5

  Copy full SHA for c9fddd5

• skip for windows

  bhimrazy

  committed
  6332a2e

  Copy full SHA for 6332a2e

Commits on Sep 23, 2024

• Merge branch 'main' into fix/large-num-chunks-error

  bhimrazy

  authored
  4b619d6

  Copy full SHA for 4b619d6

Commits on Sep 24, 2024

• close mmap before delete

  bhimrazy

  committed
  6c6d6f2

  Copy full SHA for 6c6d6f2

• revert close change: causing segmentation error

  bhimrazy

  committed
  5e8793b

  Copy full SHA for 5e8793b

Commits on Oct 7, 2024

• Merge branch 'main' into fix/large-num-chunks-error

  bhimrazy

  authored
  0d2b855

  Copy full SHA for 0d2b855

Commits on Oct 9, 2024

• moved close function to execute only if the current chunk is fully consumed

  bhimrazy

  committed
  a42993d

  Copy full SHA for a42993d

• updated the condition

  bhimrazy

  committed
  feab694

  Copy full SHA for feab694

• removed unused type:ignore

  bhimrazy

  committed
  bddfcdf

  Copy full SHA for bddfcdf

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 4f3f394
  pre-commit-ci[bot]

  committed
  4f3f394

  Copy full SHA for 4f3f394

• revert ignore for memoryview

  bhimrazy

  committed
  9e8d3eb

  Copy full SHA for 9e8d3eb

• merged changes

  bhimrazy

  committed
  2f5cfe9

  Copy full SHA for 2f5cfe9