Allow to change cache directory path?

[cnoco1at3]

🚀 Feature

Thanks for the great work! Wondering if it's possible to change the cache directory to a user specified one, e.g. having an additional parameter for StreamingDataset.

Motivation

Right now this cache path is hardcoded and cannot be changed unless we hacked into the constants.py file. However, in our case, we have an extremely slow disk that happens to be mounted to the home directory, where is currently used by litdata for caching. This becomes the biggest bottleneck even though our connection to S3 is fast enough.

[github-actions]

Hi! thanks for your contribution!, great first issue!

[cnoco1at3]

Actually found a ad-hoc method to make this work, which is not documented I believe. For those curious, here's a minimalist example:

from litdata.streaming.resolver import Dir
from litdata import StreamingDataset

input_dir = "<path-to-input-dir>"
if input_dir.startswith("s3://"):
    input_dir = Dir(url=input_dir, path="<path-to-cache-dir>")
dataset = StreamingDataset(input_dir)

[csy1204]

@cnoco1at3

you can find a documentation about this in Specify cache directory section. #Readme

[tchaton]

Hey @cnoco1at3, thanks for the kind words. I have updated the README, so it is more clear.