Feat: Using fsspec to download files

[deependujha]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #181

• Basic setup and working for fsspec done and tested for S3 & GS.

• Failing tests will be modified to match fsspec tests after approval.

• Requirements.txt file will also be optimized after approval.

---

A small script to test

import litdata as ld

# modify it to azure or gcp endpoint, and give it a try (tested for s3 & gs)
s3_uri = 's3://my-dummy-bucket-litdata/deep_data/' 

dataset = ld.StreamingDataset(s3_uri, shuffle=True)
dataloader = ld.StreamingDataLoader(dataset)

for sample in dataloader:
    print(sample)
    print("="*80)
print("\nAll done\n")

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[deependujha]

tested successfully on S3 and GS for optimize(mode= none | append | overwrite), checkpoint, merge_datasets, streaming_dataset.

All the # todo: add support for other providers removed.

[codecov]

Codecov Report

Attention: Patch coverage is 55.76923% with 92 lines in your changes missing coverage. Please review.

Project coverage is 78%. Comparing base (2f78ec1) to head (b5ec077).
Report is 4 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #348   +/-   ##
===================================
- Coverage    78%    78%   -0%     
===================================
  Files        34     33    -1     
  Lines      5008   4983   -25     
===================================
- Hits       3928   3890   -38     
- Misses     1080   1093   +13     

[bhimrazy]

Here's a sneak peek of my version of the benchmark tests. I think we might need to run a few more to ensure everything is in good shape.

cc: @Borda @tchaton

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
5685611	Triggered	Generic High Entropy Secret	79a3ad8	tests/streaming/test_resolver.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[deependujha]

For error

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.35.21 requires botocore<1.36.0,>=1.35.21, but you have botocore 1.35.16 which is incompatible.

• Run command

pip install -U boto boto3 botocore aiobotocore

---

Current update

• Docs updated (fsspec storage options) ✅
• Deleted commented files ✅
• Tested on all three storage providers (S3, GCS, Azure blob) ✅

---

[tchaton]

Awesome work !