Please add s3 path support to optimize (read and write to s3)

[rakro101]

🚀 Feature

add s3 path support to optimize function

Motivation

To directly read and write to s3

Pitch

I want be able to add path from s3 to the optimize function and that the optimize function and direcly output to s3.

Alternatives

None

Additional context

None

[tchaton]

Hey @rakro101,

Would you be interested in making a contribution ?

Best,
T.C

[rakro101]

Hey @tchaton,

Edit: Writing to s3 from local works.

ATM I looking at reading from s3.

For the contribution I have to talk to my boss. I will come back to you.

Bests
rakro101

[rakro101]

Solved with the S3Path Lib and some Modifications etc can read and write from
and to S3. @tchaton should add this to the studio? Maybe not this branch because optimize function can do it.

[tchaton]

Hey @rakro101. Anything you can upstream to this repo ?

[rakro101]

No, optimize is able to read and write to s3, we had to modify the input function, https://lightning.ai/lightning-ai/studios/convert-parquets-to-lightning-streaming?section=data+processing : convert_parquet_to_lightning_data(parquet_file) not the optimize itself. This is why i closed the issue @tchaton. Thank you for the great implementation!