Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the serialization of scalar valued tensors #431

Conversation

enrico-stauss
Copy link
Contributor

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #424.

PR review

Anyone in the community is free to review the PR once the tests have passed.

Did you have fun?

Make sure you had fun coding 🙃

@enrico-stauss
Copy link
Contributor Author

By the way, I was concerned for backwards compatibility so I ran a quick check and this fix does not break backwards compatibility. Users with datasets that contain scalar valued tensors and which serialized as pickle can still deserialize correctly.

Copy link

codecov bot commented Dec 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78%. Comparing base (7df78d3) to head (ce5f8d4).
Report is 1 commits behind head on main.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #431   +/-   ##
===================================
  Coverage    78%    78%           
===================================
  Files        35     35           
  Lines      5106   5106           
===================================
  Hits       4006   4006           
  Misses     1100   1100           

Copy link
Collaborator

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it ! Thanks @enrico-stauss

@tchaton tchaton merged commit fa5b13c into Lightning-AI:main Dec 3, 2024
29 checks passed
@tchaton
Copy link
Collaborator

tchaton commented Dec 3, 2024

@enrico-stauss Any feedbacks, suggestions about LitData so far ?

@enrico-stauss
Copy link
Contributor Author

I haven't had very advanced use-cases but from what I can tell LitData works smoothly most of the times and if something comes up, it is a quick fix.

Will keep using it for the resource efficiency alone, so you'll probably see me around :)

@tchaton
Copy link
Collaborator

tchaton commented Dec 4, 2024

Hey @enrico-stauss. That's great. Would you be willing to post on social about LitData ? I am trying to promote it a bit this month.

@enrico-stauss
Copy link
Contributor Author

I (intentionally) don't really have much of a social media presence, so I don't think I can help you here @tchaton.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Scalar torch Tensors not recognized by (NoHeader)TensorSerializer -> Serialized as Pickle
2 participants