Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimise input and output chunk sizes #11

Open
the-mysh opened this issue Feb 9, 2024 · 1 comment
Open

Optimise input and output chunk sizes #11

the-mysh opened this issue Feb 9, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@the-mysh
Copy link
Collaborator

the-mysh commented Feb 9, 2024

When writing a file, data are loaded in chunks and the output bytes are saved in chunks.

The sizes of input and output chunks cna be specified by the user.
Size of an input chunk is currently expressed in the number of rows from a dataset.
Size of an output chunk is expressed in the number of bytes.

Both should be optimized for different shapes of datasets - combinations of different lengths (numbers of rows) and widths (number of columns, number of images, number of columns in images).

@the-mysh
Copy link
Collaborator Author

the-mysh commented Feb 9, 2024

Related to issue 12

@the-mysh the-mysh added the enhancement New feature or request label Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant