You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On network file systems, we can probably get better IO performance in CompaSOHaloCatalog by reading multiple files in parallel. Hopefully the time-intensive parts of the IO release the GIL, so we could just spin up Python-level threads and use a queue to pass the loaded arrays back to the main thread.
Note that each IO thread will additionally spin off 1 to 4 Blosc threads to do the decompression.
The text was updated successfully, but these errors were encountered:
On network file systems, we can probably get better IO performance in
CompaSOHaloCatalog
by reading multiple files in parallel. Hopefully the time-intensive parts of the IO release the GIL, so we could just spin up Python-level threads and use aqueue
to pass the loaded arrays back to the main thread.Note that each IO thread will additionally spin off 1 to 4 Blosc threads to do the decompression.
The text was updated successfully, but these errors were encountered: