feat: 2 and 3d async chunked loading

[tlambert03]

this draft PR moves much of the logic for indexing into data into a Chunker class that takes care of the asynchronous requests to the data store, and yields chunks back to the viewer, which then need only worry about putting them at the appropriate place/channel in the canvas. It's working pretty well (note that in the demo, I've added an artificial sleep of 20 ms to every individual chunk request, and chunked it overly dramatically)

This demo shows:

• we can load data in chunks, regardless of whether the data store is a chunked storage. here, I break up the XY plane into chunks of (64, 32), to show that square chunks are not necessary. Obviously, the performance benefit here would depend on aligning chunk requests with the chunks of the storage.
• we can specify the order in which to request chunks. here i load based on the euclidean distance from the center of the array... but it could easily be proximity to any coordinate (such as where the viewer is centered), and could also be used for frustum culling
• when we move the slider, previous requests are canceled, and only the last request goes through
• multi-channel works fine
• (at 0:15) we can still request more dimensions that are being displayed, with the slice slider, and each chunk is responsible for reducing itself down to the number of dimensions being viewed.

Untitled.mov

the implementation should extend to 3d relatively straightforwardly. Ultimately, all that the chunker cares about is that you tell it the full extent of the data you want, and how you'd like to chunk it up, and all the viewer cares about is receiving the final chunks, and the offset required to place them on the canvas

lots of cleanup needed, and need to make sure that it doesn't affect performance of the simple cases

[tlambert03]

nooo...

my naive approach hit a snag... need to be directly updating the 3d texture objects

[tlambert03]

asynchronous 3d chunked loading!

Screen.Recording.2024-06-10.at.8.41.55.PM.mov

[tlambert03]

note to self... some array types (such as tensorstore and dask arrays) provide their own async chunk retrieval and futures patterns. There should be a way to defer to those for array types that support it

[alisterburt]

@tlambert03 finally back from a workshop and checking this out properly - it's wonderful, you're my favorite :)

[tlambert03]

:) thanks
after some encouraging proofs of principle, i've gone back to thinking harder about the request/response model. it's all getting a little blurrier in my head. I definitely think there's a nice generalizable slicing abstraction to be had here, that doesn't overcomplicate things, but it's slightly elusive at the moment. would love to chat about it

[alisterburt]

this sounds like fun, let's find a time to chat - will ping