BigDataViewer #52
Comments
|
Hi @tischi Thank you! In fact I am also very much interested in bring in BigDataViewer to ImageJ.JS. In general, since N5/Zarr are designed for scalability and to live with remote dataset, also as I understand, BigDataViewer don't need much resources to run for visualising the data. For the feasibility, I don't see any obvious obstacle that prevent us from not making it work, we can compile the plugin and patch the transport layer, then it should just work. On that end, we have tried to compile ImageJ2, and still need some work on it to reduce the loading time. The best way forward is to try that with ImageJ2 For now, because we only support ImageJ-1 plugins for now, I am not sure whether it's possible to hack a custom BigDataViewer version that works with ImageJ-1. I saw the dependency list of BDV, and it's looks ok. Maybe @tpietzsch can explain a bit here? Otherwise, maybe we can also think about running BDV independent of ImageJ2, say build a web version of BDV, take away the imagej-1, only keep the SciJava framework and necessary components. @ctrueden What do you think about this? We haven't discuss this but potentially we can also produce separate web apps for these "big" plugins. With that said, I have very limited bandwidth to work on this right now, but this could be definitely something we can aim for, for example we are thinking about doing a "ImageJ in the browser" hackathon with @fjug and @ctrueden . |
|
That sounds amazing! In fact, your N5/Zarr comment is spot on. I think it would be very nice to re-use all that code in a browser. Thus, also your idea of a stand-alone www-BDV is very interesting, imho. For example, there is this website: https://blog.openmicroscopy.org/file-formats/community/2020/11/04/zarr-data/ Finding resources (and time) to do the implementation is certainly an issue, however maybe we could use this thread to figure out who would, in principle, think that a www-BDV is a useful aim and then see how to make it happen in terms of resources. I also cc @axtimwalde as I am very curious about his opinion. |
|
BigDataViewer relies on multi-threading on the CPU for rendering in reasonable time. It also relies on reasonable size memory to cache remote data. Both, multithreading and memory are non-existent or severly limited in the browser, so I am expecting a mildly depressing user experience. On the other hand, the browser has access to standardized GL via WebGL and a great reslicing viewer using WebGL already exists with Neuroglancer which already supports N5 sources. So, all things equal, I am not seeing what BDV in the webbrowser brings to the table (other than that it is cool to get it to work). May be I am too negative about these things, so sorry for lowering the temperature. I may be very wrong. |
No need to be sorry! It's good to evaluate properly whether it would be worth the effort! Personally, I have no clue about the limitations of things running in the browser, so your opinion is very valuable, to me at least! |
|
@axtimwalde Good to have your thoughts here! This is definitely the time to evaluate this from both sides.
Yes, in general I agree with you that the browser is rather limited in terms of compute resources, the 4 GB memory per tab limit for example is rather limiting. And indeed this would mean the loss of user experience.
That is true if we only aim for yet another viewer for zarr and N5. However, we can perhaps think about taking advantage of the plugins in the Fiji world. This is to say, the web BDV is not just a viewer, it can allow further customisation by combining BDV with other SciJava/ImageJ plugins. As mentioned, right now, the work we did in compiling ImageJ-2 suffers the issue of long loading time due its package size, however, if we only ship necessary components for BDV + additional plugin and serve as independent web apps. This might be still interesting for many applications, especially for plugins already or will be built on top of BDV. Another potential direction to pursue (but not sure if possible) is taking out the rendering part of BDV, and delegate the rendering part to Neuroglancer or potentially ITK/VTK viewer, then we can use the API layer of BDV to bridge with other SciJava/ImageJ plugin. In all cases, we won't do any heavy computation in the browser, but rather simple ones, e.g. custom the UI, facilitate the exploration, etc. Finally, if we really want the performance, we can also think about running BDV purely on the server side, and bridge it with the viewers + ImageJ.JS with plugins in the browser. I was discussing this with @thewtex (and he recently supported zarr in the ITK/VTK viewer). I am just throwing some thoughts here, wondering what others think about these. |
|
Refactoring is still underway in ITK/VTK viewer, but the idea is to eventually support server side rendering with BDV et. al.. |
@tischi No worries, "on the server side" means we can run ImageJ/Fiji on a server (typically close to the data), then similar to ImageJ server, we can make a plugin to communicate with ImageJ.JS running in the browser via HTTP or websocket. Like that, the data will be rendered by BDV in the server, and the rendered view will be sent to the ITK/VTK viewer in the ImageJ.JS in the browser. When user do pan or zoom in ITK/VTK, the request will be sent to BDV to render. Is this clear? |
Cristal clear! Thanks a lot! I guess scaling the server might be a bit of an issue, but I also guess living in the days of kybernetes that could probably be done automagically?! |
|
Hi again, is there any significant development that I missed that makes server side rendering a promising concept? I only remember past efforts that were depressingly non-interactive (e.g. http://www.emouseatlas.org/eAtlasViewer_ema/application/ema/wlz/EMA80.php published in https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-13-122). Putting aside the scalability issues with serving expensive customized output to many independent users (instead of unprocessed cacheble compressed data), there was always the limiting issue of how many interactive frames per second you can actually get across the wire. Tiled JPEGs certainly don't do it. The only acceptable solution that I know of is to live-encode video streams. With that, I find this to be pretty close to what NX, AnyDesk, or TeamViewer are doing to achieve reasonable remote desktop experience. So, if this works for remote desktop, why would I not use a remote desktop application to bind to my fat server and run stuff there? What is the benefit of a dedicated BDV rendering server (close to the data) over a dedicated all purpose server (close to the data)? |
|
@axtimwalde yes, scalability would still be an issue. Effective frame encoding is certainly part of the story of a workable solution. A remote desktop is a good solution if that is what is wanted and available. With ImageJ.js / ImJoy, it is also possible to integrate into custom web applications.
The cloud could still be more magical 🧙 🐰, but it is getting easier with services like Coiled. Is BDV scriptable from Python? |
|
If you start the JVM from Python, yes. https://www.youtube.com/watch?v=Ddo5z5qGMb8 |
|
@axtimwalde Regarding the question about why we need a dedicated viewer rather than using a generic screen sharing tool or X11 forwarding, I think that is mainly because screen sharing are often laggy for simple tasks such as using markup tools to draw polygons. The issues is that every UI interaction will require a round-trip to the server. If we instead make a dedicated viewer to interface with the remote viewer, we can be smart about what to be done directly in the browser and what to be scheduled remotely. I.e.: most interactions such as drawing ROI can be handled directly by the browser, the remote BDV instance can serve the current volume as a virtual multi-resolution pyramid in zarr format to the browser (such that we can use vizarr or ITK/VTK viewer to visualize it). Since all the view change will be handled directly in the browser by requesting chunks (combined with local caching) just like vizarr did, the user experience should be rather good. When we need to perform transformations, alignment, non-orthogonal reslicing to the whole volume, we let BDV do it and the result will be served as virtual zarr arrays. I did a rough comparison with napari + VNC screenshare and the Kaibu(ITK/VTK viewer) on the same binder machine: https://www.youtube.com/watch?v=0La0iCaDlME And you can see the experience is rather bad when using markup tools because of the latency. In addition, with a web implementation of viewers, we can easily adapt to different screen size while on mobile it's much harder to use screenshare tools. What do you think about this? |
|
@tischi We added basic NGFF/zarr support with #58 It's implemented as a virtual stack in ImageJ. We can load remote chunks in a lazy fashion. However, the current limitation is on the XY plane, i.e., if we have large tissue image for example, we will have to load the entire image as a whole due to the design of ImageJ virtual stack. I am wondering whether you have any suggestion on a minimal implementation with pure ImageJ-1 to allow browsing multi-resolution pyramid. |
|
@oeway |
@tischi That's what I felt too, so we will need to move to ImageJ2 then, the pending issue for now is the significant size of the ImageJ2/SciJava stack, it makes it super long to load the app, unless we find a way to cut the depedencies. I would like to try another approach as we discussed here, use ImageJ.JS or ITK/VTK viewer as the frontend, and connect to a remote Fiji + bigdataviewer. But there are a few technical details I need to figure out, would you have time in the coming days/weeks so we can discuss over a call? Maybe at the GatherTown? |
|
Yes, let's organize something. Could you send me a mail not to clutter this issue? |
Amazing stuff @oeway!!
I was wondering about the limitations of this approach...
Concretely, is it conceivable to have BigDataViewer running in the web?
https://github.com/bigdataviewer/bigdataviewer-core
cc @joshmoore
The text was updated successfully, but these errors were encountered: