Dynamic generation of the LUTs png

[Tom-TBT]

Hello,
here's the change I propose to generate the LUT png dynamically within a view of webgateway.

I opened the PR so we can discuss it from here, but I haven't yet solved the version compatibility problems between OMERO.web and the plugins.

What I have done:

• change views.listLuts_json to return index for all luts on the server (before, only the ones in LUTS_IN_PNG had index > -1)
• Added a views.luts_png that generates a png, with a custom-made reader for different LUT encoding (binary, text, tsv)
• Changed the URL of the loaded png from url('../img/luts_10.png') to url('/webgateway/luts_png/')

Several other points:

• I generate the gradient used for the shading of color bars. How did you make it originally? The original gradient isn't exactly linear and stops at about 2/3rd of the array:
  
  I guess that the gradient stopping at half is for display. Here I have a slight change with numpy.linspace(255, 0, 180, dtype="uint8") but display is very similar
• The generation of the LUT png takes about 2 seconds on my dev server, but after the first time, it's cached with Django. However, this will be slow on servers that don't have the caching setup.
• The caching uses a hash of the LUT names present on the server. If a LUT is added or removed, the hash changes and it generates a new image. I don't think invalidating the older cache matters here, as reverting a change in the LUTs would still reuse the "old cache" (anyway the old cache should eventually timeout)

[will-moore]

@Tom-TBT I created it by importing a single-channel Image into OMERO that is a gradient from black to white, like the "fake" images (or I created it via the API conn.createImageFromNumpySeq() - can't remember which), then I applied each LUT to it in turn and had the rendering engine render it. Then stitched each of those together.
So, it is simpler logic because it lets the rendering engine do the work, and it ensures that you get the same LUT that will be used for rendering actual images.

[Tom-TBT]

then I applied each LUT to it in turn and had the rendering engine render it. Then stitched each of those together.

Sorry, I only meant for the thing called "gradient.png" in LUTS_IN_PNG. Overall I'm okay with the generation of the png (I would nonetheless double check my LUT reader implementation, but I'm getting the raw RGB values out of it).

[will-moore]

Ah - understood. I don't remember about that I'm afraid. But if you have something that looks similar then that's fine.

[will-moore]

The original "gradient.png" (pre-dating LUTs) was probably created manually in photoshop, then copied into the LUTs image later.

[will-moore]

Need to update this docstring and mention version etc.

[Tom-TBT]

docstring updated with mention to v5.28.0

[will-moore]

Everything seems to be working, but I didn't manage to test adding a new LUT. I duplicated a LUT and renamed it to create a new one in the luts directory on my Docker server, but after restarting I'm not seeing the new one in the lists (nothing to do with this PR).

As I understand it, to use the dynamic LUTs list, clients need to use /webgateway/luts_png/ for the image and /webgateway/luts/?version=2 for the JSON listing?

So it should be possible for a client/app e.g. figure, to try reverse("webgateway_luts_png") and if the URL isn't found then stick with the old behaviour, but if it is then use the dynamic png and listing.

[Tom-TBT]

As I understand it, to use the dynamic LUTs list, clients need to use /webgateway/luts_png/ for the image and /webgateway/luts/?version=2 for the JSON listing?

Yes, that way omero-web is retrocompatible with the plugins.

So it should be possible for a client/app e.g. figure, to try reverse("webgateway_luts_png") and if the URL isn't found then stick with the old behaviour, but if it is then use the dynamic png and listing.

Thanks for the tip, I tried something like this, to then return the appropriate URL from a view of the plugin.
With that, the plugins are also compatible with earlier versions of omero-web. Figure and iViewer handle differently the LUT previews, so they will be patched with different implementations too.

[will-moore]

I guess e.g. omero-figure could load the LUTS json with /webgateway/luts/?version=2 and check whether it gets back version:1 or version:2. If it gets version 2 then use /webgateway/luts_png/ for the image etc.

[knabar]

I like this approach of dynamically generating the LUTS preview PNG.

Having the listLuts_json view handle a version argument seems to introduce a lot of conditionals both in that view and on the client side; could the view just return the old JSON structure with the new information added, e.g.

{
  luts: [{
    png_index: -1,
    png_index_new: 25,
    ...
  }],
  png_luts: ...,
  png_luts_new: ...,
}

This way old clients would still work, and new clients would also still use the same old URL and only need to check for the existence of png_luts_new - if it's there, they can use the new PNG URL.

[Tom-TBT]

Could the view just return the old JSON structure with the new information added

I changed the implementation. Now the new index and list are returned next to the old one. Implementing it here doesn't require any checks since changes on the client and server are packaged together (it won't be the case for figure and iviewer).

I have a question about the caching of the png. A new PNG is generated each time a LUT is renamed, added, or removed because there won't be a matching key in the cache.

But what should be the timeout of the cache? The generation of the PNG takes a few seconds. Setting it to None would prevent the cache from expiring and there would be a "slowdown" only the first time after a change in the LUT list. (It seems to me that the cache is invalidated when restarting omero-web). Would you be ok having that cache timeout set to None?

[knabar]

This looks good to me.

Since the cached value depends on the available LUTs, I don't see a problem with caching indefinitely.

[Tom-TBT]

Thank you Andreas, I thus changed the timeout value to None.

[knabar]

@will-moore do you mind taking a final look at this PR before I merge? Thank you!

[jburel]

@will-moore could you please validate the following:

• server with luts (Perceptibly uniform luts openmicroscopy#6398) and this PR: 8 new LUTs in the menu
• server without the LUTS and this PR: same as before but the menu should be dynamically generated
• server with luts and without this PR: menu without the new 8 LUTs

[will-moore]

• 1) server with luts (Perceptibly uniform luts openmicroscopy#6398) and this PR: 8 new LUTs in the menu
• 2) server without the LUTS and this PR: same as before but the menu should be dynamically generated
• 3) server with luts and without this PR: menu without the new 8 LUTs - see Perceptibly uniform luts openmicroscopy#6398 (comment)

1. Looks good with new LUTs displayed in webclient Preview panel (also works in webgateway viewer):

2. LUT generated dynamically, without server PR (no new LUTs). Menu looks fine too: