How to serve a multimodal multi-resolution dataset with MONAILabel (PET CT/MR)

[mike-barrow]

We have some PET CT/ PET MR images we wish to serve for annotating using MONAILabel
I see that deepedit supports multimodal data as discussed here and implemented in PR #753 .

However I did not get the details on how to create a 4D nrrd file from two nrrd sequences that have differing resolution. @lassoan mentioned creating an xml header and zipping it up with the two nrrd sequences, but I cant figure it out.
The closest idea I had was zipping a 3d slicer project file (mrml) and the nrrd's up and putting that on the label server, but it doesn't work as the MONAILabel plugin on the slicer side doesn't recognize it.

Our radiologists want the native resolution CT/MR and we don't have a system with enough memory to upsample the PET to use the normal 4d nrrd trick of concatenating sequences of the same size together.

Any thoughts?

[diazandr3s]

Hi @mike-barrow,

This is a good question. It is highly recommended to keep each modality in a separate file. Also, the assumption here is that the modalities are co-registered and of the same size.
Here we presented a possible solution to this: #241 (comment)
You could make MONAI Label to point to a single modality, and then have a transform that loads the other modalities for training. This transform has to be implemented for infer as well.

Hope this helps,