Run monai label through a server, gets stuck at line (monailabel.datastore.local:130) - Auto Reload: True;

[Giulia-dedo]

Hello!!
I am running monai label in a singularity container [https://hub.docker.com/r/projectmonai/monailabel] on a server.
I successfully bind the radiology folder with the one containing my data, then I open the singularity shell and set the GPU to use with export CUDA_VISIBLE_DEVICES=4.

I then run
monailabel start_server -a ./code/app/radiology -s ./code/app/radiology/T1gadoALL/monailabel/ --conf models segmentation

I get this output, and then the code remains stuck, no errors, no other information.

It is not even creating the "datastore_v2.json" file.
What am I doing wrong?

On a local server it was working properly in seconds.

Thank you in advance :)

[diazandr3s]

Hi @Giulia-dedo,

Thanks for sharing these details.
This is strange. Which file format is your data in? Which MONAI Label version are you using? Have you tried using all the GPUs? - without exporting the visible device's variable.

Let us know

[Giulia-dedo]

Hi @diazandr3s,
Thanks for the reply. My images are nii.gz files, 3D. I put them in the correct folder structure as indicated by the guidelines.

As for the version of MONAI Label, I downloaded the latest version of the container, and also tried version 0.8.0, which gives me the same problem. I have not tried to use all the GPUs because I am on a server shared with other people, and it is customary in our lab to just use one at the time (although we usually call it in a different way e.g. CUDA_VISIBLE_DEVICES=2 python ./Folderpath/Scriptname.py.

My pipelines goes like this:

• download sif file: singularity build monai_label.sif docker://projectmonai/monailabel
• convert .sif to a singularity sandbox container singularity build --sandbox ./monai_label monai_label.sif
• bind folders export SINGULARITY_BIND="/nvme/giulia/MONAI_label_MergedFolder:/code/app/radiology"
• enter the singularity shell singularity shell --nv ./monai_label
• check that the binding has worked cd /code/app/radiology/T1gadoALL/monailabel (I see the images, then I go back)
• run the code export CUDA_VISIBLE_DEVICES=4' and 'monailabel start_server -a ./code/app/radiology -s ./code/app/radiology/T1gadoALL/monailabel/ --conf models segmentation

It gets stuck on the line indicated before without creating the .json file that it should.

By checking the server, it shows a 0% use of GPU and 2-3% of CPU (but I do have access to the GPU as normal).

Moreover, once it is launched it is impossible to stop Ctrl+C or exit command do not work and the only way to stop it is to kill the gnu screen and start again from the binding.

Hope this is enough!

[Giulia-dedo]

I found out that the problem was with the binding. By binding to export SINGULARITY_BIND="/nvme/giulia/MONAI_label_MergedFolder:/data/radiology" (with "data" being a previously non existing/used folder), and consequently running monailabel start_server -a ./code/app/radiology -s /data/radiology/T1gadoALL/monailabel/ --conf models segmentation it works. I do not exactly understand why, but as long as it works I'm happy.

Now I have to find a way to create a different link to the server, as the program now outputs http://0.0.0.0:8000 which doesn't work for me, but is a issue unrelated with the topic of the conversation. I consider this issue resolved, thanks for the feedback

[diazandr3s]

Thanks for the update, @Giulia-dedo
You should be able to see other REST APIs by using http://localhost:8000 in the browser