Errors Downloading Masks from CVAT

[bilalcodehub]

Hi,

I am currently using MONAILabel with CVAT in a Docker environment, utilising the default endoscopy app for 2D segmentation of surgical images. After completing a round of annotation and marking the job as finished, I encountered the following error:

Can't initialize NVML
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
[2024-12-05 19:16:03,247] [1330] [MainThread] [INFO] (main:199) - +++ Adding Trainer:: tooltracking => <lib.trainers.tooltracking.ToolTracking object at 0x793bd0660040>
[2024-12-05 19:16:03,247] [1330] [MainThread] [INFO] (main:217) - +++ Adding Strategy:: tooltracking_epistemic => <monailabel.tasks.activelearning.epistemic.Epistemic object at 0x793bd068d030>
[2024-12-05 19:16:03,247] [1330] [MainThread] [INFO] (main:220) - Active Learning Strategies:: ['random', 'tooltracking_epistemic']
[2024-12-05 19:16:03,357] [1330] [MainThread] [INFO] (monailabel.tasks.infer.bundle:142) - Model: endoscopic_tool_segmentation; Type: segmentation; Label: background => 0
[2024-12-05 19:16:03,357] [1330] [MainThread] [INFO] (monailabel.tasks.infer.bundle:142) - Model: endoscopic_tool_segmentation; Type: segmentation; Label: tools => 1
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (main:234) - +++ Adding Scoring Method:: tooltracking_epistemic => <lib.scoring.cvat.CVATEpistemicScoring object at 0x793bd068cfd0>
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (main:237) - Active Learning Scoring Methods:: ['tooltracking_epistemic']
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (monailabel.tasks.scoring.epistemic_v2:103) - Starting Epistemic Uncertainty scoring
[2024-12-05 19:16:03,364] [1330] [MainThread] [INFO] (monailabel.tasks.scoring.epistemic_v2:118) - EPISTEMIC:: Total unlabeled images: 384; max_samples: 0
[2024-12-05 19:16:03,377] [1330] [MainThread] [INFO] (monailabel.tasks.scoring.epistemic_v2:162) - EPISTEMIC:: {'total': 384, 'skipped': 384, 'executed': 0, 'latency': 0.013}
[2024-12-05 19:16:05,931] [1330] [MainThread] [INFO] (lib.scoring.cvat:63) - Existing Task Status: completed; Scoring/Result: {'total': 384, 'skipped': 384, 'executed': 0, 'latency': 0.013}
[2024-12-05 19:16:05,932] [1330] [MainThread] [INFO] (lib.scoring.cvat:76) - Latest Active Learning Task in CVAT is under progress/not-consumed. Skip to create new!
[2024-12-05 19:16:05,932] [1330] [MainThread] [INFO] (__main__:61) - Result: {"total": 384, "skipped": 384, "executed": 0, "latency": 0.013}
[2024-12-05 19:16:06,620] [1190] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 0
[2024-12-05 19:17:04,063] [1190] [Thread-5] [INFO] (monailabel.datastore.cvat:190) - Preparing to download/update final labels from: 3 => 3 => ActiveLearning_Iteration_1
[2024-12-05 19:17:14,409] [1190] [Thread-5] [ERROR] (monailabel.datastore.cvat:238) - 0 => Failed to download...
[2024-12-05 19:17:24,758] [1190] [Thread-5] [ERROR] (monailabel.datastore.cvat:237) - /tmp/tmpqbjfyeyo.zip is not a zip file
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/monailabel/datastore/cvat.py", line 205, in download_from_cvat
    shutil.unpack_archive(tmp_zip, tmp_folder)
  File "/usr/lib/python3.10/shutil.py", line 1314, in unpack_archive
    func(filename, extract_dir, **kwargs)
  File "/usr/lib/python3.10/shutil.py", line 1201, in _unpack_zipfile
    raise ReadError("%s is not a zip file" % filename)
shutil.ReadError: /tmp/tmpqbjfyeyo.zip is not a zip file
[2024-12-05 19:17:24,760] [1190] [Thread-5] [ERROR] (monailabel.datastore.cvat:238) - 1 => Failed to download...
[2024-12-05 19:17:35,103] [1190] [Thread-5] [ERROR] (monailabel.datastore.cvat:237) - /tmp/tmpqbjfyeyo.zip is not a zip file
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/monailabel/datastore/cvat.py", line 205, in download_from_cvat
    shutil.unpack_archive(tmp_zip, tmp_folder)
  File "/usr/lib/python3.10/shutil.py", line 1314, in unpack_archive
    func(filename, extract_dir, **kwargs)
  File "/usr/lib/python3.10/shutil.py", line 1201, in _unpack_zipfile
    raise ReadError("%s is not a zip file" % filename)
shutil.ReadError: /tmp/tmpqbjfyeyo.zip is not a zip file
[2024-12-05 19:17:35,104] [1190] [Thread-5] [ERROR] (monailabel.datastore.cvat:238) - 2 => Failed to download...

It appears that the MONAILabel server is failing to download the results due to a corrupted or improperly created archive file. Could someone please guide me on how to resolve this issue?

Many thanks and
Kind regards,
Bilal

[SachidanandAlle]

Might be due to cvat taking more time to prepare the zip for download.
https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/datastore/cvat.py#L182

Try to increase the wait time between retries. Default is 10 seconds i guess.
before that try to see how much time it takes directly over cvat to do the similar operation (via API call)
Download URI: https://github.com/Project-MONAI/MONAILabel/blob/main/monailabel/datastore/cvat.py#L192

[SachidanandAlle]

didn't go through new api changes (if any) to handle such downloads. may be they have added better async/poll options. may be try to visit the cvat api and see if something can be taken care while downloading the image+masks via cvat api

[bilalcodehub]

Thank you for taking the time to reply and for your suggestions.

I have increased the time limit to 30 seconds as suggested, but unfortunately, I am still encountering the same error.

I also attempted to request the downloads directly via the terminal using the following command:

curl -X GET -u "username:password" "http://127.0.0.1:8080/api/tasks/3/annotations?action=download&format=Segmentation+mask+1.1" -o output.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   114  100   114    0     0    342      0 --:--:-- --:--:-- --:--:--   343

While the command executes successfully, the output appears to be a text file rather than a valid ZIP archive. Upon inspecting the file, it contains the following error message:

file apps/output.zip
apps/output.zip: ASCII text, with no line terminators

Any idea why this could happen?

[SachidanandAlle]

Also verify if the url is correct for latest cvat. May be in the browser go to your cvat and click download. Use browser debug/network tab check what Ajax request is sent for download. Or check cvat api for downloading image+,mask data

[bilalcodehub]

I am getting the following URLs that CVAT sends during the download:

http://127.0.0.1:8080/api/events?org=
http://127.0.0.1:8080/api/requests/export:task-3-annotations-in-Segmentation_mask_1@1-format-by-1?org=
http://127.0.0.1:8080/api/requests/export:task-3-annotations-in-Segmentation_mask_1@1-format-by-1?org=
http://127.0.0.1:8080/api/tasks/3/dataset/export?org=&filename=output2.zip&format=Segmentation+mask+1.1&save_images=false

It seems like the API endpoints have changed. We need to request the export first before asking to download the dataset.

Any suggestion to change the code to successfully export and kickstart the AL?

[SachidanandAlle]

is it possible for you/team to make the changes needed (test everything to be ok) on cvat.py and raise a PR.
thanks :)

[bilalcodehub]

I’m currently working on it. I’ve just figured out the curl commands to create the job and download the annotations.

[bilalcodehub]

I got it working. I am raising a PR for this now.

[SachidanandAlle]

Refer: #1795