Create protocol for manual lesion segmentation

[jcohenadad]

Some relevant discussions:

• Fix manual MS lesion segmentation from dataset version: 7570a2b796aedff16316645e6a85877486c76752 model_seg_ms_mp2rage#56
• Inconsistencies between raters on lesion segmentation model_seg_ms_mp2rage#13
• Ground truth labeling protocol #20

We should use https://github.com/spinalcordtoolbox/manual-correction

[plbenveniste]

@cspino and I are working on manual correction of MS lesion in the spinal cord predicted by the model (refer to ACTRIMS abstract). So far, the proposed method is the following :

• Download SCT
• Clone manual correction repository
• Download segmentation predictions : from `duke/projects/canproco/2023_11_08_lesion_seg_models_and_results/M12_inference/inference_results_Dataset555_nnUNet222_fold2_2d_BIDS``
• Download the config file (looks like the following):

FILES_LESION:

• sub-cal056_ses-M12_STIR.nii.gz
• sub-cal072_ses-M12_STIR.nii.gz
• sub-edm011_ses-M12_PSIR.nii.gz
• sub-edm013_ses-M12_PSIR.nii.gz
• sub-mon001_ses-M12_PSIR.nii.gz
• sub-mon004_ses-M12_PSIR.nii.gz
• sub-tor006_ses-M12_PSIR.nii.gz
• sub-tor007_ses-M12_PSIR.nii.gz
• sub-van005_ses-M12_PSIR.nii.gz
• sub-van010_ses-M12_PSIR.nii.gz

• Create a virtual environment :

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

• Run the following command for manual correction :

• python manual_correction.py -path-img <INPUT_PATH> -config <CONFIG_FILE> -path-label <LABEL_PATH> -suffix-files-lesion _lesion-manual

However, we identified certains things to do beforehand:

• Copy predictions to CanProCo branch
• Create a json file for each file with generated by "Generated by 2D nnUNet model available at duke/projects/canproco/2023_11_08_lesion_seg_models_and_results/nnunet_region-based_(SC+lesion)_training_on_STIR_and_inverted_PSIR_data/nnUNet_results/Dataset222_RegionBasedLesionSeg/nnUNetTrainer__nnUNetPlans__2d model used in ACTRIMS 2024 abstract"
• Modify manual_correction script to change Fsleyes parameters : such as the intensity parameters depending on the contrast
• Manually correct images

@jcohenadad Could you give me feedback on the json file data for the automatically segmented lesions

[valosekj]

Great summary! 👍🏻

For SCI derivatives (segmented using the SCISeg model and then manually corrected), I use the following jSON sidecar:

{
    "GeneratedBy": [
        {
            "Name": "SCISeg nnUNet3D",
            "Version": "Dataset183_tSCI3SitesALPhase2Seed710",
            "Date": "2024-01-16"
        },
        {
            "Author": "Jan Valosek",
            "Note": "Manually corrected",
            "Date": "2024-01-18 10:42:24"
        }
    ]
}

Or, if a mask is created completely manually, I use:

{
    "GeneratedBy": [
        {
            "Author": "Jan Valosek",
            "Date": "2024-01-18 10:42:24"
        }
    ]
}

Unfortunately, manual-correction.py supports only the second JSON option. I usually write a simple loop to add info about a model to the JSON sidecar. UPDATE - an issue opened: spinalcordtoolbox/manual-correction#75

Also see our intranet convention here.

[valosekj]

Modify manual_correction script to change Fsleyes parameters : such as the intensity parameters depending on the contrast

This is already supported; see the -fsleyes-dr flag here.

[plbenveniste]

In our case, what do you think of ?

{
    "GeneratedBy": [
        {
            "Name": "nnunet_region-based_(SC+lesion)_training_on_STIR_and_inverted_PSIR_data nnUNet2D",
            "Version": "Dataset222_RegionBasedLesionSeg",
            "Date": "2024-01-16"
        },
        {
            "Author": "Pierre-Louis Benveniste",
            "Note": "Manually corrected",
            "Date": "2024-01-18 10:42:24"
        }
    ]
}

[jcohenadad]

"Name": "nnunet_region-based_(SC+lesion)_training_on_STIR_and_inverted_PSIR_data nnUNet2D" --> ideally it should point to the actual model (eg: URL to the model hosted on the assets of a published release on github). Or maybe the version should point to the versioned model... i don't know what's best. But in any case we need a link, otherwise in 5 years another student might give the same name to the model, and then we have a conflict

"Note": "Manually corrected", --> i don't think this field is necessary

[plbenveniste]

Release done ! Format of json files will be :

{
    "GeneratedBy": [
        {
            "Name": "2D nnUNet model stored at https://github.com/ivadomed/canproco/releases/tag/r20240125",
            "Version": "2D region-based nnUNet fold 2",
            "Date": "2024-01-16"
        },
        {
            "Author": "Pierre-Louis Benveniste",
            "Date": "2024-01-18 10:42:24"
        }
    ]
}

[jcohenadad]

Great! I have a comment about naming the model #74

Also, putting the version URL under "Name" defeats the purpose of the field "Version".

How about instead putting the name of the model (ie: the packaged ZIP file, like model_ms_seg_sc-lesion_regionBased.zip) under "Name", and the URL to the release under "Version"?

[valosekj]

"Note": "Manually corrected", --> i don't think this field is necessary

I used this because, in some cases, the automatic seg was okay and needed no corrections. To distinguish this, I used:

    "Note": "Visually verified",

or, if manual correction was required:

    "Note": "Manually corrected",

[jcohenadad]

 "Note": "Visually verified",

I agree, this is a good idea. We should probably add this information in our intranet so as to standardize our terminology @NathanMolinier

BTW: Does your manual_correction script automatically identifies if there was a modification in the segmentation, and then applies either "Visually verified" or "Manually corrected"?

[valosekj]

BTW: Does your manual_correction script automatically identifies if there was a modification in the segmentation, and then applies either "Visually verified" or "Manually corrected"?

Yes! The script tracks the modification times of a file to distinguish if the file was just opened or actually modified.

[plbenveniste]

• The model name was updated in the release.
• Segmentations were added to branch plb/m12_lesion_seg and pushed to git-annex
• Json files were created for each predicted segmentation

The json files have the following format :

{
        "GeneratedBy": [
            {
                "Name": "2D nnUNet model model_ms_seg_sc-lesion_regionBased.zip",
                "Version": "https://github.com/ivadomed/canproco/releases/tag/r20240125",
                "Date": "2024-01-26"
            }
        ]
    }

[NathanMolinier]

"Note": "Visually verified",

I agree, this is a good idea. We should probably add this information in our intranet so as to standardize our terminology @NathanMolinier

BTW: Does your manual_correction script automatically identifies if there was a modification in the segmentation, and then applies either "Visually verified" or "Manually corrected"?

Regarding this point, I discussed with @valosekj and I think that we should stick to our standards and use the field "Name" instead of "Note". For more information about this please check this comment in manual correction.

Manual correction script needs to be updated for sure regarding json sidecars. Also few JSON sidecar examples are presented in our new convention.

[NathanMolinier]

The field "SpatialReference": "orig" should also be added @plbenveniste

[plbenveniste]

Great idea @NathanMolinier ! However, because of consistency, I suggest adding this field to the entire dataset and not just to these specific segmentations. An issue was created #75

[NathanMolinier]

Yes indeed ! All the json sidecars should have this "SpatialReference" 👍

[valosekj]

Regarding the "SpatialReference", you could test manual-correction PR #76, which now adds "SpatialReference" automatically.