Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Train a baseline REGION-BASED model from scratch on dcm-zurich-lesions datasets using nnUNet #1

Open
5 tasks done
valosekj opened this issue Feb 1, 2024 · 8 comments
Assignees

Comments

@valosekj
Copy link
Member

valosekj commented Feb 1, 2024

This issue summarizes experiments related to Dataset601_DCMlesions (region-based model trained to using a single input channel (T2w_ax) to segment both SC and lesions).

dataset.json:

    "channel_names": {
        "0": "acq-ax_T2w"
    },
    "labels": {
        "background": 0,
        "sc": [
            1,
            2
        ],
        "lesion": 2
    },
    "regions_class_order": [
        1,
        2
    ],
nnUNetPlans.json:
{
    "dataset_name": "Dataset601_DCMlesions",
    "plans_name": "nnUNetPlans",
    "original_median_spacing_after_transp": [
        3.5999999046325684,
        0.5,
        0.5
    ],
    "original_median_shape_after_transp": [
        15,
        320,
        319
    ],
    "image_reader_writer": "SimpleITKIO",
    "transpose_forward": [
        0,
        1,
        2
    ],
    "transpose_backward": [
        0,
        1,
        2
    ],
    "configurations": {
        "2d": {
            "data_identifier": "nnUNetPlans_2d",
            "preprocessor_name": "DefaultPreprocessor",
            "batch_size": 31,
            "patch_size": [
                320,
                320
            ],
            "median_image_size_in_voxels": [
                320.0,
                319.0
            ],
            "spacing": [
                0.5,
                0.5
            ],
            "normalization_schemes": [
                "ZScoreNormalization"
            ],
            "use_mask_for_norm": [
                false
            ],
            "UNet_class_name": "PlainConvUNet",
            "UNet_base_num_features": 32,
            "n_conv_per_stage_encoder": [
                2,
                2,
                2,
                2,
                2,
                2,
                2
            ],
            "n_conv_per_stage_decoder": [
                2,
                2,
                2,
                2,
                2,
                2
            ],
            "num_pool_per_axis": [
                6,
                6
            ],
            "pool_op_kernel_sizes": [
                [
                    1,
                    1
                ],
                [
                    2,
                    2
                ],
                [
                    2,
                    2
                ],
                [
                    2,
                    2
                ],
                [
                    2,
                    2
                ],
                [
                    2,
                    2
                ],
                [
                    2,
                    2
                ]
            ],
            "conv_kernel_sizes": [
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ],
                [
                    3,
                    3
                ]
            ],
            "unet_max_num_features": 512,
            "resampling_fn_data": "resample_data_or_seg_to_shape",
            "resampling_fn_seg": "resample_data_or_seg_to_shape",
            "resampling_fn_data_kwargs": {
                "is_seg": false,
                "order": 3,
                "order_z": 0,
                "force_separate_z": null
            },
            "resampling_fn_seg_kwargs": {
                "is_seg": true,
                "order": 1,
                "order_z": 0,
                "force_separate_z": null
            },
            "resampling_fn_probabilities": "resample_data_or_seg_to_shape",
            "resampling_fn_probabilities_kwargs": {
                "is_seg": false,
                "order": 1,
                "order_z": 0,
                "force_separate_z": null
            },
            "batch_dice": true
        },
        "3d_fullres": {
            "data_identifier": "nnUNetPlans_3d_fullres",
            "preprocessor_name": "DefaultPreprocessor",
            "batch_size": 2,
            "patch_size": [
                16,
                320,
                320
            ],
            "median_image_size_in_voxels": [
                15.0,
                320.0,
                319.0
            ],
            "spacing": [
                3.5999979972839355,
                0.5,
                0.5
            ],
            "normalization_schemes": [
                "ZScoreNormalization"
            ],
            "use_mask_for_norm": [
                false
            ],
            "UNet_class_name": "PlainConvUNet",
            "UNet_base_num_features": 32,
            "n_conv_per_stage_encoder": [
                2,
                2,
                2,
                2,
                2,
                2,
                2
            ],
            "n_conv_per_stage_decoder": [
                2,
                2,
                2,
                2,
                2,
                2
            ],
            "num_pool_per_axis": [
                2,
                6,
                6
            ],
            "pool_op_kernel_sizes": [
                [
                    1,
                    1,
                    1
                ],
                [
                    1,
                    2,
                    2
                ],
                [
                    1,
                    2,
                    2
                ],
                [
                    2,
                    2,
                    2
                ],
                [
                    2,
                    2,
                    2
                ],
                [
                    1,
                    2,
                    2
                ],
                [
                    1,
                    2,
                    2
                ]
            ],
            "conv_kernel_sizes": [
                [
                    1,
                    3,
                    3
                ],
                [
                    1,
                    3,
                    3
                ],
                [
                    3,
                    3,
                    3
                ],
                [
                    3,
                    3,
                    3
                ],
                [
                    3,
                    3,
                    3
                ],
                [
                    3,
                    3,
                    3
                ],
                [
                    3,
                    3,
                    3
                ]
            ],
            "unet_max_num_features": 320,
            "resampling_fn_data": "resample_data_or_seg_to_shape",
            "resampling_fn_seg": "resample_data_or_seg_to_shape",
            "resampling_fn_data_kwargs": {
                "is_seg": false,
                "order": 3,
                "order_z": 0,
                "force_separate_z": null
            },
            "resampling_fn_seg_kwargs": {
                "is_seg": true,
                "order": 1,
                "order_z": 0,
                "force_separate_z": null
            },
            "resampling_fn_probabilities": "resample_data_or_seg_to_shape",
            "resampling_fn_probabilities_kwargs": {
                "is_seg": false,
                "order": 1,
                "order_z": 0,
                "force_separate_z": null
            },
            "batch_dice": false
        }
    },
    "experiment_planner_used": "ExperimentPlanner",
    "label_manager": "LabelManager",
    "foreground_intensity_properties_per_channel": {
        "0": {
            "max": 2102.0,
            "mean": 312.2383728027344,
            "median": 304.0,
            "min": 0.0,
            "percentile_00_5": 100.0,
            "percentile_99_5": 654.0,
            "std": 92.65310668945312
        }
    }
}

Dataset601_DCMlesions will be trained on dcm-zurich-lesions and dcm-zurich-lesions-20231115 datasets using nnUNetv2 region-based approach (i.e., segmenting both SC and lesions).

Manual lesion GTs are available for both datasets.

TODO

  • create SC seg GT for dcm-zurich-lesions-20231115 - run inference using the SCIseg model using segment_sc.sh - done; SCIseg worked relatively well -- only 5 of 38 SC segs needed slight manual corrections. Labels are now pushed
  • double-check if there is an overlap in subjects between both datasets (should not be the case) - no overlap, see comment below
  • convert both datasets from BIDS to nnUNet using dataset_conversion/convert_bids_to_nnUNetv2_region-based.py
  • make sure lesions are part of SC (sct_maths -add) - not needed; see here
  • run training
@valosekj valosekj self-assigned this Feb 1, 2024
@naga-karthik
Copy link
Member

Thanks for opening the issue, Jan!

create SC seg GT for dcm-zurich-lesions-20231115

Regarding this, I pushed a script for generating QC for both SC and lesions for dcm-zurich-* datasets. Once the GT SC seg is generated, please use/update that script for QC visualization!

double-check if there is overlap in subjects between both datasets (should not be the case)

As for this, Patrick confirmed in an email on Nov 15th 2023 that the original dcm-zurich dataset was used in the neurology paper by Scheuren et al. and does not overlap with the latest cohort dcm-zurich-lesions-20231115.

run training

and lastly, for this, I noticed that the dcm-zurich dataset also contains the compression label for all subjects. How are these labels obtained? I was thinking of also including this in the model's prediction so it outputs lesion seg, compression label (and possibly SC seg). It seems that the identification of compression site is important for DCM subjects ...

@naga-karthik naga-karthik changed the title Model 1 - region-based (trained on both SC and lesions) Train a baseline model from scratch on dcm-zurich datasets using nnUNet Feb 2, 2024
@naga-karthik naga-karthik changed the title Train a baseline model from scratch on dcm-zurich datasets using nnUNet Train a baseline model from scratch on dcm-zurich-lesions datasets using nnUNet Feb 2, 2024
@valosekj
Copy link
Member Author

valosekj commented Feb 2, 2024

Regarding this, I pushed a script for generating QC for both SC and lesions for dcm-zurich-* datasets. Once the GT SC seg is generated, please use/update that script for QC visualization!

SCIseg worked well -- only 5 subjects needed corrections; labels are pushed.
Great, thanks for the script! The script worked perfectly! Maybe we could create a folder under duke/projects and copy QC there?

As for this, Patrick confirmed in an email on Nov 15th 2023 that the original dcm-zurich dataset was used in the neurology paper by Scheuren et al. and does not overlap with the latest cohort dcm-zurich-lesions-20231115.

Great, thanks for confirmation!

and lastly, for this, I noticed that the dcm-zurich dataset also contains the compression label for all subjects. How are these labels obtained?

Compression labels are single pixels with a value 1 at the level of compression (details). They were created manually by me or Sandrine.

I was thinking of also including this in the model's prediction so it outputs lesion seg, compression label (and possibly SC seg). It seems that the identification of compression site is important for DCM subjects ...

Yes, this would be super relevant! We have discussed this idea many times with Julien and Sandrine; context here. Note that we have compression labels also for >100 subjects in dcm-zurich. But, dcm-zurich does not contain lesion GTs.

@naga-karthik
Copy link
Member

Maybe we could create a folder under duke/projects and copy QC there?

Sounds good!

Compression labels are single pixels with a value 1 at the level of compression

Right, usually there are more than 1 levels of compression, right? I think for sub-02 I only saw 1 slice labeled despite seeing more compressions, hence my confusion!

Note that we have compression labels also for >100 subjects in dcm-zurich. But, dcm-zurich does not contain lesion GTs.

Nice! at least the model could be pre-trained for detecting compression sites, which, hypothetically could be useful for lesion segmentation (i.e. hyperintensities most likely appear around compressions, if I'm not wrong)

@valosekj
Copy link
Member Author

valosekj commented Feb 2, 2024

Maybe we could create a folder under duke/projects and copy QC there?

Sounds good!

SC seg and lesion QC copied to ~/duke/projects/ml_dcm_seg/dcm-zurich-lesions-20231115_QC_2024-02-02/qc

Right, usually there are more than 1 levels of compression, right? I think for sub-02 I only saw 1 slice labeled despite seeing more compressions, hence my confusion!

Yes, usually, DCM patients have so-called multi-level compression, i.e., compressions at several levels (e.g., disc C4/C5 and disc C5/C6). So far, we label each compression with a single pixel, meaning that for two compressions, we would have two pixels with the value 1.

Nice! at least the model could be pre-trained for detecting compression sites, which, hypothetically could be useful for lesion segmentation (i.e. hyperintensities most likely appear around compressions, if I'm not wrong)

Correct, indeed, I believe that hyperintensities most likely appear around compressions. We can verify this with clinicians during a future meeting.

@valosekj
Copy link
Member Author

valosekj commented Feb 2, 2024

This comment summarizes training progress.

commands

Dataset merging and conversion from BIDS to nnUNet

python ~/code/model-seg-dcm/dataset_conversion/convert_bids_to_nnUNetv2.py 
--path-data
~/data/dcm-zurich-lesions
~/data/dcm-zurich-lesions-20231115
--region-based
--split 0.8 0.2
--seed 42
--dataset-number 601
--dataset-name DCMlesions
--path-out
${nnUNet_raw}/dcm-zurich-lesions_combined_nnunet

Resulting in:

Number of training and validation images (across all sites): 41
Number of test images (across all sites): 11
Number of test images in dcm-zurich-lesions: 3
Number of test images in dcm-zurich-lesions-20231115: 8

Training

conda activate nnunet
cd ~/code/model-seg-dcm
./training/01_run_training_dcm-zurich-lesions.sh

Note: all folds below were trained using the default nnUNetTrainer trainer (i.e., the sum of Cross Entropy Loss and Dice Loss with the smoothing term).

fold 0 - crashed to zero

image

fold 1 - crashed to zero

image

fold 2 - crashed to zero

image

fold 3 - crashed to zero

image

fold 4 - crashed to zero

image

@valosekj valosekj changed the title Train a baseline model from scratch on dcm-zurich-lesions datasets using nnUNet Train a baseline REGION-BASED model from scratch on dcm-zurich-lesions datasets using nnUNet Feb 6, 2024
@valosekj
Copy link
Member Author

valosekj commented Feb 6, 2024

Since all folds trained with the default nnUNetTrainer trainer collapsed to zero (see comment above), I am now trying nnUNetTrainerDiceCELoss_noSmooth (i.e., without the smoothing term of the Dice loss) (relevant issues here and here).

@valosekj
Copy link
Member Author

valosekj commented Feb 8, 2024

nnUNetTrainerDiceCELoss_noSmooth seems to be working (at least, the model is not collapsing to zero)!

fold 0, seed42

image

fold 1, seed42

image

@valosekj
Copy link
Member Author

valosekj commented Mar 6, 2024

nnUNetTrainerDiceCELoss_noSmooth__nnUNetPlans__3d_fullres/fold_1

`dcm-zurich-lesions`
Test Phase Metrics [ANIMA] for sc:
	Jaccard --> Mean: 0.941, Std: 0.005
	Dice --> Mean: 0.970, Std: 0.003
	Sensitivity --> Mean: 0.973, Std: 0.003
	Specificity --> Mean: 1.000, Std: 0.000
	PPV --> Mean: 0.966, Std: 0.004
	NPV --> Mean: 1.000, Std: 0.000
	RelativeVolumeError --> Mean: 0.682, Std: 0.511
	HausdorffDistance --> Mean: 1.138, Std: 0.195
	ContourMeanDistance --> Mean: 0.187, Std: 0.029
	SurfaceDistance --> Mean: 0.001, Std: 0.001
Test Phase Metrics [ANIMA] for lesion:
	Jaccard --> Mean: 0.321, Std: 0.070
	Dice --> Mean: 0.482, Std: 0.079
	Sensitivity --> Mean: 0.341, Std: 0.098
	Specificity --> Mean: 1.000, Std: 0.000
	PPV --> Mean: 0.923, Std: 0.109
	NPV --> Mean: 1.000, Std: 0.000
	RelativeVolumeError --> Mean: -61.155, Std: 16.415
	HausdorffDistance --> Mean: 4.061, Std: 2.801
	ContourMeanDistance --> Mean: 1.032, Std: 0.396
	SurfaceDistance --> Mean: 0.061, Std: 0.002
	PPVL --> Mean: 0.667, Std: 0.471
	SensL --> Mean: 0.611, Std: 0.283
	F1_score --> Mean: 0.556, Std: 0.416
`dcm-zurich-lesions-20231115`
Test Phase Metrics [ANIMA] for sc:
	Jaccard --> Mean: 0.923, Std: 0.029
	Dice --> Mean: 0.960, Std: 0.016
	Sensitivity --> Mean: 0.959, Std: 0.023
	Specificity --> Mean: 1.000, Std: 0.000
	PPV --> Mean: 0.961, Std: 0.025
	NPV --> Mean: 1.000, Std: 0.000
	RelativeVolumeError --> Mean: -0.087, Std: 3.824
	HausdorffDistance --> Mean: 2.108, Std: 0.964
	ContourMeanDistance --> Mean: 0.283, Std: 0.109
	SurfaceDistance --> Mean: 0.008, Std: 0.022
Test Phase Metrics [ANIMA] for lesion:
	Jaccard --> Mean: 0.468, Std: 0.173
	Dice --> Mean: 0.616, Std: 0.181
	Sensitivity --> Mean: 0.588, Std: 0.186
	Specificity --> Mean: 1.000, Std: 0.000
	PPV --> Mean: 0.696, Std: 0.241
	NPV --> Mean: 1.000, Std: 0.000
	RelativeVolumeError --> Mean: -1.583, Std: 45.172
	HausdorffDistance --> Mean: 4.900, Std: 4.649
	ContourMeanDistance --> Mean: 1.932, Std: 3.322
	SurfaceDistance --> Mean: 0.936, Std: 2.303
	PPVL --> Mean: 0.823, Std: 0.252
	SensL --> Mean: 0.917, Std: 0.220
	F1_score --> Mean: 0.857, Std: 0.227

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants