RuntimeError during Training with MonaiLabel

[yys9905]

I'm currently training a segmentation model with MONAI Label. After each epoch ends, I encounter the following three errors 3 to 5 times:

Exception in thread Thread-13:
Traceback (most recent call last):
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/intensity/dictionary.py", line 1057, in __call__
    d[key] = self.converter(d[key])
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/intensity/array.py", line 1200, in __call__
    out_t: torch.Tensor = gaussian_filter(img_t.unsqueeze(0)).squeeze(0)
  File "/home/ean/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 606, in forward
    return separable_filtering(x=x, kernels=_kernel)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 250, in separable_filtering
    return _separable_filtering_conv(x, _kernels, pad_mode, spatial_dims - 1, spatial_dims, _paddings, n_chs)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/networks/layers/simplelayers.py", line 199, in _separable_filtering_conv
    padded_input = F.pad(input_, _sum_reversed_padding_repeated_twice, mode=pad_mode)
  File "/home/ean/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 4364, in _pad
    return _VF.constant_pad_nd(input, pad, value)
RuntimeError: The input size 0, plus negative padding 0 and 0 resulted in a negative output size, which is invalid. Check dimension 2 of your input.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 322, in __call__
    result = execute_compose(
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.intensity.dictionary.GaussianSmoothd object at 0xfffe4a8aaf40>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/data/thread_buffer.py", line 49, in enqueue_values
    for src_val in self.src:
  File "/home/ean/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in __next__
    data = self._next_data()
  File "/home/ean/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 570, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/ean/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ean/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ean/.local/lib/python3.8/site-packages/monai/data/dataset.py", line 109, in __getitem__
    return self._transform(index)
  File "/home/ean/.local/lib/python3.8/site-packages/monai/data/dataset.py", line 95, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/home/ean/.local/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0xfffe4a00b250>

Despite these errors, the next epoch proceeds and I can see the train_loss gradually but surely decreasing. I am wondering if it's okay to continue training like this despite the errors. Additionally, I would like to know how to fix these errors.

Please advise.

[diazandr3s]

Hi @yys9905,

Sorry, I missed this thread.
Which model of the radiology app are you using? DeepEdit or Segmentation? Which type of dataset are you using?
In any case, you may need to modify/update the pre-transforms so they are customised to your dataset.

Let us know,

[amitjc]

Encountered similar error while trying to train the Radiology segmentation model.

@diazandr3s seek your inputs to overcome the issue.

In any case, you may need to modify/update the pre-transforms so they are customised to your dataset.

How to modify the pre-transforms?

Please advice.

Thank you.

Best Regards,
Amit.

[diazandr3s]

Hi @amitjc,

Could you please ellborate a bit more on this?

Encountered similar error while trying to train the Radiology segmentation model.

Could you please share the logs and more information about the dataset and use case?

How to modify the pre-transforms?

If using the segmentation model in the radiology app, you can either use the default pre-transforms or modify/add custom ones here:

MONAILabel/sample-apps/radiology/lib/trainers/segmentation.py

Lines 73 to 95 in 159e142

	def train_pre_transforms(self, context: Context):
	return [
	LoadImaged(keys=("image", "label")),
	NormalizeLabelsInDatasetd(keys="label", label_names=self._labels), # Specially for missing labels
	EnsureChannelFirstd(keys=("image", "label")),
	EnsureTyped(keys=("image", "label"), device=context.device),
	Orientationd(keys=("image", "label"), axcodes="RAS"),
	Spacingd(keys=("image", "label"), pixdim=self.target_spacing, mode=("bilinear", "nearest")),
	NormalizeIntensityd(keys="image", nonzero=True),
	CropForegroundd(
	keys=("image", "label"),
	source_key="image",
	margin=10,
	k_divisible=[self.roi_size[0], self.roi_size[1], self.roi_size[2]],
	),
	GaussianSmoothd(keys="image", sigma=0.4),
	ScaleIntensityd(keys="image", minv=-1.0, maxv=1.0),
	RandSpatialCropd(
	keys=["image", "label"],
	roi_size=[self.roi_size[0], self.roi_size[1], self.roi_size[2]],
	random_size=False,
	),
	SelectItemsd(keys=("image", "label")),

Hope this helps,

[amitjc]

Hello @diazandr3s ,

Thank you for your responses.

Could you please elaborate a bit more on this?

I've been training MONAILabel for liver, HCC, Hepatic veins and portal vein segmentation, when i encountered the error. No changes have been made to the data or segmentations during training. Similar data and segmentations were used for training on 20 similar cases before!

Could you please share the logs and more information about the dataset and use case?

Please find the error log here.

Look forward to further guidance.

Thanks and Regards,
Amit.

[diazandr3s]

Hi @amitjc,

Can you please change this argument to source_key="image" and try again?

Otherwise, I'd suggest commenting out the whole transform:

MONAILabel/sample-apps/radiology/lib/trainers/segmentation.py

Lines 119 to 124 in 159e142

	CropForegroundd(
	keys=("image", "label"),
	source_key="label",
	margin=10,
	k_divisible=[self.roi_size[0], self.roi_size[1], self.roi_size[2]],
	),

Let us know,

[amitjc]

Thanks a ton @diazandr3s.

Can you please change this argument to source_key="image" and try again?

Changing this parameter, as suggested, has resolved the problem.

Best Wishes,
Amit.