You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When training an AttentionUnet, the model collapsed to zero outputting nan for Validation Loss, Training Loss, and Soft Dice:
2024-04-16 07:20:41.119 | INFO | __main__:run_training:251 - Epoch 995 Validation Loss: nan Hard Dice: 0.002024-04-16 07:20:42.117 | INFO | __main__:run_training:284 - Checkpoint saved at epoch 9952024-04-16 07:21:20.168 | INFO | __main__:run_training:245 - Epoch 996/1000 Training Loss: nan Soft Dice: nan2024-04-16 07:21:21.264 | INFO | __main__:run_training:284 - Checkpoint saved at epoch 9962024-04-16 07:22:05.010 | INFO | __main__:run_training:245 - Epoch 997/1000 Training Loss: nan Soft Dice: nan
nan is problematic because it cannot be plotted, resulting in a figure without the epochs with nan:
We could replace nans in such cases with 0. @naga-karthik, what do you think?
The text was updated successfully, but these errors were encountered:
When training an AttentionUnet, the model collapsed to zero outputting
nan
for Validation Loss, Training Loss, and Soft Dice:nan
is problematic because it cannot be plotted, resulting in a figure without the epochs withnan
:We could replace
nans
in such cases with0
. @naga-karthik, what do you think?The text was updated successfully, but these errors were encountered: