Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace nan with 0 to support loss plotting #13

Open
valosekj opened this issue Apr 16, 2024 · 0 comments
Open

Replace nan with 0 to support loss plotting #13

valosekj opened this issue Apr 16, 2024 · 0 comments

Comments

@valosekj
Copy link
Member

valosekj commented Apr 16, 2024

When training an AttentionUnet, the model collapsed to zero outputting nan for Validation Loss, Training Loss, and Soft Dice:

2024-04-16 07:20:41.119 | INFO     | __main__:run_training:251 - Epoch 995 	 Validation Loss: nan 	 Hard Dice: 0.00
2024-04-16 07:20:42.117 | INFO     | __main__:run_training:284 - Checkpoint saved at epoch 995
2024-04-16 07:21:20.168 | INFO     | __main__:run_training:245 - Epoch 996/1000 	 Training Loss: nan 	 Soft Dice: nan
2024-04-16 07:21:21.264 | INFO     | __main__:run_training:284 - Checkpoint saved at epoch 996
2024-04-16 07:22:05.010 | INFO     | __main__:run_training:245 - Epoch 997/1000 	 Training Loss: nan 	 Soft Dice: nan

nan is problematic because it cannot be plotted, resulting in a figure without the epochs with nan:

loss_plots

We could replace nans in such cases with 0. @naga-karthik, what do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant