Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. #12

Open
hgy123 opened this issue Aug 12, 2023 · 0 comments

Comments

@hgy123
Copy link

hgy123 commented Aug 12, 2023

Hello!
when using the IPN-V2 network, I would like to obtain segmentation results for Largevessels and change the code from five class network to a two class network(the results of the five class network are normal). The following error occurred:
C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [832,0,0] Assertion t >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [480,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [481,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [929,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [930,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [33,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [800,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [804,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [805,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [704,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [705,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [706,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [903,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [904,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [329,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [330,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [129,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [130,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [131,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [232,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [233,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [234,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [104,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [105,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [207,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [208,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [683,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [684,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [685,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [780,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [781,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [1006,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [1007,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [555,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [556,0,0] Assertiont >= 0 && t < n_classesfailed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\NLLLoss2d.cu:103: block: [0,0,0], thread: [6,0,0] Assertiont >= 0 && t < n_classesfailed. Traceback (most recent call last): File "D:\download\Code\3D Baselines\train.py", line 161, in <module> train_net(net=net,device=device) File "D:\download\Code\3D Baselines\train.py", line 68, in train_net loss = Loss_CE(pred, train_annotations)+ 0.6 * Loss_DSC(pred, train_annotations) File "C:\Users\hgyy\AppData\Roaming\Python\Python39\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "D:\download\Code\3D Baselines\utils.py", line 226, in forward target=make_one_hot(target.long(),shape) File "D:\download\Code\3D Baselines\utils.py", line 160, in make_one_hot result.scatter_(1, input.cpu(), 1) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile withTORCH_USE_CUDA_DSA to enable device-side assertions.
My environment: python=3.9.16 torch=2.0.0 torchvision=0.15.1
Where did I make a mistake?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant