NaN error

[JonathanLehner]

I disabled anomaly detection since it crashed.
Without it I get the error:

Warning: NaN or Inf found in input tensor.
Error: svd_cuda: the updating process of SBDSDC did not converge (error: 3)

this was in epoch 2, it does not always happen. With distort_images: true

[JonathanLehner]

with anomaly detection I get the warning:
UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
warnings.warn('Anomaly Detection has been enabled. '

[JonathanLehner]

and after some time I get "RuntimeError: Function 'PowBackward0' returned nan values in its 0th output."

[karfly]

Hi, @JonathanLehner.
I also noticed such instability during training an Algebraic Model, but I got such error only after >10 epochs.

[JonathanLehner]

So how can we fix this error? I cannot finish training with it

…

On Tue, Nov 10, 2020 at 4:05 PM Karim Iskakov ***@***.***> wrote: Hi, @JonathanLehner <https://github.com/JonathanLehner>. I also noticed such instability during training an Algebraic Model, but I got such error only after >10 epochs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#112 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFQOPHAVONED6GQPXCSP7GDSPFJEDANCNFSM4ST2FCHA> . [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": " #112 (comment)", "url": " #112 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": " https://github.com" } } ]