Not possible to send infs and nans

[wjaskowski]

What is the reason to don't allow sending inf's and nan's as metric values? I imagine that it is impossible to plot them but this is still some information.

[kamil-kaczmarek]

Hi @wjaskowski thanks for reaching out.

Indeed right now we do not accept NaN/None/(+/-)Inf values.

However, we had some internal discussion about it.

One idea is to make it similar to what TensorBoard does: each Nan/None is displayed as a graphic icon, like a triangle or star. Location on "y" axis is determined by preceding numeric value.
Location on "x" axis is preceding value +1.

what do you think?

[wjaskowski]

Sounds good. You might also want to consider placing those triangles on the bottom/top of the visible plot.

But visualization is one thing - the most important one is to be able to send and download the data.

[kamil-kaczmarek]

Thanks for suggestion 🙂

We will consider it as well.

[fwindolf]

This is a somewhat stale issue but the first thing that comes up when you google the behaviour.

Any news/updates on this?

[SiddhantSadangi]

Hello @fwindolf ,
This feature request is quite deep in our backlog, so currently, there is no ETA for it, unfortunately.
Is this behavior a blocker for your workflows?

[fwindolf]

Not really a blocker, but having NaNs occur during training for whatever reason seems to be a common enough problem to justify experiment tracking not completely breaking imo.

So a

run["my_metric"].append(1.0)
run["my_metric"].append(float("nan"))
run["my_metric"].append(3.0)

will only show the 1.0. I see why adding NaN support would open up quite a few edge cases for visualizations etc, but maybe a short term fix could be simply ignoring NaN, +-inf etc during the list iteration when syncing the metric.

[SiddhantSadangi]

Would replacing the nan/inf values with 0/some high-end value while logging be a viable workaround in your case?
Something like:

import math
metric = float("nan")
if math.isna(metric ):
    run["my_metric'].append(0)

I've also submitted your feedback around ignoring NaN/inf to the product team. Thank you :)

[SiddhantSadangi]

Hello @fwindolf ,
Just checking if the above workaround works for you

[fwindolf]

Sorry I missed the notification of the last comment.

We solved it by not logging nans as 0, inf as a big number which is okay for now. It skews the readability of graphs but it's better than not seeing anything.

Thanks for forwarding the issue!

[SiddhantSadangi]

Did you mean not logging nans as 0? :)

[rschiewer]

Hi there! Is this something that is actively worked on? I experienced some solid trouble recently because my training process diverged and the logging did not show where the NANs started to show up at first. This can be very valuable information for debugging. What is bad about the way e.g. tensorboard handles NAN/inf?

While the workaround is fine in most cases, my model showed values of around zero all the time and then started to diverge so replacing NANs with zeros in principle works but is not ideal in my situation.

[SiddhantSadangi]

Hello @rschiewer ,

The product team is currently scoping this. This seems to involve relatively high engineering effort, so there is no ETA as of now, unfortunately :(

In your case, since the values hover around zero, can you replace NaNs with a high value so that they show up in charts, and you can then know when your model starts diverging?

[SiddhantSadangi]

Hey everyone! Just a quick update here.

Neptune v1.8.3 now skips trying to log NaN and Inf values and throws a warning instead. This means you no longer have to check for nan/inf values in your code🥳

[DanTremonti]

Bump!

Neptune v1.8.3 now skips trying to log NaN and Inf values and throws a warning instead.

Thanks for this change team.

One idea is to make it similar to what TensorBoard does: each Nan/None is displayed as a graphic icon, like a triangle or star.

This sounds like a very useful way to visualize out of range values. I suppose logging things going wrong is a crucial part of experimenting and in this way it will also help with comparison against different experiments. Eager to hear your thoughts on this.

[SiddhantSadangi]

Hey @DanTremonti 👋
As you can see, this is a long-standing feature request 😃

There are several aspects we need to consider before finalizing a solution here, mainly related to front-end performance, as any custom rendering will need to be managed at the front end.

Although we are aware of this request, we don't anticipate any changes in the short term as we are currently focusing on performance improvements.

I'll keep this thread updated ✅

[DanTremonti]

Thank you for the quick response @SiddhantSadangi
I'm happy to hear that this feature request is being consider :)

[brynhayder]

Thank you for your work.

I don't see this skipping behaviour in the docs of either append or FloatSeries. For the benefit of future users, please could you add it?

When the warning says the values are skipped, is the counter on the series advanced or not? Considering logging loss values throughout training, you might want the step counter to remain consistent regardless of whether the loss was nan. The series would then contain missing values, which the user knows should at least be invalid float. Either way, please can you make the behaviour clear in the docs?

[SiddhantSadangi]

Hey @brynhayder 👋
Thanks for your comment!

The skipping behaviour is mentioned here: https://docs.neptune.ai/help/value_of_unsupported_type/#working-around-none-inf-or-nan. However, we should also mention this with the API reference of append. I'll pass this on to the docs team ✅

The counter on the series is currently not advanced on encountering nan or inf values. But your feedback does make sense and I totally resonate. I'll pass this feedback to the product team ✅