Scale Microgrid Test Error

[Paulm991]

Trying to run the ScaleMicrogrid test is throwing an error on the 'develop' branch. The error output is as follows:

"ScaleMicrogrid" start time: Jul 10 14:10 EDT
Output:
Test the Relative Error
Test with Nsize = 2 passes!
Test the Relative Error
Test with Nsize = 4 passes!
Test the Relative Error
Test with Nsize = 8 fails!
Some tests fail!!

Test time = 3.82 sec
Test Failed.
"ScaleMicrogrid" end time: Jul 10 14:10 EDT
"ScaleMicrogrid" time elapsed: 00:00:03
This was performed with the coin-or/Ipopt#12 release of GridKit, Sundials 6.7.0, Ipopt 3.14.16, and SuiteSparse 5.10.1.

[reid-g]

Some updates on this. I have verified that all model parameters are correct and rhs outputs are correct. So these are not it.

The "true" solution vectors I generate are from MATLAB based ODE form of the model. Since no analytical solution to the model available . No Jacobian is used. This is done with extreme tolerances. Relative Tolerance 1e-14 and Absolute Tolerance 1e-14. I have done both ode15s and ode23tb solutions. I have done also the new MATLAB integrated IDA way as well. They all match up correctly with only some numerical rounds difference between.

This same error appears in hardwired setups (no GridKit) as well. This is for both ODE and DAE forms hardwired.

Still actively looking to see what is the issue.

[pelesh]

Unfortunately, #30 did not fix this issue. See this log.

[nkoukpaizan]

The test failure seems to be non-deterministic. Different versions of SUNDIALS and/or different machines produce different error norms.

For example, from my runs on Frontier:

Test the Relative Error for N = 2
2-Norm Relative Error: 1.85946e-06
Test with Nsize = 2 passes!
Test the Relative Error for N = 4
2-Norm Relative Error: 7.05324e-06
Test with Nsize = 4 passes!
Test the Relative Error for N = 8
2-Norm Relative Error: 6.77802e-06
Test with Nsize = 8 passes!
All tests pass!!

But from the Github actions:

12: Test the Relative Error for N = 2
12: 2-Norm Relative Error: 2.80821e-06
12: Test with Nsize = 2 passes!
12: Test the Relative Error for N = 4
12: 2-Norm Relative Error: 2.25243e-06
12: Test with Nsize = 4 passes!
12: Test the Relative Error for N = 8
12: 2-Norm Relative Error: 0.000147299
12: Test with Nsize = 8 fails!

Both of these are with SUNDIALS v7.1.1.

[reid-g]

The test failure seems to be non-deterministic. Different versions of SUNDIALS and/or different machines produce different error norms.

For example, from my runs on Frontier:

Test the Relative Error for N = 2
2-Norm Relative Error: 1.85946e-06
Test with Nsize = 2 passes!
Test the Relative Error for N = 4
2-Norm Relative Error: 7.05324e-06
Test with Nsize = 4 passes!
Test the Relative Error for N = 8
2-Norm Relative Error: 6.77802e-06
Test with Nsize = 8 passes!
All tests pass!!

But from the Github actions:

12: Test the Relative Error for N = 2
12: 2-Norm Relative Error: 2.80821e-06
12: Test with Nsize = 2 passes!
12: Test the Relative Error for N = 4
12: 2-Norm Relative Error: 2.25243e-06
12: Test with Nsize = 4 passes!
12: Test the Relative Error for N = 8
12: 2-Norm Relative Error: 0.000147299
12: Test with Nsize = 8 fails!

Both of these are with SUNDIALS v7.1.1.

This appears to be due to compiler optimizations and how rounding is handled. I am able to exactly (every digit) replicate the results from Github actions on my machine by simply changing optimization flag from O0 to O1. I did not catch this since I was working in O0. My results were the same going from SUNDIALS v6.6.0 to v7.2.0 at optimization level O0.

There are trig functions utilized in the DG component. I suspect compiler optimizations are handling them differently. Plus the time scales are quite small in the initial time. The errors are worse on N=8 vs N=4 and N=2

I have found two routes (at least for my machine) to handle the problem.

• Reduce the SCALE_MICROGRID_REL_TOL and SCALE_MICROGRID_ABS_TOL to 1e-6. This gives consistent results between O0 and O1. I am in favor of this option as we are already "bottoming out".
• Add the compiler flag -mfpmath=387 to increase temporary float precision. This prevents the large increase in error in N=8 at the current tolerances. However, this would only be useful for this problem and may slow performance.

[pelesh]

I have found two routes (at least for my machine) to handle the problem.

• Reduce the SCALE_MICROGRID_REL_TOL and SCALE_MICROGRID_ABS_TOL to 1e-6. This gives consistent results between O0 and O1. I am in favor of this option as we are already "bottoming out".
• Add the compiler flag -mfpmath=387 to increase temporary float precision. This prevents the large increase in error in N=8 at the current tolerances. However, this would only be useful for this problem and may slow performance.

I suggest you make a PR with your solution to nicholson/buildsystem branch. That way we test it instantly. Please document your choices for selecting specific tolerances.

I suggest building code as RelWithDebInfo in CMake. With GCC, it will set flags to -g -O2. I would also test with -O3, as well.