Add Build and Run-Time Options to Control Floating-Point Contraction and Flush-to-Zero

[biergaizi]

When debugging or optimizing the FDTD kernel, ensuring code correctness is of interest. One of the best validation method is to ensure all floating-point numbers are bit-identical to previous versions. However, it's well known that floating-point operations are inexact, the significant digits depend on compiler code generation and FPU settings.

In the openEMS codebase, there are two obstacles that may prevent numerical reproducibility.

1. FPU Denormal Flush-to-Zero: On x86_64 CPUs, very small floating-point values require microcode assists, with a heavy performance penalty. Thus it's standard practice to enable Flush-to-Zero. For troubleshooting, there should be an advanced command-line flag to disable it.

2. FP Contraction: The floating-point operations may be reordered is several ways, one of them is the famous -fun-unsafe-math-optimization and -Ofast by GCC. But even when using only safe math optimization, FMA instructions may still change the numerical precision. This behavior can be disabled using a command-line flag to turn off "FP Contraction". This flag should be added to the default CXXFLAGS in a CMake debugging-mode build.

Implementing both features is not sufficient for numerical reproducibility (the SSE engine's VV, VI, IV, VV operator compression still changes the order of computation, from my personal experience), but are necessary.

Both features exist in my private codebase for over a year, for development and testing of my never-finished engine rewrite project. I'm now posting this as an issue serving both as a developer note and a TODO list for myself to eventually upstream it.

[thliebig]

I am a bit surprised by this as I would have never imagined to test like this due to the mentioned issues with floating points. And it is my experience that if you make an error with coefficient calculations, the engine just blows up and that's it.
In my opinion we should invest in a higher level test-suite that is comparing final output against expected results.
This is e.g. what international standards like IEC/IEEE 62704-1 and 63195-2&4 require to validate FDTD codes to be compliant for human exposure calculations.

[biergaizi]

I am a bit surprised by this as I would have never imagined to test like this due to the mentioned issues with floating points.

I've used this technique successfully to identify very small errors caused by off-by-1 memory corruptions, which do not show up as a difference in the post-processing simulation results because they only affect very few cells. I think it can be a useful technique if your changes are meant to be 100% equivalent to the previous version, such as a new array implementation. But if the kernel has an algorithmic-level change, it won't work as you said.

In my opinion we should invest in a higher level test-suite that is comparing final output against expected results. This is e.g. what international standards like IEC/IEEE 62704-1 and 63195-2&4 require to validate FDTD codes to be compliant for human exposure calculations.

Yes, I think we need both. Numerical check only for internal developer use, and high-level check for validation.