Build for v0.0.36-90/93 (MSVC)

[thliebig]

Edit:
a new build has been release v0.0.36-93!

Windows Build of v0.0.36-90 (SLOW, DO NOT USE):

Windows Build of v0.0.36-93:

• Windows build using the MSVC
• Allows build of python modules, builds for python 3.13 and 3.14 included

See Windows Python Install Instructions

---

This discussion was created from the release Build for v0.0.36-93 (MSVC).

[thliebig]

After countless hours I managed to setup a new Win11 machine (using docker) and get some MSVC up and get this compiled...

Please, all Windows users, give it some testing and feedback... I think we need to make a proper release soon and hopefully more frequently in the future...

[VolkerMuehlhaus]

Yes, I think it will be nice to have support for older Python versions as well. I got Python 3.13 from the Windows appstore, similar to the previously installed 3.10. And as a beginner mistake, I really damaged my environments now, so I openEMS is up and running and all my own software is crap now. Beginner mistake with swapping Python versions I guess ...

One good thing is that the previous openEMS conflict with scikit-rf is solved, the old wheel had some version conflict with incompatible internal formats in a library (numpy? not sure)

[VolkerMuehlhaus]

@thliebig First impression: this solver version feels ultra slow!

I need to check if I have some Windows results using the old version for 1:1 comparison. For now, here is a comparison with a recent Linux build on another machine.

Your new Windows build on 8-core Ryzen 7:

 ----------------------------------------------------------------------
 | openEMS 64bit -- version v0.0.36-90-g342e7ce
 | (C) 2010-2023 Thorsten Liebig <thorsten.liebig@gmx.de>  GPL license
 ----------------------------------------------------------------------
        Used external libraries:
                CSXCAD -- Version: v0.6.3-48-gcec39c1
                hdf5   -- Version: 1.12.0
                          compiled against: HDF5 library version: 1.12.0
                tinyxml -- compiled against: 2.6.2
                fparser
                boost  -- compiled against: 1_72
                vtk -- Version: 8.2.0
                       compiled against: 8.2.0

Create FDTD operator (compressed SSE + multi-threading)
FDTD simulation size: 138x124x28 --> 479136 FDTD cells
FDTD timestep is: 2.03998e-15 s; Nyquist rate: 8170 timesteps @3e+10 Hz
openEMS::SetupFDTD: Warning, the timestep seems to be very small --> long simulation. Check your mesh!?
Excitation signal length is: 93622 timesteps (1.90987e-10s)
Max. number of timesteps: 1000000000 ( --> 10681.3 * Excitation signal length)
Create FDTD engine (compressed SSE + multi-threading)
Running FDTD engine... this may take a while... grab a cup of coffee?!?
[@     2m24s] Timestep:         2042 || Speed:    6.8 MC/s (7.074e-02 s/TS) || Energy: ~1.14e-24 (- 0.00dB)
[@     3m40s] Timestep:         4084 || Speed:   12.9 MC/s (3.708e-02 s/TS) || Energy: ~3.14e-24 (- 0.00dB)
[@     4m31s] Timestep:         6126 || Speed:   19.1 MC/s (2.508e-02 s/TS) || Energy: ~6.06e-24 (- 0.00dB)
[@     5m11s] Timestep:         8168 || Speed:   24.6 MC/s (1.947e-02 s/TS) || Energy: ~1.44e-23 (- 0.00dB)
[@     5m43s] Timestep:        10210 || Speed:   30.2 MC/s (1.587e-02 s/TS) || Energy: ~1.73e-22 (- 0.00dB)
[@     6m11s] Timestep:        12252 || Speed:   35.3 MC/s (1.356e-02 s/TS) || Energy: ~1.43e-21 (- 0.00dB)

A very similar simulation on 16-core Ryzen 9 running Ubuntu 24.04:

---------------------------------------------------------------------- 
 | openEMS 64bit -- version v0.0.36-88-g332ca7d
 | (C) 2010-2023 Thorsten Liebig <thorsten.liebig@gmx.de>  GPL license
 ---------------------------------------------------------------------- 
	Used external libraries:
		CSXCAD -- Version: v0.6.3-45-g7cf8533
		hdf5   -- Version: 1.10.10
		          compiled against: HDF5 library version: 1.10.10
		tinyxml -- compiled against: 2.6.2
		fparser
		boost  -- compiled against: 1_83
		vtk -- Version: 9.1.0
		       compiled against: 9.1.0

Create FDTD operator (compressed SSE + multi-threading)
FDTD simulation size: 137x124x45 --> 764460 FDTD cells 
FDTD timestep is: 1.78698e-15 s; Nyquist rate: 9326 timesteps @3.00024e+10 Hz
openEMS::SetupFDTD: Warning, the timestep seems to be very small --> long simulation. Check your mesh!?
Excitation signal length is: 106877 timesteps (1.90987e-10s)
Max. number of timesteps: 1000000000 ( --> 9356.55 * Excitation signal length)
Create FDTD engine (compressed SSE + multi-threading)
Running FDTD engine... this may take a while... grab a cup of coffee?!?
[@        4s] Timestep:         2331 || Speed:  431.0 MC/s (1.773e-03 s/TS) || Energy: ~1.31e-24 (- 0.00dB)
[@        8s] Timestep:         4662 || Speed:  438.5 MC/s (1.744e-03 s/TS) || Energy: ~3.04e-24 (- 0.00dB)
[@       14s] Timestep:         9324 || Speed:  590.8 MC/s (1.294e-03 s/TS) || Energy: ~1.76e-23 (- 0.00dB)
[@       19s] Timestep:        13986 || Speed:  739.3 MC/s (1.034e-03 s/TS) || Energy: ~1.51e-21 (- 0.00dB)
[@       23s] Timestep:        18648 || Speed:  844.8 MC/s (9.049e-04 s/TS) || Energy: ~2.52e-20 (- 0.00dB)
[@       29s] Timestep:        25641 || Speed:  899.9 MC/s (8.495e-04 s/TS) || Energy: ~1.48e-19 (- 0.00dB)
[@       33s] Timestep:        32634 || Speed: 1132.1 MC/s (6.752e-04 s/TS) || Energy: ~2.17e-18 (- 0.00dB)
[@       37s] Timestep:        37296 || Speed:  885.4 MC/s (8.634e-04 s/TS) || Energy: ~1.85e-17 (- 0.00dB)

[VolkerMuehlhaus]

@thliebig Just ran another model that I know well, also ultra slow using the new build:

Running FDTD engine... this may take a while... grab a cup of coffee?!?
[@     4m40s] Timestep:          993 || Speed:    1.3 MC/s (2.821e-01 s/TS) || Energy: ~1.17e-24 (- 0.00dB)
[@     7m08s] Timestep:         1986 || Speed:    2.4 MC/s (1.493e-01 s/TS) || Energy: ~6.11e-24 (- 0.00dB)

Something seems to be wrong. My cup is too small !?

[VolkerMuehlhaus]

@thliebig Please don't wait for me, I have stopped using or testing this Windows build. My openEMS installation on Windows is broken and it's too difficult for me to handle multiple installs in parallel.

[thliebig]

@VolkerMuehlhaus okay that is very odd... and sorry for your python mixup. I never use the Appstore. I e.g. just downloaded the python installer from python.org... It handles multiple python just fine...

[thliebig]

This build is very slow unfortunately as @VolkerMuehlhaus already found out.
I have identified the issue. The MSVC (Visual Studio 2019) I build this with seems to completely not understand how the new Array interface works that @biergaizi thankfully implemented. Switching from the legacy []-operator to the better ()-operator seems to fix this.
Thus I will work on getting rid of the old operator, I had this planned anyhow as the new operator is more readable too in my opinion...

Any comments on this e.g. from @biergaizi.

Performance of nested operator[] should be the same as flat operator() (identical assembly code generation even on MSVC).

It seems I found a MSVC that does not... Oh well, a Microsoft tool not working right... a surprise to exactly no one 😏

[VolkerMuehlhaus]

MSVC (Visual Studio 2019)

I have no experience with MSVC, but wonder if a newer version might create more efficient code? Isn't there a newer community edition suitable for this FOSS work?

[thliebig]

Well maybe, but that is just not good enough... VS2019 and even 2017 have to produce reasonable results...

I have an initial version. It seems to even gain a bit of speed with this changes. But it will be a massive change with lots of potential for regression (stuff breaking that used to work).

[thliebig]

I have "released" a new build v0.0.36-93 which should have fixes for the issues discussed here. Please give it a try and report here, thanks

[VolkerMuehlhaus]

Please give it a try and report here

Much better now!! Speed with v0.0.36-93 looks reasonable given the difference in CPU between my Windows and Linux machines.

Starting FDTD simulation for excitation  [1]
 ----------------------------------------------------------------------
 | openEMS 64bit -- version v0.0.36-93-g7b9cd51
 | (C) 2010-2025 Thorsten Liebig <thorsten.liebig@gmx.de>  GPL license
 ----------------------------------------------------------------------
        Used external libraries:
                CSXCAD -- Version: v0.6.3-48-gcec39c1
                hdf5   -- Version: 1.12.0
                          compiled against: HDF5 library version: 1.12.0
                tinyxml -- compiled against: 2.6.2
                fparser
                boost  -- compiled against: 1_72
                vtk -- Version: 8.2.0
                       compiled against: 8.2.0

Create FDTD operator (compressed SSE + multi-threading)
FDTD simulation size: 109x79x31 --> 266941 FDTD cells
FDTD timestep is: 1.85951e-15 s; Nyquist rate: 2444 timesteps @1.1002e+11 Hz
openEMS::SetupFDTD: Warning, the timestep seems to be very small --> long simulation. Check your mesh!?
Excitation signal length is: 28012 timesteps (5.20885e-11s)
Max. number of timesteps: 1000000000 ( --> 35699 * Excitation signal length)
Create FDTD engine (compressed SSE + multi-threading)
Running FDTD engine... this may take a while... grab a cup of coffee?!?
[@        4s] Timestep:         3055 || Speed:  181.9 MC/s (1.467e-03 s/TS) || Energy: ~8.52e-24 (- 0.00dB)
[@        8s] Timestep:         6110 || Speed:  193.4 MC/s (1.380e-03 s/TS) || Energy: ~1.05e-21 (- 0.00dB)
[@       12s] Timestep:         9776 || Speed:  228.3 MC/s (1.169e-03 s/TS) || Energy: ~2.63e-19 (- 0.00dB)
[@       17s] Timestep:        13442 || Speed:  243.7 MC/s (1.095e-03 s/TS) || Energy: ~1.36e-18 (- 0.00dB)
[@       21s] Timestep:        17719 || Speed:  253.6 MC/s (1.053e-03 s/TS) || Energy: ~1.20e-18 (- 1.69dB)
[@       25s] Timestep:        21996 || Speed:  256.5 MC/s (1.041e-03 s/TS) || Energy: ~1.24e-19 (-11.53dB)
[@       30s] Timestep:        26273 || Speed:  247.2 MC/s (1.080e-03 s/TS) || Energy: ~2.41e-21 (-28.66dB)
Multithreaded Engine: Best performance found using 6 threads.
[@       35s] Timestep:        30550 || Speed:  255.0 MC/s (1.047e-03 s/TS) || Energy: ~4.12e-22 (-36.32dB)
[@       39s] Timestep:        34827 || Speed:  257.4 MC/s (1.037e-03 s/TS) || Energy: ~3.33e-22 (-37.25dB)
[@       44s] Timestep:        39104 || Speed:  250.2 MC/s (1.067e-03 s/TS) || Energy: ~8.22e-22 (-33.32dB)
[@       48s] Timestep:        43381 || Speed:  252.4 MC/s (1.058e-03 s/TS) || Energy: ~4.98e-22 (-35.50dB)
[@       53s] Timestep:        47658 || Speed:  253.3 MC/s (1.054e-03 s/TS) || Energy: ~3.15e-22 (-37.49dB)
[@       57s] Timestep:        51935 || Speed:  254.6 MC/s (1.048e-03 s/TS) || Energy: ~6.82e-23 (-44.13dB)
Time for 51935 iterations with 266941.00 cells : 57.57 sec
Speed: 240.82 MCells/s
FDTD simulation completed successfully for excitation  [1]
Creating  S-parameter file

[thliebig]

That was fast testing... thanks

[biergaizi]

I strongly disagree with the change of the memory layout of my original implementation. According to the commit note,

Note 2: The array_nijk uses the "correct" memory layout again.
If for better performance the engine wants to use a ijkn layout, a new
class (specifically for the engine?) needs to be created and used.
--> More performance tests needed

The ijkn memory layout has been chosen with extensive benchmarking across 12 different hardware platforms over a period of multiple months after considerable monetary expenses. In all tests, the ijkn layout performs almost consistently better in comparison to the nijk layout. This has been comprehensively documented in an earlier Pull Request #117, supported by hard data. These benchmark data has also proven the fact that calculating the linear address using operator() instead of manually has no performance impact because the bottlenecks exists elsewhere, not the indexing.

Based on the considerations above, this is why the new nD array uses the ijkn physical layout, while keeping the original nijk syntax.

a new class (specifically for the engine?) needs to be created and used.

I think It's impractical to create a new engine for each minor technical variations (as you've said in the past, you thought there are already "too many engines", while I wanted to add a separate engine for the experimental change - it's confusing that our opinion has suddenly reversed by 180 degrees after that discussion). If the existing change has proven to be safe with positive returns on nearly all platforms, I believe it's safe to apply that change in the existing engine.

[thliebig]

@biergaizi you misunderstood me. I do not want to create yet another engine. What I meant is that we/I need to create another "array_ijkn" (for example) or an "array_4D" which should internally use the ijkn layout. This one should be used exclusively for the engine and might even be changeable by compiler option or maybe even runtime option.

The reason is that I have started using the array_nijk for field storage and export too and it is very practical to directly use the data buffer for the hdf5 export. But that needs the "correct" memory layout. And with correct I mean the memory layout that the class name suggests.
And no, this is not up for debate.

But you do not need to do anything here. I will make this change soon...

I hope that clarifies things?

[biergaizi]

What I meant is that we/I need to create another "array_ijkn" (for example) or an "array_4D" which should internally use the ijkn layout. [...] The reason is that I have started using the array_nijk for field storage and export too and it is very practical to directly use the data buffer for the hdf5 export. But that needs the "correct" memory layout.

Yes, this clarifies my problem perfectly. HDF5 internally expects the binary NIJK layout, so we really need an NIJK array class, otherwise the data cannot be used with field dumps. For the engine itself, an IJKN class is still used exclusively with it. So case closed, no more problems.

---

But since we're talking about Windows... This situation highlights that we still don't have any solution regarding to Windows support in development. The biggest problem right now is that we do not have any idea to support or reject any technical changes on Windows, because we don't have any Windows tests. So if the test problem can be fixed, the decisions can be made objectively based on data, instead of subjective "code feelings", or an extrapolation from Linux/Unix to Windows.

When your time allows, please try documenting the entire Windows workflow, including the exact MSVC / IDE version used, or the dependent pre-built VTK artifacts. Even a partial documentation or workflow (e.g. screenshots or video captures) is better than no documentation. When the documentation is ready, I'll try replicate the entire Windows workflow locally. It may be a week-long struggle but it's not magic, so that should eventually succeed. Once that goal is achieved, I'll design a performance test suite for MSVC, similar to the Unix/Linux test I did.

All technical decisions such as operator() or operator[], or NIJK or IJKN, will be made based on objective benchmark data. If its performance on Windows is consistent with Unix/Linux, my "ArrayLib" decision would be used. If not, I'll then investigate if it can be fixed. If still not, the original openEMS design is kept as is.

You said that new designs or new compiler versions have risks of introducing regressions, I propose that we shouldn't keep the original decision because the it was the original one, instead, this problem can be be solved using the same objective test suite. I'll code some representative test cases to compare the memory dumps from old and new MSVC compilers, if these test cases show no sign of troubles, I believe we have all the reason to upgrade the MSVC version (or switching to a Visual Studio + clang workflow, but that is pending investigation).

Overall, Windows is currently the blind spot of the project because of a lack of a reproducible workflow. If we can get the workflow documented, we're free to make changes without worrying about compatibility. Otherwise, if we do not have a documented Windows workflow, there's no data besides "it doesn't work on my machine."

[thliebig]

When your time allows, please try documenting the entire Windows workflow, including the exact MSVC / IDE version used.

I fully agree. I will try to create such a workflow doc and create a zip with all dependencies for download...

[thliebig]

Btw. if you want to have a look at the changes I talked about: I just pushed them to the "WIP" (work in progress) branch.
(Please be aware that I always fore-push update that branch...)

[thliebig]

@biergaizi, as promised I have created a new array called ArrayENG (with ArrayNIJK as base) specifically for the Operator&Engine. It can use different layouts, but for now it uses IJKN (which on my machine is about 10% slower for whatever reason, I guess my Ryzen1 is just way too old).
But currently I only applied it for the optimized engines... The very basic engine still uses the ArrayNIJK...

Please have a look at the changes. I'm not really good at template programming...