LAPACK tests are failing with OpenBLAS-0.3.20 and GCC-11.3.0

[maxim-masterov]

Creating this issue to properly log all the progress.

How it started

It was observed that the VASP6 installation with foss/2022a lead to inaccurate results. After some digging the culprit was found - DGGEV subroutine from LAPACK. To simplify debugging of the problem we isolated LAPACK tests from the official netlib distribution (3.10.1) and started to run them using different combinations of compiler flags and OpenBLAS versions.

What we have

The following tests are performed on AMD EPYC ROME (zen2 architecture):

• OpenBLAS/0.3.15-GCC-10.3.0 (taken from foss/2021a) results in ~130 failed tests:

[wimr@int1 OUTPUT]$ grep failed foss-2021a-openblas-0.3.15/* | grep -v "error exits"
foss-2021a-openblas-0.3.15/ced.out: CEV:    4 out of  1096 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/ced.out: CVX:   24 out of  5196 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/cgd.out: CGV drivers:      5 out of   1092 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/cgd.out: CGV drivers:      6 out of   1092 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/zed.out: ZEV:    8 out of  1100 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/zed.out: ZVX:   36 out of  5208 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/zgd.out: ZGV drivers:     25 out of   1092 tests failed to pass the threshold
foss-2021a-openblas-0.3.15/zgd.out: ZGV drivers:     22 out of   1092 tests failed to pass the threshold 

• OpenBLAS/0.3.20-GCC-11.3.0 (taken from foss/2022a) results in ~4.2k failed tests

[wimr@int1 OUTPUT]$ grep failed foss-2022a-openblas-0.3.20/* | grep -v "error exits"
foss-2022a-openblas-0.3.20/ced.out: CEV:   30 out of  1122 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/ced.out: CVX:  194 out of  5366 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgd.out: CGV drivers:    129 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgd.out: CGV drivers:    135 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgd.out: CGS drivers:    123 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgd.out: CGS drivers:    126 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgg.out: CGG:  119 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgg.out: CGG:  115 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgg.out: CGG:  117 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/cgg.out: CGG:  116 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgd.out: DGS drivers:    129 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgd.out: DGS drivers:    129 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgd.out: DGV drivers:    166 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgd.out: DGV drivers:    171 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgg.out: DGG:  143 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgg.out: DGG:  146 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgg.out: DGG:  163 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/dgg.out: DGG:  150 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgd.out: SGS drivers:    144 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgd.out: SGS drivers:    132 out of   1560 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgd.out: SGV drivers:    173 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgd.out: SGV drivers:    186 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgg.out: SGG:  153 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgg.out: SGG:  140 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgg.out: SGG:  147 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/sgg.out: SGG:  150 out of  2184 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/zed.out: ZEV:   50 out of  1142 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/zed.out: ZVX:  296 out of  5468 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/zgd.out: ZGV drivers:     52 out of   1092 tests failed to pass the threshold
foss-2022a-openblas-0.3.20/zgd.out: ZGV drivers:     50 out of   1092 tests failed to pass the threshold

• OpenBLAS/0.3.15-GCC-11.3.0 (new build) results in ~4.2k failed tests

@zao got similar results on Ryzen 9 3900X (zen2 desktop) when built LAPACK tests with full foss/2022a and buildenv that picked up FlexiBLAS as the USE_OPTIMIZED_BLAS implementation:

[easybuild@eb-rocky8 build-lapack-ob-0.3.20-benv]$ grep failed TESTING/testing_results.txt
 SGG:  163 out of  2184 tests failed to pass the threshold
 SGG:  159 out of  2184 tests failed to pass the threshold
 SGG:  162 out of  2184 tests failed to pass the threshold
 SGG:  155 out of  2184 tests failed to pass the threshold
 SGS drivers:    144 out of   1560 tests failed to pass the threshold
 SGS drivers:    159 out of   1560 tests failed to pass the threshold
 SGV drivers:    180 out of   1092 tests failed to pass the threshold
 SGV drivers:    184 out of   1092 tests failed to pass the threshold
  STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 DGG:  161 out of  2184 tests failed to pass the threshold
 DGG:  150 out of  2184 tests failed to pass the threshold
 DGG:  166 out of  2184 tests failed to pass the threshold
 DGG:  151 out of  2184 tests failed to pass the threshold
 DGS drivers:    135 out of   1560 tests failed to pass the threshold
 DGS drivers:    156 out of   1560 tests failed to pass the threshold
 DGV drivers:    174 out of   1092 tests failed to pass the threshold
 DGV drivers:    172 out of   1092 tests failed to pass the threshold
 CEV:   30 out of  1122 tests failed to pass the threshold
 CVX:  194 out of  5366 tests failed to pass the threshold
 CGG:  122 out of  2184 tests failed to pass the threshold
 CGG:  118 out of  2184 tests failed to pass the threshold
 CGG:  129 out of  2184 tests failed to pass the threshold
 CGG:  121 out of  2184 tests failed to pass the threshold
 CGV drivers:    135 out of   1092 tests failed to pass the threshold
 CGV drivers:    121 out of   1092 tests failed to pass the threshold
 CGS drivers:    126 out of   1560 tests failed to pass the threshold
 CGS drivers:    135 out of   1560 tests failed to pass the threshold
 ZHS:    1 out of  1764 tests failed to pass the threshold
 ZHS:    1 out of  1764 tests failed to pass the threshold
 ZHS:    1 out of  1764 tests failed to pass the threshold
 ZHS:    1 out of  1764 tests failed to pass the threshold
 ZEV:   50 out of  1142 tests failed to pass the threshold
 ZVX:  296 out of  5468 tests failed to pass the threshold
 ZGV drivers:     54 out of   1092 tests failed to pass the threshold
 ZGV drivers:     39 out of   1092 tests failed to pass the threshold

The main question - are failing tests caused by FlexiBLAS or by the optimization flags?

Update 1

From @zao :
Stripping -ftree-vectorize from the build flags that buildenv sets (leaving -O2 -march=native) makes it behave, so it's probably the better vectorizer in GCC11 lifting up some latent problem in OpenBLAS. It wouldn't be the first time...

STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 CEV:    4 out of  1096 tests failed to pass the threshold
 CVX:   24 out of  5196 tests failed to pass the threshold
 CGV drivers:      5 out of   1092 tests failed to pass the threshold
 CGV drivers:      5 out of   1092 tests failed to pass the threshold
 ZEV:    8 out of  1100 tests failed to pass the threshold
 ZVX:   36 out of  5208 tests failed to pass the threshold
 ZGV drivers:     26 out of   1092 tests failed to pass the threshold
 ZGV drivers:     26 out of   1092 tests failed to pass the threshold

Update 2

From @zao
I've set up a fresh environment on a Haswell machine, got the same grade of broken outcome as on our zen2 so not µarch-dependent. Steps:

Make and install a buildenv-default-GCC-11.3.0.eb
$ ml GCC/11.3.0 OpenBLAS/0.3.20 CMake/3.23.1
$ ml buildenv  # defines all the various flags variables to "-O2 -ftree-vectorize -march=native"
$ tar xf v3.10.1.tar.gz  # extract LAPACK sources
$ cmake -B build-tests lapack-3.10.1/ -DUSE_OPTIMIZED_BLAS=ON -DBUILD_TESTING=ON -DBLAS_LIBRARIES=$EBROOTOPENBLAS/lib/libopenblas.so
$ cmake --build build-tests -j 4 && cmake --build build-tests -t test
$ (cd lapack-3.10.1; ./lapack_testing.py; grep failed TESTING/testing_results.txt)

Update 3

From @zao

Ran some exhaustive tests on zen2 from GCC 9.5.0 through GCC 12.2.0 with OpenBLAS 0.3.20. It's not looking great.
I'll try to provide data later but it seems that starting with GCC 12 we get elevated test error rates even without -ftree-vectorize , but builds with the flag have fewer categories of test errors comparatively than GCC 11 does.
Interesting enough, even on the 9.5 and 10 series there's slightly different error counts if you look at with/without the flag. I don't know enough about this test suite to tell whether any errors at all is a problem.

Update 4

I got the following number of numerical errors using lapack_testing.py -p x -t eig from the LAPACK distribution:

Build with GCC-11.3:

• -O2 -march=znver2 -funroll-all-loops -fno-math-errno -ftree-vectorize : 4090
• -O2 -march=znver2 -funroll-all-loops -fno-math-errno : 136
• -O2 -march=znver2 -fno-math-errno : 136
• -O2 -fno-math-errno : 7

Build with GCC-10.3:

• -O2 -march=znver2 -funroll-all-loops -fno-math-errno -ftree-vectorize : 136

every OpebBLAS version was built manually using GCC/11.3.0 or GCC/10.3.0 module (no FlexiBLAS involved)

Way to reproduce:

$ wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.1.tar.gz
$ tar -xf v3.10.1.tar.gz
$ cd lapack-3.10.1
$ cp make.inc.example make.inc
$
$ # Modify make.inc by removing paths to BLASLIB, CBLASLIB, TMGLIB and LAPACKELIB
$ # Change LAPACKLIB to, e.g. $(EBROOTOPENBLAS)/lib/libopenblas.so
$
$ cd TESTING
$ make 
$ cd .. 
$ lapack_testing.py -p x -t eig

[boegel]

@akesandgren Any thoughts on this?

[boegel]

@maxim-masterov Did you check whether building OpenBLAS/0.3.20-GCC-11.3.0 without -ftree-vectorize fixes the problems you are seeing with VASP?

[boegel]

W.r.t. things looking worse with GCC 12, that's probably because the auto-vectorizer is enabled by default there, see also https://www.phoronix.com/news/GCC-12-Auto-Vec-O2 (hat tip @zao)

[martin-frbg]

Maybe related to https://github.com/xianyi/OpenBLAS/pull/3745/files resp. the crashes on OSX that disabling tree-vectorize fixes

[martin-frbg]

Seems the added -march=znver2 is the main source of numerical errors at least with the current develop branch and gcc-12.1. (Not too keen on testing/patching outdated OpenBLAS releases). This would suggests the deviations result from gcc itself choosing particular instructions/sequences when compiling plain C or Fortran code.
NB with the LAPACK testsuite it is important to read the testing_results.txt to see the magnitude of the errors reported as most thresholds are low

[martin-frbg]

Update:errors look significant (1e6 and above) but majority arise already from compiling just the netlib-derived LAPACK part with gfortran -march=znver2

[akesandgren]

One quick note on LAPACK testing. It is imperative to compile things under TESTING and MATGEN with -O0.
Otherwise compilers are likely to introduce errors where there are none... and to compare with a -O0 compiled libblas.
If that still returns errors, then you have a compiler (or possibly LAPACK code) problem.

[martin-frbg]

also things like Reference-LAPACK/lapack#679 where tests expects exact same result as from their own non-optimized BLAS...

[maxim-masterov]

@maxim-masterov Did you check whether building OpenBLAS/0.3.20-GCC-11.3.0 without -ftree-vectorize fixes the problems you are seeing with VASP?

Not with OpenBLAS/0.3.20-GCC-11.3.0. A colleague of mine built a newer version - OpenBLAS/0.3.21-GCC-11.3.0 without the -ftree-vectorize flag and used it to built VASP6/6.3.2-foss-2022a. He said that this version gave plausible results in VASP.

[martin-frbg]

OpenBLAS 0.3.21 (vs 0.3.20) updated the copy of Reference-LAPACK to 3.10.1 plus fixes, which may have fixed your use case of GGEV through changes therein. I am not immediately aware of any other change that would have affected thread safety or resilience against more aggressive optimisation, but unfortunately I am not equipped to test either VASP or EPYC (although I am a computational chemist by training - now self-employed in an unrelated sector)

[maxim-masterov]

Some more results. To avoid questions like "why we didn't use the internal LAPACK tests available with OpenBLAS releases and used LAPACK tests taken directly from netlib". I downloaded OpenBLAS-0.3.20 and compiled it with GCC/11.3.0 on the AMD EPYC ROME (zen2) machine. Then I built netlib's LAPACK tests available from the OpenBLAS release.

To change optimization flags I modified the Makefile.rule file in the root folder of untared OpenBLAS. The variables that I used were: COMMON_OPT, FCOMMON_OPT, NO_AVX, NO_AVX2, and NO_AVX512. Playing around with this set of variables allowed me to change the compiler flags used to build both OpenBLAS and LAPACK tests.

Every OpenBLAS version and LAPACK test were built from scratch in a new folder after untaring the OpenBLAS tarball (so, no make clean).

Steps to reproduce:

$ module purge
$ module load 2022 GCC/11.3.0
$ wget https://github.com/xianyi/OpenBLAS/archive/refs/tags/v0.3.20.tar.gz
$ tar -xf v0.3.20.tar.gz
$ cd OpenBLAS-0.3.20
$ vim Makefile.rule
   # modify optimization flags
$ make -j 32
...
$ make PREFIX=${PWD}/install install
...
$ cd lapack-netlib/TESTING
$ make -j 32
$ cd .. && python3 ./lapack_testing.py -t eig -p x

Results

The lists of flags from below are taken from the log, therefore there are some repetitions, e.g. two -O0 flags in one line.
All tests were performed using the lapack_testing.py script available in the LAPACK distribution form netlib. I tested only eigensolvers, since they caused the inaccuracy problem originally discovered in VASP (as indicated in the first comment).

1

Flags: gfortran -O0 -O0 -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	889893		0	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	889893		0	(0.000%)	0	(0.000%)	
COMPLEX          	336649		0	(0.000%)	0	(0.000%)	
COMPLEX16         	336649		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	2453084		0	(0.000%)	0	(0.000%)

2

Flags: -O2 -O2 -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	889893		0	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	889893		0	(0.000%)	0	(0.000%)	
COMPLEX          	336649		0	(0.000%)	0	(0.000%)	
COMPLEX16         	336649		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	2453084		0	(0.000%)	0	(0.000%)

3

Flags: -O2 -O2 -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1 -mavx -mavx2 -mavx2

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	886365		3	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	889893		0	(0.000%)	0	(0.000%)	
COMPLEX          	336649		0	(0.000%)	0	(0.000%)	
COMPLEX16         	336649		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	2449556		3	(0.000%)	0	(0.000%)

4

Flags: -O2 -funroll-all-loops -fno-math-errno -O2 -funroll-all-loops -fno-math-errno -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1 -mavx -mavx2 -mavx2

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	886365		3	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	889893		0	(0.000%)	0	(0.000%)	
COMPLEX          	336649		0	(0.000%)	0	(0.000%)	
COMPLEX16         	336649		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	2449556		3	(0.000%)	0	(0.000%)

5

Flags: -O2 -funroll-all-loops -fno-math-errno -ftree-vectorize -O2 -funroll-all-loops -fno-math-errno -ftree-vectorize -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1 -mavx -mavx2 -mavx2

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	848925		1187	(0.140%)	0	(0.000%)	
DOUBLE PRECISION	875853		1153	(0.132%)	0	(0.000%)	
COMPLEX          	322609		985	(0.305%)	0	(0.000%)	
COMPLEX16         	336649		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	2384036		3325	(0.139%)	0	(0.000%)

6

Flags: -O2 -funroll-all-loops -fno-math-errno -march=znver2 -O2 -funroll-all-loops -fno-math-errno -march=znver2 -Wall -frecursive -fno-optimize-sibling-calls -m64 -fPIC -msse3 -mssse3 -msse4.1 -mavx -mavx2 -mavx2

Output:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	889893		0	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	889893		0	(0.000%)	0	(0.000%)	
COMPLEX          	328369		38	(0.012%)	0	(0.000%)	
COMPLEX16         	328369		94	(0.029%)	0	(0.000%)	

--> ALL PRECISIONS	2436524		132	(0.005%)	0	(0.000%)

To me, these results show that the more aggressive implicit vectorisation we use, the more LAPACK tests fail with OpenBLAS-0.3.20 and GCC-11.3.0.

Also, I think that the test # 1 should also indicate that there are no problems with a compiler, since it uses -O0 to compile both OpenBLAS and LAPACK tests. Do I understand it correctly, @akesandgren?

[boegel]

@maxim-masterov Can you check if you seeing the same problems for OpenBLAS-0.3.20-GCC-11.2.0.eb too?

I plan to open a PR for the relevant OpenBLAS easyconfigs to disable the use of -ftree-vectorize where it's required, so we can include those updated easyconfigs in the upcoming EasyBuild release (v4.6.2), that's the best we can do short term I think...

Longer term, we should enhance the OpenBLAS easyblock to more carefully check the result of the tests being run (and perhaps also expand the set of tests being run).