Different result for -hdr_6x6i_level 12 on Windows vs. Linux and macOS for extreme input

[MarkCallow]

The result from the following command is different on Windows compared to that on macOS and Linux.

bin/basisu -hdr_6x6i -hdr_6x6i_level 12 -output_file R16G16B16_SFLOAT_2D_8x8_LEVEL_12.ktx2 R16G16B16_SFLOAT_2D_8x8.exr

This is the source file: R16G16B16_SFLOAT_2D_8x8.exr.zip

For levels 1 to 11, the result is identical on all 3 OSes. I used GCC 11 on Linux, Clang 17 on macOS and CLangCL 19.1.5 on Windows. The compiler does not seem to matter though. The same result happens with ClangCL or MSVC (19.44.35222).

The largest pixel difference compared to the Linux/macOS result is 612.0 (compared after transcoding to HALF). As a percentage of the smaller of the 2 pixel values that is only 2.2% not nothing major. The input image is a stress tester from our test suite, so an edge case. With a, perhaps, more normal input, Desk.exr that you have been using, both level 11 and level 12 show similar to each other, and much smaller compared to the 8x8 case, differences from the macOS generated images. I haven't tried to compare the Linux generated output in this case.

[richgel999]

If you run basisu with -stats (can also add -debug) you can see our internal PSNR's between the platforms - that's the first thing I would look at. How close are they?

Unless you can use exactly the same compiler, compiler version, and compiler settings on all these platforms, it's very unlikely you can get the output to exactly match because the floating point expressions (which are heavily utilized throughout our encoders) will be computed differently. We could try playing with our float compiler options (like ARM's does with astcenc), but this just slows down execution for no actual guarantee of increased determinism.

This is why we've adopted WASM (web-based and WASI) here for this sort of testing. wasmtime runs .WASM executables virtually everywhere, and it has a truly deterministic and well-specified floating point execution model. We include precompiled .WASM executables in the bin directory, which we keep up to date.

Example between Windows 11 and Ubuntu Linux:

Linux:

richg@ryzen9:/mnt/c/dev/bu_3_20/basis_universal/bin$ ./runwt.sh  -stats -debug -hdr_6x6i -hdr_6x6i_level 12 Desk.exr
Basis Universal LDR/HDR GPU Texture Supercompression System v2.10.0 (WASI Threaded)
...
----------------------------------------------------------------------------
basis_compressor::create_basis_file_and_transcode
Total .basis output file size: 231127, 3.285 bits/texel
basist::basisu_transcoder_init: Took 0.001000 ms
basisu_compressor::start_transcoding() took 0.000ms
Transcoded to BC6H in 32.049ms, 17562357.684205 texels/sec
Alternate transcode in 5.082ms, 110754821.216268 texels/sec
Total .ktx2 output file size: 231247, 3.287 bits/texel
basis_compressor::write_output_files_and_compute_stats
Wrote compressed output file "Desk.ktx2"
Output file size: 231247, 3.29 bits/texel, LZ compressed file size: 230811, 3.28 bits/texel
ASTC/BC6H half float space error metrics (a piecewise linear approximation of log2 error):
Slice: 0

ASTC channels:
R:   Max: 11075.000000 Mean: 106.490631 RMS: 210.584198 PSNR: 49.860950 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
G:   Max: 12128.000000 Mean: 141.488693 RMS: 343.309937 PSNR: 45.615738 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
B:   Max: 11932.000000 Mean: 158.446014 RMS: 329.478516 PSNR: 45.972923 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H channels:
R:   Max: 11565.000000 Mean: 113.709724 RMS: 220.285507 PSNR: 49.469749 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
G:   Max: 12019.000000 Mean: 157.718185 RMS: 369.703461 PSNR: 44.972397 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
B:   Max: 12239.000000 Mean: 173.304062 RMS: 355.714294 PSNR: 45.307438 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

ASTC RGB: Max: 12128.000000 Mean: 135.475113 RMS: 300.423859 PSNR: 46.774776 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H RGB: Max: 12239.000000 Mean: 148.243988 RMS: 322.355408 PSNR: 46.162766 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

ASTC Log2 RGB: Max: 115.250000 Mean: 0.049326 RMS: 0.129078 PSNR: 17.782976 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H Log2 RGB: Max: 116.375000 Mean: 0.053109 RMS: 0.138426 PSNR: 17.175619 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

Compression succeeded to file "Desk.ktx2" size 231247 bytes in 61.413 secs, 3.287 bits/texel
Total successes: 1 failures: 0
job_pool::~job_pool

Windows 11:

C:\dev\bu_3_20\basis_universal\bin>runwt.bat -stats -debug -hdr_6x6i -hdr_6x6i_level 12 Desk.exr -stats
Basis Universal LDR/HDR GPU Texture Supercompression System v2.10.0 (WASI Threaded)
...
----------------------------------------------------------------------------
basis_compressor::create_basis_file_and_transcode
Total .basis output file size: 231127, 3.285 bits/texel
basist::basisu_transcoder_init: Took 0.002000 ms
basisu_compressor::start_transcoding() took 0.000ms
Transcoded to BC6H in 31.743ms, 17731657.418047 texels/sec
Alternate transcode in 4.757ms, 118321631.578952 texels/sec
Total .ktx2 output file size: 231247, 3.287 bits/texel
basis_compressor::write_output_files_and_compute_stats
Wrote compressed output file "desk.ktx2"
Output file size: 231247, 3.29 bits/texel, LZ compressed file size: 230811, 3.28 bits/texel
ASTC/BC6H half float space error metrics (a piecewise linear approximation of log2 error):
Slice: 0

ASTC channels:
R:   Max: 11075.000000 Mean: 106.490631 RMS: 210.584198 PSNR: 49.860950 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
G:   Max: 12128.000000 Mean: 141.488693 RMS: 343.309937 PSNR: 45.615738 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
B:   Max: 11932.000000 Mean: 158.446014 RMS: 329.478516 PSNR: 45.972923 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H channels:
R:   Max: 11565.000000 Mean: 113.709724 RMS: 220.285507 PSNR: 49.469749 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
G:   Max: 12019.000000 Mean: 157.718185 RMS: 369.703461 PSNR: 44.972397 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
B:   Max: 12239.000000 Mean: 173.304062 RMS: 355.714294 PSNR: 45.307438 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

ASTC RGB: Max: 12128.000000 Mean: 135.475113 RMS: 300.423859 PSNR: 46.774776 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H RGB: Max: 12239.000000 Mean: 148.243988 RMS: 322.355408 PSNR: 46.162766 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

ASTC Log2 RGB: Max: 115.250000 Mean: 0.049326 RMS: 0.129078 PSNR: 17.782976 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false
BC6H Log2 RGB: Max: 116.375000 Mean: 0.053109 RMS: 0.138426 PSNR: 17.175619 dB, Any Neg: false, Half float overflow: false, Any NaN/Inf: false

Compression succeeded to file "desk.ktx2" size 231247 bytes in 74.593 secs, 3.287 bits/texel
Total successes: 1 failures: 0
job_pool::~job_pool

The output files are the same between Linux and Windows:

C:\dev\bu_3_20\basis_universal\bin>fc /b win.ktx2 linux.ktx2
Comparing files win.ktx2 and LINUX.KTX2
FC: no differences encountered

I'll run it on Mac when I get a chance, but the output should match.

We use wasmtime like this:

#!/usr/bin/env bash
wasmtime run --dir=. --dir=../test_files --wasm threads=yes --wasi threads=yes ./basisu_mt.wasm "$@"

wasmtime has optimized JIT code generation backends for x64 and ARM. We'll be adding support for WASM SIMD soon. It's our primary lowest common denominator platform at this point.

[MarkCallow]

Thank you for the information, especially about -stats. I failed to find that when reading the help, probably because I was too focused on the word "compare". I will experiment with that and wasm.

The reason I felt this worth bringing to your attention is that, at least for R16G16B16_SFLOAT_2D_8x8.exr, the differences happen only on Windows, regardless of the compiler used, and only for level 12. (I wonder if there may be an ifdef _WIN32 or similar in the level 12 code.) Only after I had initially opened the issue did I try with Desk.exr and found in that case that differences on Windows exist also for level 11.

I was experimenting with setting the tolerance for ktxdiff to a percentage of the minimum value of the two texels/components being compared as I seek an easy to understand way for users to specify the allowable tolerance. In doing so I observed that some texels with very small values (e.e. 0.009) have much larger percentage differences than some texels with very large values (e.g. 27000). This is for the texture generated from R16G16B16_SFLOAT_2D_8x8.exr with level 12 specified. Is it expected that there could be a wide variance in the percentage error?