Precision issues in FP8 GEMM: Minor inaccuracies exceeding 1e-5 threshold

[Wonicon]

Commits

XSAI: OpenXiangShan/XSAI@1a71bcaf7
CUTE: 19f9b37

Problem

A few elements (0–4 out of 16,384) in the $128 \times 128$ GEMM output exhibit precision errors exceeding the $10^{-5}$ tolerance. While these deviations are present, the magnitude of the error remains relatively small.

Example log:

Verifying results against golden reference...
  [ERR] C[61,121]:
    Result:
      Value: 4.340625000e+01 (43.406250)
      Hex: 0x422DA000
      IEEE754: sign=0, exp=132 (0x84), mant=0x2DA000
      Type: Normal (value ~= 4.340625000e+01)
    Golden:
      Value: 4.341151428e+01 (43.411514)
      Hex: 0x422DA564
      IEEE754: sign=0, exp=132 (0x84), mant=0x2DA564
      Type: Normal (value ~= 4.341151428e+01)
    Diff: 0.005264 (5.264282226e-03)

Detailed logs and test data

fp8_minor_errors_2026-04-03.zip