How to get the information of which columns have been quantified?

[huangdiudiu]

my code:

torch.manual_seed(42)

1. Creat a real-valued Linear layer

linear_fp = torch.nn.Linear(8, 4).cuda().half()

2. modify the weight

add some noise

with torch.no_grad():
linear_fp.weight[:, 1] += 0.2 * torch.randn(4).cuda().half()
linear_fp.weight[:, 4] += 0.2 * torch.randn(4).cuda().half()

print("Modified weight:", linear_fp.weight)

3. quantization：set threshold to be small

linear_int8 = bnb.nn.Linear8bitLt(
8, 4,
has_fp16_weights=Flase,
threshold=0.1
)

linear_int8.load_state_dict(linear_fp.state_dict())
linear_int8=linear_int8.to(0)

4. Forward

x = torch.randn(2, 8).cuda().half()
with torch.no_grad():
_ = linear_int8(x)

5. watch state.idx

print("State.idx:", linear_int8.state.idx.cpu().numpy())

But it always return [0,1,2,3,4,5,6,7], I think some columns with value > 0.1 should not be quantized. Did I misunderstand something?

[TimDettmers]

Closing this issue. The threshold parameter in Linear8bitLt controls outlier detection on activations during the forward pass, not on weight values. It works per-token: for each input token, columns where the activation magnitude exceeds the threshold are computed in fp16, while the rest use int8. The state.idx reflects which columns had outlier activations in the last forward pass.

With threshold=0.1, most or all activation values will exceed it, so all columns get flagged as outliers (hence idx = [0,1,2,3,4,5,6,7] meaning all columns stayed in fp16).

If you want to identify weight columns that are statistical outliers, you can detect them via a z-test on the per-column standard deviations of the weight matrix, as described in follow-up work on LLM.int8() (e.g., "FP8-LM: Training FP8 Large Language Models" and "Cubic Infant Scaling Laws").

[TimDettmers]

The threshold parameter behavior is now documented in commit d7c179c. The Linear8bitLt.__init__ docstring now explicitly states that the threshold operates on activations (inputs), not on weight values, and explains the mixed-precision decomposition mechanism.