How to create NF4 datatype

[phorcic]

The function

bitsandbytes/bitsandbytes/functional.py

Line 209 in 00c0a1a

def create_normal_map(offset=0.9677083, use_extra_value=True):

is supposed to return the NF4 datatype as specified in

bitsandbytes/bitsandbytes/functional.py

Line 733 in 00c0a1a

data = [

.

However, this is not the case as the result is not exactly the same; we are missing some decimals after the comma because of rounding. Moreover, I am not fully sure how to reproduce the exact values that are used in the NF4 paper https://arxiv.org/abs/2305.14314.

I am mostly curious to know how I can obtain the exact values that are used in the paper and in the NF4 datatype. Also, perhaps the documentation could possibly be updated. Thank you so much for clarifying.

[TimDettmers]

Closing this issue. The NF4 values hardcoded in get_4bit_type("nf4") were generated by create_normal_map(offset=0.9677083, use_extra_value=True) and then normalized to [-1, 1]. The slight differences you're seeing vs. calling create_normal_map directly are due to floating-point rounding — the hardcoded values are the canonical ones used at runtime (scipy is not required at runtime, only for regenerating the values).

For the paper's exact values: the NF4 data type is constructed by computing scipy.stats.norm.ppf() (inverse CDF of the standard normal) at evenly spaced quantiles, then normalizing. The offset parameter controls the outermost quantile boundary. Small differences from the paper's table are expected due to floating-point precision.

We are improving the documentation for these functions.

[TimDettmers]

The NF4 construction process is now documented in commit d7c179c. The docstring on create_normal_map explains how the values are derived and why the hardcoded values may differ slightly from calling the function directly (floating-point rounding). The inline comment in get_4bit_type now notes that values are hardcoded to avoid a scipy dependency at runtime.