[KHR_gaussian_splatting] Consider store scale not in log-space

[NorbertNopper-Huawei]

glTF/extensions/2.0/Khronos/KHR_gaussian_splatting/README.md

Line 331 in f966a3f

`KHR_gaussian_splatting:SCALE` is stored in log-space, so the actual scale along each principal axis is computed as `exp(scale)`. This allows for representing a wide range of scales while maintaining numerical stability.

After looking at some implementations and also doing an implementation from scratch, I think we should change KHR_gaussian_splatting:SCALE to be stored not in log-space and just linear. Otherwise, one has to make the conversion during load time or in the shader using an exp call.

The described numerical stability is mainly for training, however for rendering the value needs to be changed.

@weegeekps @javagl @lexaknyazev

[weegeekps]

I don't think we should make this change. While the numerical stability is indeed most useful for training, current shader implementations using the base paper almost universally expect the scale to be stored in log-space. Changing scale to be stored linearly would be a major change outside of what has been largely the standard within the 3DGS community since day one.

[NorbertNopper-Huawei]

It is a glTF anti-pattern, as nowhere in glTF we are doing this. If we keep log-space, one has to process the whole buffer and convert before uploading to the GPU. Or, one has to do it in the shader, which no-one is doing.

[NorbertNopper-Huawei]

For opacity, we are also not storing e.g. the value what is stored inside a PLY file. Anyway, we can keep it like this as the math is clear, however we just add a not required conversion for glTF.

[NorbertNopper-Huawei]

@javagl Where are you converting the scale?

[javagl]

I'm a bit on the fence with this one. There had been some discussion in the PR ( #2490 (comment), #2490 (comment) ), and it seemed to be resolved. The most controversial cases had been that of the SCALE (whether it is log-scale or linear), and OPACITY (whether it was in [-Inf,Inf] or [0,1]).

For the SCALE case, it seemed to be agreed upon that it is stored in log-space. (I shrugged that off, even though I'm also doing that exp and don't really know why).

Given that glTF was always aiming at delivery, and at storing data in a form that can directly be sent to a renderer (without postprocessing), one could make a strong case to change the spec, and not store the data in log-space. Whatever's in the buffer: It should be ready to go to the GPU.

I have some concerns, though: Changing it now could cause a few hiccups, but that's not a strong argument: Better now than never! When it's ratified, it can never be changed (and we could only hope that it wouldn't haunt us forever). Another concern is what people might expect: When people see SCALE, then they might expect the classical "splat-scale" which is in log space. At this point, one could even consider to call it LINEAR_SCALE to be completely explicit.

(Similarly, for the case of the OPACITY, I still think that people who see OPACITY in the context of splats might expect [-Inf,Inf], and we could/should consider calling it ALPHA to emphasize that it is in [0,1]...)

[javagl]

There was a race condition in our comments. I saw your one about the "opacity", just when I was about to send. The new one popped up a few seconds after sending 🙂

[NorbertNopper-Huawei]

For training, one is using the log-space, because of numerical stability and the nature of training data. However for rendering the scale is in used in linear space.
With opacity, we are storing the value as used for the rendering and not the one used for training. So, somehow inconsistent.
When computing the covariance matrix, normally this conversion to linear space of scale happens there. Some are doing this during load time, some are doing this during runtime. And if folks are doing this during runtime, the scale is already in linear space.

The problem I see is, that if folks a doing this during runtime, implementors have two paths for handling scale.

[NorbertNopper-Huawei]

Collecting some examples here, as an argument that SCALE should be stored in linear space:

Original paper implementation is using the linear values when calculating the covariance matrix.
https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/forward.cu#L122

Converts PLY during load time to linear and not later in the pipeline:
https://github.com/antimatter15/splat/blob/main/main.js#L532

Renderer calculating covariance matrix in shader, where data is already in linear space:
https://github.com/PV-NTR/3DGS-Vulkan/blob/main/renderer/resources/shaders/gsplat.vert#L126

Interoperability with OpenUSD, as they store linear internally as well:
https://github.com/PixarAnimationStudios/OpenUSD/blob/dev/extras/imaging/examples/hdParticleField/py3dgsPlyToUsd.py#L342

3D Gaussian splatting for Three.js uses internally linear and converts SPZ to linear:
https://github.com/mkkellogg/GaussianSplats3D/blob/main/src/loaders/spz/SpzLoader.js#L209

BabylonJS uses internally linear and converts SPZ to linear:
https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/loaders/src/SPLAT/spz.ts#L84

CesiumJS uses internally linear and converts SPZ to linear:
https://github.com/drumath2237/spz-loader/blob/169f943d0d8b25a2ee816c26ab0a5f96ecca115e/packages/core/lib/spz-wasm/gaussianCloud.ts#L35

---

Counter examples:

Using log-scale in the shader
https://github.com/cvlab-epfl/gaussian-splatting-web/blob/main/src/shaders.ts#L187

Intermediate format for conversion uses log space:
https://github.com/3dgscloud/GaussForge/blob/main/README.md?plain=1#L181

[NorbertNopper-Huawei]

@weegeekps You have some links to code, where log-space is used during runtime?

[javagl]

I don't want to delve into philosophical considerations of what "runtime" means. But eventually, the covariance matrix is computed with linear values. It's only about the question where the "source data" is converted from log space to linear (assuming that during that black-box training process, the data is essentially always in log space). It could happen

1. When exporting the data
2. When reading the data
   • a. In the loader
   • b. In the shader

Doing this in the shader is essentially free in terms of performance. Reading a float is what is expensive. The exp does not matter. (Yeah, people tend to disagree here, but they never show a benchmark). Doing it in the loader could cause an overhead, and (more importantly) imply the preprocessing that glTF is usually trying to avoid. But the question could be avoided by already exporting the linear values, which minimizes the client-side processing in any case. (I'll let others chime in, but iff it is changed, I'd reiterate that the attribute should be called LINEAR_SCALE to avoid confusion).

[NorbertNopper-Huawei]

It should be vice versa, as e.g. here the implementor says this is log scale: https://github.com/cvlab-epfl/gaussian-splatting-web/blob/main/src/shaders.ts#L187
This is the only example I found, where the covariance calculation takes log scale.
The other links above including BabylonJS and ThreeJS, expect after load of the splat data, that the scale - as intended for a renderer - is linear. Also the original implementation!

My concern is not mainly about the performance. It is more about the acceptance for the adaptors. In the shader code, the scale (or already encoded in the covariance matrix) is in linear space (except this one example). So, either you have to convince the adaptor, hey, please use for glTF non-linear in your shader - which makes no sense or - hey, adapt your gTF loader the first time and convert the data from non-linear to linear, considering it could be not just float but also half-float or quantized data - which also makes no sense.

So, how you say it, this should happen during "When exporting the data" like we always did for glTF.

[NorbertNopper-Huawei]

In the implementation I am working (sharing at some point), I am using a storage buffer for the splat data.
In the best case, if the layout inside the glTF aligns to the data layout for the renderer, is it a one time upload without any conversion.
In the worst case, I need to adapt the layout, by doing some memcpy.
And then, I can use the shaders commonly found out there.

[NorbertNopper-Huawei]

And this does not conflict with upcoming 3DGS compressions like SPZ. After(!) the data is decoded form SPZ, as e.g. seen in the links for Three.JS and BabylonJS, adaptors and implementors are using the linear data. How one is compressing the 3DGS data in the future, at some point in the pipeline the data has to become linear for scale.

[weegeekps]

Yes, SPZ is log-encoded. In the case of CesiumJS, we use spz-loader to load the SPZ and it does the math for us as the data is decoded.

This decision by SPZ is intentional as the scaling factors in log-space allows for more stable quantization. We discovered this ourselves when experimenting with Meshopt compression prior to adopting SPZ. Keeping the scales in log-space completely eliminates an entire class of quantization issues around the wide dynamic range of Gaussians in larger scenes.

Additionally, and perhaps more importantly, there's a genuine risk of losing precision for small Gaussians. This was actually the thing that keyed us into something being wrong when we experimented with converting to linear space at rest ourselves. Gaussians with tiny log-scales can easily lose detail due to floating-point underflow and will make the scene blurry.

I understand your point about this being outside of the norm for glTF and I also understand that everyone converts this to linear space before rendering, typically during load. Despite that and given the issues with storing in linear space I've outlined above, I strongly feel storing scales in linear space would be a major mistake. Especially so since we purposely chose to allow scale data in signed byte and signed short form to allow quantization.

(Similarly, for the case of the OPACITY, I still think that people who see OPACITY in the context of splats might expect [-Inf,Inf], and we could/should consider calling it ALPHA to emphasize that it is in [0,1]...)

We chose OPACITY because that's how it's referred to in the 3DGS world. To be fair OPACITY is probably more accurate, as in many 3D Graphics textbooks typically refers to the uniform transparency in entire object whereas ALPHA often refers to a per-pixel value in a texture map. I don't really feel strongly one way or the other, but OPACITY just makes more sense to me.

[NorbertNopper-Huawei]

What is required for compression or for training data is not relevant for this extension.

In the second link you provided, CesiumJS is converting to a linear float value. And then also in this rendering pipeline the values are treated in linear space.

So, for glTF this value can be stored, as with 32-bit float nothing is lost when stored and optimal for rendering. Also, no shaders need to be changed in all major implementations.

Because of the current approach, one has to loop through the glTF content and apply exp to it. For all the other data fields this is not required and the buffers can be utilized like any other buffer in glTF on the GPU.