[Feature] KTransformers Integration to Support CPU/GPU Hybrid Inference for MoE Models

[Atream]

Motivation

While hybrid CPU/GPU inference alleviates memory constraints by leveraging CPU DRAM alongside GPU VRAM bandwidth, achieving high throughput remains challenging due to synchronization overheads and limited CPU compute efficiency. This PR upstreams the KTransformers approach (SOSP ’25), enabling GPU Tensor Parallelism + CPU/GPU Hybrid Expert Parallelism for MoE models—supporting hybrid prefilling and decoding that utilize kt-kernel AMX-optimized CPUs together with GPUs. With this design, dense layers benefit from high-throughput multi-GPU execution, while experts are flexibly scheduled across both CPUs and GPUs, maximizing hardware utilization and reducing bottlenecks.
KTransformers will be incorporated into SGLang as a library backend. Building on this backend, sglang generalizes the design to support multi-GPU tensor parallelism and CPU/GPU hybrid expert parallelism, while broadening coverage to additional models and weight formats.

Benchmark Results (Preview)

These are preliminary results of KTransformers with a single GPU, reflecting the performance of this feature in a single-card setting. More detailed benchmark data will be provided in follow-up updates. The figures below show the throughput performance of KTransformers on a dual-socket server with Intel® Xeon® Platinum 8452Y CPUs (36 cores × 2, 1 TB DDR5), equipped with an NVIDIA A100 (40 GB) for full-precision models and an NVIDIA RTX 4080 (16 GB) for quantized models. We evaluate on DeepSeek-V3-0324 (DS-3), DeepSeek-V2.5-1210 (DS-2), and Qwen2-57B-A14B (QW-2), comparing KTransformers against Llama.cpp and Fiddler across both the prefill and decode phases.

In the prefill phase, KTransformers consistently outperforms both baselines across all prompt lengths. While Llama.cpp shows advantages in short-prompt scenarios through aggressive operator fusion, and Fiddler benefits from AMX acceleration for long prompts, KTransformers surpasses both by leveraging AMX-optimized CPU kernels and improved CPU/GPU coordination. For example, our CPU MoE kernel achieves 21.3 TFLOPS on DS-3, a 3.98× improvement over the PyTorch baseline.

In the decode phase, KTransformers (without Expert Deferral) achieves 2.42×–4.09× speedups over Fiddler and 1.25×–1.76× over Llama.cpp on full-precision models. With quantized models, the gains are even larger (1.77×–1.93× vs. Llama.cpp), primarily due to reduced kernel execution time and our efficient CUDA Graph-based scheduling, which reduces GPU launch overhead from over 20% to nearly zero.

Roadmap

1. Hybrid inference with compressed tensor format + AMX kernel integration + CUDA Graph support. init support for KTransformers Heterogeneous Computing #11487
2. Support hybrid quant config.
3. Support more weight formats (eg. GPTQ, AWQ).
4. refactor to use experts_map instead of num_gpu_experts.
5. Avoid padding when using DP Attention.
6. Hotness aware expert distribution.
7. Experts deferral. Support Expert Deferral Mechanism in KTransformers #12586
8. Add unit tests.
9. Add tutorial and deployment guide.
10. Supporting speculative decoding.
11. Support more models (eg. Qwen3, GLM4.5, Kimi-K2)

Related resources

Repo：https://github.com/kvcache-ai/ktransformers
SOSP25 Paper：https://madsys.cs.tsinghua.edu.cn/publication/ktransformers-unleashing-the-full-potential-of-cpu/gpu-hybrid-inference-for-moe-models/

CC: @Atream @ovowei @chenht2022 @Azure-Tang @ErvinXie

[aikitoria]

Why is this integration AMX exclusive? The original KTransformers library was not.

[james0zan]

Other platform support will also be upstreamed

[DocShotgun]

Hello, and thank you for this! Finally an inference backend that can fully utilize my dual Xeon + single GPU setup.

I was able to download the Deepseek-v3-0324 quants from modelscope and deploy them locally, but I was wondering if you would be able to provide additional guidance on how to correctly convert model weights to use with this backend?

@intervitens managed to create quants identical to the ones provided on modelscope for GLM-4.5-Air by using llm-compressor with the included yaml recipe for the GPU portion, and the provided script for the CPU portion (which does have a typo where "subpool" should be "threadpool").

However we've been hitting some roadblocks with Deepseek. I modified the script to save safetensors files per layer rather than accumulating them until the end (or else Deepseek conversion requires more than 768gb RAM), and managed to convert the original fp8 Deepseek-V3-0324 to an INT4 quant that will load and infer without error, however produces no valid output - just empty string - possibly an overflow during inference? The resulting layer tensors are also not hash-identical like the ones for GLM-4.5-Air.

Update: It seems that something is broken with the conversion script's fp8 input support. Dequanting Deepseek to bf16 first seems to work.

Update 2: Uploaded some compatible weights to HF for anyone interested to try out.

Update 3: We seem to be having some problems with poor output quality - most noticeably with occasional misspelled words during generation. I initially attributed this to the RTN int4 experts, but it still seems to happen with int8 experts. Intervitens also got an aime24 score of 0.466 using fp8 non-expert params on GPU with int8 experts on CPU, while various Openrouter providers range from 0.5-0.63 for Deepseek V3 0324.

[KMSorSMS]

Hello, and thank you for this! Finally an inference backend that can fully utilize my dual Xeon + single GPU setup.

I was able to download the Deepseek-v3-0324 quants from modelscope and deploy them locally, but I was wondering if you would be able to provide additional guidance on how to correctly convert model weights to use with this backend?

@intervitens managed to create quants identical to the ones provided on modelscope for GLM-4.5-Air by using llm-compressor with the included yaml recipe for the GPU portion, and the provided script for the CPU portion (which does have a typo where "subpool" should be "threadpool").

However we've been hitting some roadblocks with Deepseek. I modified the script to save safetensors files per layer rather than accumulating them until the end (or else Deepseek conversion requires more than 768gb RAM), and managed to convert the original fp8 Deepseek-V3-0324 to an INT4 quant that will load and infer without error, however produces no valid output - just empty string - possibly an overflow during inference? The resulting layer tensors are also not hash-identical like the ones for GLM-4.5-Air.

Update: It seems that something is broken with the conversion script's fp8 input support. Dequanting Deepseek to bf16 first seems to work.

Update 2: Uploaded some compatible weights to HF for anyone interested to try out.

Update 3: We seem to be having some problems with poor output quality - most noticeably with occasional misspelled words during generation. I initially attributed this to the RTN int4 experts, but it still seems to happen with int8 experts. Intervitens also got an aime24 score of 0.466 using fp8 non-expert params on GPU with int8 experts on CPU, while various Openrouter providers range from 0.5-0.63 for Deepseek V3 0324.

@ovowei I guess we haven't supported RTN int4 or int8 converted from fp8, as we need to write the fp8 dequantization, which is missing.
And seems the int8 support has something wrong. We will fix it soon. @Atream

[DocShotgun]

@ovowei I guess we haven't supported RTN int4 or int8 converted from fp8, as we need to write the fp8 dequantization, which is missing. And seems the int8 support has something wrong. We will fix it soon. @Atream

For the record, I also encountered the problem with degraded-appearing outputs and frequent misspelled words using the Deepseek V3 0324 INT4 quants provided on modelscope (in conjunction with the fp8+GPTQ4 GPU weights) - which I initially just attributed to quality loss from RTN channel-wise INT4 quantization - until I encountered the same problems using our own converted INT8 weights.

We did push the weights (and mirrored some of the modelscope weights) to HuggingFace if you'd like to inspect them: https://huggingface.co/CPU-Hybrid-MoE

This is the first backend that can really utilize all of the hardware on my dual Xeon + RTX Pro 6000 setup, so I'm definitely looking forward to future updates.

[KMSorSMS]

For the record, I also encountered the problem with degraded-appearing outputs and frequent misspelled words using the Deepseek V3 0324 INT4 quants provided on modelscope (in conjunction with the fp8+GPTQ4 GPU weights) - which I initially just attributed to quality loss from RTN channel-wise INT4 quantization - until I encountered the same problems using our own converted INT8 weights.

We did push the weights (and mirrored some of the modelscope weights) to HuggingFace if you'd like to inspect them: https://huggingface.co/CPU-Hybrid-MoE

This is the first backend that can really utilize all of the hardware on my dual Xeon + RTX Pro 6000 setup, so I'm definitely looking forward to future updates.

There is a bug that the kt-amx-method choice comes from env directly, which means the command line arg is not working. But since your output is not nonsense words, I guess you have set it correctly by env. We will try to locate if the int8 kernel has something wrong, and also give official weights for you to check soon.

[DocShotgun]

There is a bug that the kt-amx-method choice comes from env directly, which means the command line arg is not working. But since your output is not nonsense words, I guess you have set it correctly by env. We will try to locate if the int8 kernel has something wrong, and also give official weights for you to check soon.

Yes, export AMX_METHOD=AMXINT8 was necessary to get it to convert or infer INT8 CPU weights at all (or else it produces absolute gibberish). The outputs I was getting with either INT4 or INT8 were coherent text for the most part, but with significant degradation in quality including misspelled words even with truncation of min-p 0.1 and top-p 0.9 with a temperature of 1.

Some examples are like:
"raucous peal of lafter"
"mischvous ardor"
"cruitial moment"

I thought perhaps it was a tokenizer issue as the included tokenizer with the GPU weights did not match the size/hash of the original DeepSeek V3 0324, but replacing the tokenizer and reloading the model did not entirely eliminate this problem.

[KMSorSMS]

Yes, export AMX_METHOD=AMXINT8 was necessary to get it to convert or infer INT8 CPU weights at all (or else it produces absolute gibberish). The outputs I was getting with either INT4 or INT8 were coherent text for the most part, but with significant degradation in quality including misspelled words even with truncation of min-p 0.1 and top-p 0.9 with a temperature of 1.

Some examples are like: "raucous peal of lafter" "mischvous ardor" "cruitial moment"

I thought perhaps it was a tokenizer issue as the included tokenizer with the GPU weights did not match the size/hash of the original DeepSeek V3 0324, but replacing the tokenizer and reloading the model did not entirely eliminate this problem.

What's the GPU weights you are using? (fp8+GPTQ4 GPU weights from there: https://modelscope.cn/models/ApproachingAI2024/DeepSeek-V3-0324-GPU-weight/files ?).

And thanks for your discovery and help. We are going to check this and do some tests to figure it out.

[DocShotgun]

What's the GPU weights you are using? (fp8+GPTQ4 GPU weights from there: https://modelscope.cn/models/ApproachingAI2024/DeepSeek-V3-0324-GPU-weight/files ?).

And thanks for your discovery and help. We are going to check this and do some tests to figure it out.

Yes, those, with either the INT4 NUMA2 CPU weights provided on modelscope, or our quanted INT8 NUMA4 CPU weights that are uploaded to HuggingFace.

Also is glm4moe considered a working architecture on this backend? I see that there are weights for GLM 4.5 and GLM 4.5 Air. Intervitens tried to replicate the quant method on GLM 4.6 using this GPTQ recipe. However even with bf16 non-expert GPU weights and INT8 CPU weights, he ran into issues with the output degrading into infinite looping in the thinking phase.

My launch command was:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
SGLANG_ENABLE_JIT_DEEPGEMM=0 \
AMX_METHOD=AMXINT8 \
python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 5000 \
  --model /mnt/data/models/DeepSeek-V3-0324-GPU-FP8-GPTQ4 \
  --kt-amx-weight-path /mnt/data/models/DeepSeek-V3-0324-CPU-NUMA4-AMXINT8 \
  --kt-cpuinfer 128 \
  --kt-threadpool-count 4 \
  --kt-num-gpu-experts 0 \
  --kt-amx-method AMXINT8 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --context-length 131072 \
  --max-total-tokens 131072 \
  --served-model-name DeepSeek-V3-0324 \
  --enable-mixed-chunk \
  --disable-shared-experts-fusion

Note:

• SGLANG_ENABLE_JIT_DEEPGEMM=0 seems to be needed for RTX Pro 6000 Blackwell because otherwise sglang tries to use DeepGEMM which only supports sm_90 and sm_100
• I had to switch to the flashinfer attention backend because the triton attention backend's kernels require too much shared memory for sm_120 and fail to compile (possibly also some assumptions about sm_120 behaving like sm_90 or sm_100 when it's actually more similar to sm_89)
• It's a dual socket Xeon server with 2 SNCs per socket, thus the configuration being 4 NUMA nodes by default

[KMSorSMS]

Yes, those, with either the INT4 NUMA2 CPU weights provided on modelscope, or our quanted INT8 NUMA4 CPU weights that are uploaded to HuggingFace.

Also is glm4moe considered a working architecture on this backend? I see that there are weights for GLM 4.5 and GLM 4.5 Air. Intervitens tried to replicate the quant method on GLM 4.6 using this GPTQ recipe. However even with bf16 non-expert GPU weights and INT8 CPU weights, he ran into issues with the output degrading into infinite looping in the thinking phase.

My launch command was:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
SGLANG_ENABLE_JIT_DEEPGEMM=0 \
AMX_METHOD=AMXINT8 \
python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 5000 \
  --model /mnt/data/models/DeepSeek-V3-0324-GPU-FP8-GPTQ4 \
  --kt-amx-weight-path /mnt/data/models/DeepSeek-V3-0324-CPU-NUMA4-AMXINT8 \
  --kt-cpuinfer 128 \
  --kt-threadpool-count 4 \
  --kt-num-gpu-experts 0 \
  --kt-amx-method AMXINT8 \
  --attention-backend flashinfer \
  --trust-remote-code \
  --mem-fraction-static 0.98 \
  --chunked-prefill-size 4096 \
  --max-running-requests 1 \
  --context-length 131072 \
  --max-total-tokens 131072 \
  --served-model-name DeepSeek-V3-0324 \
  --enable-mixed-chunk \
  --disable-shared-experts-fusion

Note:

• SGLANG_ENABLE_JIT_DEEPGEMM=0 seems to be needed for RTX Pro 6000 Blackwell because otherwise sglang tries to use DeepGEMM which only supports sm_90 and sm_100
• I had to switch to the flashinfer attention backend because the triton attention backend's kernels require too much shared memory for sm_120 and fail to compile (possibly also some assumptions about sm_120 behaving like sm_90 or sm_100 when it's actually more similar to sm_89)
• It's a dual socket Xeon server with 2 SNCs per socket, thus the configuration being 4 NUMA nodes by default

Great. And yes, we haven't supported the glm4.5 on this release, we will add more modules after our features and code are stable.

[jyizheng]

Which ktransformer commit is used to run all above experiments?

[DocShotgun]

Which ktransformer commit is used to run all above experiments?

If you mean my testing where I ran into the slight incoherence issues with Deepseek V3 0324, I was using kt-kernel from kvcache-ai/ktransformers@d7ab3b4.

[jyizheng]

Hello, and thank you for this! Finally an inference backend that can fully utilize my dual Xeon + single GPU setup.

I was able to download the Deepseek-v3-0324 quants from modelscope and deploy them locally, but I was wondering if you would be able to provide additional guidance on how to correctly convert model weights to use with this backend?

@intervitens managed to create quants identical to the ones provided on modelscope for GLM-4.5-Air by using llm-compressor with the included yaml recipe for the GPU portion, and the provided script for the CPU portion (which does have a typo where "subpool" should be "threadpool").

However we've been hitting some roadblocks with Deepseek. I modified the script to save safetensors files per layer rather than accumulating them until the end (or else Deepseek conversion requires more than 768gb RAM), and managed to convert the original fp8 Deepseek-V3-0324 to an INT4 quant that will load and infer without error, however produces no valid output - just empty string - possibly an overflow during inference? The resulting layer tensors are also not hash-identical like the ones for GLM-4.5-Air.

Update: It seems that something is broken with the conversion script's fp8 input support. Dequanting Deepseek to bf16 first seems to work.

Update 2: Uploaded some compatible weights to HF for anyone interested to try out.

Update 3: We seem to be having some problems with poor output quality - most noticeably with occasional misspelled words during generation. I initially attributed this to the RTN int4 experts, but it still seems to happen with int8 experts. Intervitens also got an aime24 score of 0.466 using fp8 non-expert params on GPU with int8 experts on CPU, while various Openrouter providers range from 0.5-0.63 for Deepseek V3 0324.

Thank you for the information. I am wondering how much CPU memory is needed to run deepseek v3 0324. Is 256 GB enough?