Question : Have you tried gemma 4 - 31b ? And can I use the latest llama cpp with this project ?

[x4080]

Nice works, ini this address, someone said that just using --mmap it can run directly big model directly with llama cpp (i tried but failed)

https://thoughts.jock.pl/p/local-llm-35b-mac-mini-gemma-swap-production-2026

Thats why your project seems intereseting

Have you tried gemma 4 - 31b or qwen 3.5 35b a3b ? And can I use the latest llama cpp with this project ?

Thanks

[t8]

Hey there,

I haven't gotten around to testing Gemma 4 with Hypura yet, but I don't see any reason why it wouldn't work. I have tested it with some Qwen models and they worked great.

Let me know what you find!

[AlexHarrowell]

It will not. Trying the LMStudio version of Gemma 4 26B-4B as here:

huggingface.co/lmstudio-community/gemma-4-26B-A4B-it-GGUF/blob/main/gemma-4-26B-A4B-it-Q4_K_M.gguf

fails with the message:

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4'
llama_model_load_from_file_impl: failed to load model

this suggests a check somewhere for an arch flag that is assumed to be LLaMa-like.

[AlexHarrowell]

It gets to the point of opening an interactive chat and prints a whole lot of model debug info that can only have come from the GGUF itself:

ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices
ggml_metal_library_init: using embedded metal library
ggml_metal_library_init: loaded in 0.028 sec
ggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)
ggml_metal_device_init: GPU name: MTL0
ggml_metal_device_init: GPU family: MTLGPUFamilyApple9 (1009)
ggml_metal_device_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_device_init: GPU family: MTLGPUFamilyMetal3 (5001)
ggml_metal_device_init: simdgroup reduction = true
ggml_metal_device_init: simdgroup matrix mul. = true
ggml_metal_device_init: has unified memory = true
ggml_metal_device_init: has bfloat = true
ggml_metal_device_init: has tensor = false
ggml_metal_device_init: use residency sets = true
ggml_metal_device_init: use shared buffers = true
ggml_metal_device_init: recommendedMaxWorkingSetSize = 11453.25 MB
llama_model_load_from_file_impl: using device MTL0 (Apple M3) (unknown id) - 10922 MiB free
llama_model_loader: loaded meta data with 46 key-value pairs and 658 tensors from /Users/aharrowell/Desktop/gemma-4-26B-A4B-it-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = gemma4
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.sampling.top_k i32 = 64
llama_model_loader: - kv 3: general.sampling.top_p f32 = 0.950000
llama_model_loader: - kv 4: general.sampling.temp f32 = 1.000000
llama_model_loader: - kv 5: general.name str = Gemma 4 26B A4B
llama_model_loader: - kv 6: general.finetune str = it
llama_model_loader: - kv 7: general.size_label str = 26B-A4B
llama_model_loader: - kv 8: gemma4.block_count u32 = 30
llama_model_loader: - kv 9: gemma4.context_length u32 = 262144
llama_model_loader: - kv 10: gemma4.embedding_length u32 = 2816
llama_model_loader: - kv 11: gemma4.feed_forward_length u32 = 2112
llama_model_loader: - kv 12: gemma4.attention.head_count u32 = 16
llama_model_loader: - kv 13: gemma4.attention.head_count_kv arr[i32,30] = [8, 8, 8, 8, 8, 2, 8, 8, 8, 8, 8, 2, ...
llama_model_loader: - kv 14: gemma4.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 15: gemma4.rope.freq_base_swa f32 = 10000.000000
llama_model_loader: - kv 16: gemma4.attention.layer_norm_rms_epsilon f32 = 0.000001
llama_model_loader: - kv 17: gemma4.expert_count u32 = 128
llama_model_loader: - kv 18: gemma4.expert_used_count u32 = 8
llama_model_loader: - kv 19: gemma4.attention.key_length u32 = 512
llama_model_loader: - kv 20: gemma4.attention.value_length u32 = 512
llama_model_loader: - kv 21: gemma4.final_logit_softcapping f32 = 30.000000
llama_model_loader: - kv 22: gemma4.attention.sliding_window u32 = 1024
llama_model_loader: - kv 23: gemma4.attention.shared_kv_layers u32 = 0
llama_model_loader: - kv 24: gemma4.embedding_length_per_layer_input u32 = 0
llama_model_loader: - kv 25: gemma4.attention.sliding_window_pattern arr[bool,30] = [true, true, true, true, true, false,...
llama_model_loader: - kv 26: gemma4.attention.key_length_swa u32 = 256
llama_model_loader: - kv 27: gemma4.attention.value_length_swa u32 = 256
llama_model_loader: - kv 28: gemma4.expert_feed_forward_length u32 = 704
llama_model_loader: - kv 29: gemma4.rope.dimension_count u32 = 512
llama_model_loader: - kv 30: gemma4.rope.dimension_count_swa u32 = 256
llama_model_loader: - kv 31: tokenizer.ggml.model str = gemma4
llama_model_loader: - kv 32: tokenizer.ggml.tokens arr[str,262144] = ["", "", "", "", ...
llama_model_loader: - kv 33: tokenizer.ggml.scores arr[f32,262144] = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,262144] = [3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,514906] = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv 36: tokenizer.ggml.bos_token_id u32 = 2
llama_model_loader: - kv 37: tokenizer.ggml.eos_token_id u32 = 1
llama_model_loader: - kv 38: tokenizer.ggml.unknown_token_id u32 = 3
llama_model_loader: - kv 39: tokenizer.ggml.padding_token_id u32 = 0
llama_model_loader: - kv 40: tokenizer.ggml.mask_token_id u32 = 4
llama_model_loader: - kv 41: tokenizer.chat_template str = {%- macro format_parameters(propertie...
llama_model_loader: - kv 42: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 43: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 44: general.quantization_version u32 = 2
llama_model_loader: - kv 45: general.file_type u32 = 15
llama_model_loader: - type f32: 392 tensors
llama_model_loader: - type q5_0: 32 tensors
llama_model_loader: - type q8_0: 28 tensors
llama_model_loader: - type q4_K: 192 tensors
llama_model_loader: - type q6_K: 14 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 15.63 GiB (5.32 BPW)

but then craps out:

llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'gemma4'
llama_model_load_from_file_impl: failed to load model

[x4080]

@AlexHarrowell Thanks for the heads up

[AlexHarrowell]

I am going to try the Unsloth 8-bit quantize later today.

…

On Tue, Apr 7, 2026 at 5:41 AM x4080 ***@***.***> wrote: *x4080* left a comment (t8/hypura#8) <#8 (comment)> @AlexHarrowell <https://github.com/AlexHarrowell> Thanks for the heads up — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATJPQXNEXGYTPRSKNWAJCT4USBHBAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJWGQ4TKOBWGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexHarrowell]

Tried. Same error. (Inspect and estimate for both also says it will place on the GPU but also run CPU only but that's a second order issue after it actually runs.) On Tue, Apr 7, 2026 at 10:40 AM Alexander Harrowell ***@***.***> wrote:

…

I am going to try the Unsloth 8-bit quantize later today. On Tue, Apr 7, 2026 at 5:41 AM x4080 ***@***.***> wrote: > *x4080* left a comment (t8/hypura#8) > <#8 (comment)> > > @AlexHarrowell <https://github.com/AlexHarrowell> Thanks for the heads up > > — > Reply to this email directly, view it on GitHub > <#8 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AATJPQXNEXGYTPRSKNWAJCT4USBHBAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJWGQ4TKOBWGI> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[t8]

Will mess around with this today and see what I can figure out

[AlexHarrowell]

Right, having found the "HOWTOAddModel.md" file, this will be because I thought it really did run any user-specified GGUF. Instead it needs a bunch of tinkering for each one and you send a pull request when you're done?

…

On Tue, Apr 7, 2026 at 5:51 PM Tate Berenbaum ***@***.***> wrote: *t8* left a comment (t8/hypura#8) <#8 (comment)> Will mess around with this today and see what I can figure out — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATJPQWEYVXNH4AM4EHWWN34UUW2DAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMBQG42TSNZRHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[t8]

Closed by #9.

What the fix does:

1. Bumps vendored llama.cpp to current master (66c4f9ded) — that's where the gemma4 architecture support lives, in ggml-org/llama.cpp#21309 plus seven follow-up patches. The vendored copy was just stale (March 13 vs the April 2 merge).
2. Three small Hypura adaptations: a GateUp variant for the MoE expert classifier (gemma4 fuses ffn_gate_up_exps where Mixtral keeps gate/up separate), array-valued attention.head_count_kv parsing (gemma4 has per-layer KV head counts because of interleaved sliding-window attention), and parse_special=true in the tokenize wrapper (was hardcoded false, which made chat-template tokens like <|turn> get tokenized as raw bytes and produce gibberish output).

Quick correction, @AlexHarrowell — there's no HOWTOAddModel.md file in this repo, and never has been. Whatever told you that was hallucinating. Hypura doesn't keep its own per-architecture tensor name lists; it classifies tensors by name patterns (blk.N.attn_q, ffn_gate_inp, _exps., etc.) that are consistent across llama.cpp model architectures. So adding a new model usually doesn't require Hypura-side work — it requires the architecture to exist in upstream llama.cpp first. Once it's there, Hypura mostly just works, and the three adjustments above were the only gaps gemma4 exposed.

Verified end-to-end against the same lmstudio-community/gemma-4-26B-A4B-it-Q4_K_M.gguf you're using. On an M1 Max 32GB:

• Loads cleanly, classified into the SparseMoeMmap path (6.25% activation → ~1 GB active working set fits in unified memory)
• 51 tok/s decode, 577 tok/s prompt eval
• Coherent output: The capital of France is **Paris**.

One important caveat: gemma4 is an instruction-tuned reasoning model and needs chat-formatted prompts to behave well. hypura run --prompt doesn't auto-apply the chat template yet, so for now you have to construct it manually:

hypura run \
  --prompt "$(printf '<|turn>user\nWrite a short poem about cats.<turn|>\n<|turn>model\n')" \
  --max-tokens 100 \
  ./gemma-4-26B-A4B-it-Q4_K_M.gguf

Note that gemma4 uses <|turn> / <turn|> / <|think|>, not the <start_of_turn> markers from gemma3 — those don't exist in gemma4's vocab. Without these markers (or with the wrong ones), even with the fixes above, the model produces nonsense because it never saw raw text in training. Auto-template-application is a worthwhile follow-up.

A couple of caveats for anyone finding this thread later:

• Multimodal bundles (e.g., Ollama's gemma4 pull, which packs the LLM plus vision + audio + projector into one GGUF) currently fail to load with done_getting_tensors: wrong number of tensors; expected 2131, got 720. That's an upstream limitation in llama_model_load — the LLM-only loader doesn't consume the multimodal projector tensors. The text-only lmstudio file in this thread doesn't hit it.
• @AlexHarrowell — re your inspect/estimate showing both GPU placement and CPU-only run, that's a separate concern about boundary cases on smaller GPUs (M3 has ~10 GB Metal working set vs M1 Max's ~22 GB). Once you can confirm the text-only model loads, want to open a follow-up issue with the actual inspect/estimate output? It's probably a budget threshold tweak.

[AlexHarrowell]

I am pretty sure none of Finder, unix ls, or TextEdit are subject to hallucinations. The path is: /hypura/vendor/llama.cpp/docs/development/HOWTO-add-model.md and the content is as follows: # Add a new model architecture to `llama.cpp` Adding a model requires few steps: 1. Convert the model to GGUF 2. Define the model architecture in `llama.cpp` 3. Build the GGML graph implementation After following these steps, you can open PR. Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially: - [cli](/tools/cli/) - [completion](/tools/completion/) - [imatrix](/tools/imatrix/) - [quantize](/tools/quantize/) - [server](/tools/server/) ### 1. Convert the model to GGUF This step is done in python with a `convert` script using the [gguf]( https://pypi.org/project/gguf/) library. Depending on the model architecture, you can use either [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for `llama/llama2` models in `.pth` format). The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors. The required steps to implement for an HF model are: 1. Define the model `ModelBase.register` annotation in a new `TextModel` or `MmprojModel` subclass, example: ```python @ModelBase.register("MyModelForCausalLM") class MyModel(TextModel): model_arch = gguf.MODEL_ARCH.MYMODEL ``` or ```python @ModelBase.register("MyModelForConditionalGeneration") class MyModel(MmprojModel): model_arch = gguf.MODEL_ARCH.MYMODEL ``` 2. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py) Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`. Example for `falcon` model: ```python MODEL_ARCH.FALCON: [ MODEL_TENSOR.TOKEN_EMBD, MODEL_TENSOR.OUTPUT_NORM, MODEL_TENSOR.OUTPUT, MODEL_TENSOR.ATTN_NORM, MODEL_TENSOR.ATTN_NORM_2, MODEL_TENSOR.ATTN_QKV, MODEL_TENSOR.ATTN_OUT, MODEL_TENSOR.FFN_DOWN, MODEL_TENSOR.FFN_UP, ] ``` 3. Map the original tensor names to the standardize equivalent in GGUF As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist. Once you have found the GGUF tensor name equivalent, add it to the [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file. If the tensor name is part of a repetitive layer/block, the key word `bid` substitutes it. Example for the normalization tensor in attention layers: ```python block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = { # Attention norm MODEL_TENSOR.ATTN_NORM: ( "gpt_neox.layers.{bid}.input_layernorm", # gptneox "transformer.h.{bid}.ln_1", # gpt2 gpt-j refact qwen "transformer.blocks.{bid}.norm_1", # mpt ... ) } ``` `transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` in GGUF. Depending on the model configuration, tokenizer, code and tensors layout, you will have to override: - `TextModel#set_gguf_parameters` - `MmprojModel#set_gguf_parameters` - `ModelBase#set_vocab` - `ModelBase#modify_tensors` NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is the convention and several tools like `quantize` expect this to proceed the weights. ### 2. Define the model architecture in `llama.cpp` The model params and tensors layout must be defined in `llama.cpp` source files: 1. Define a new `llm_arch` enum value in `src/llama-arch.h`. 2. In `src/llama-arch.cpp`: - Add the architecture name to the `LLM_ARCH_NAMES` map. - Add the list of model tensors to `llm_get_tensor_names` (you may also need to update `LLM_TENSOR_NAMES`) 3. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`. 4. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`. NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions. ### 3. Build the GGML graph implementation This is the funniest part, you have to provide the inference graph implementation of the new model architecture in `src/llama-model.cpp`. Create a new struct that inherits from `llm_graph_context` and implement the graph-building logic in its constructor. Have a look at existing implementations like `llm_build_llama`, `llm_build_dbrx` or `llm_build_bert`. Then, in the `llama_model::build_graph` method, add a case for your architecture to instantiate your new graph-building struct. Some `ggml` backends do not support all operations. Backend implementations can be added in a separate PR. Note: to debug the inference graph: you can use [llama-eval-callback](/examples/eval-callback/). ## GGUF specification https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ## Resources - YaRN RoPE scaling ggml-org/llama.cpp#2268 - support Baichuan serial models ggml-org/llama.cpp#3009 - support attention bias ggml-org/llama.cpp#4283 - Mixtral support ggml-org/llama.cpp#4406 - BERT embeddings ggml-org/llama.cpp#5423 - Grok-1 support ggml-org/llama.cpp#6204 - Command R Plus support ggml-org/llama.cpp#6491 - support arch DBRX ggml-org/llama.cpp#6515 - How to convert HuggingFace model to GGUF format ggml-org/llama.cpp#2948

…

On Wed, Apr 8, 2026 at 12:57 AM Tate Berenbaum ***@***.***> wrote: *t8* left a comment (t8/hypura#8) <#8 (comment)> Closed by #9 <#9>. *What the fix does:* 1. Bumps vendored llama.cpp to current master (66c4f9ded) — that's where the gemma4 architecture support lives, in ggml-org/llama.cpp#21309 <ggml-org/llama.cpp#21309> plus seven follow-up patches. The vendored copy was just stale (March 13 vs the April 2 merge). 2. Three small Hypura adaptations: a GateUp variant for the MoE expert classifier (gemma4 fuses ffn_gate_up_exps where Mixtral keeps gate/up separate), array-valued attention.head_count_kv parsing (gemma4 has per-layer KV head counts because of interleaved sliding-window attention), and parse_special=true in the tokenize wrapper (was hardcoded false, which made chat-template tokens like <|turn> get tokenized as raw bytes and produce gibberish output). *Quick correction, @AlexHarrowell <https://github.com/AlexHarrowell>* — there's no HOWTOAddModel.md file in this repo, and never has been. Whatever told you that was hallucinating. Hypura doesn't keep its own per-architecture tensor name lists; it classifies tensors by name patterns ( blk.N.attn_q, ffn_gate_inp, _exps., etc.) that are consistent across llama.cpp model architectures. So adding a new model usually doesn't require Hypura-side work — it requires the architecture to exist in upstream llama.cpp first. Once it's there, Hypura mostly just works, and the three adjustments above were the only gaps gemma4 exposed. *Verified end-to-end* against the same lmstudio-community/gemma-4-26B-A4B-it-Q4_K_M.gguf you're using. On an M1 Max 32GB: - Loads cleanly, classified into the SparseMoeMmap path (6.25% activation → ~1 GB active working set fits in unified memory) - *51 tok/s decode*, 577 tok/s prompt eval - Coherent output: The capital of France is **Paris**. *One important caveat:* gemma4 is an instruction-tuned reasoning model and needs chat-formatted prompts to behave well. hypura run --prompt doesn't auto-apply the chat template yet, so for now you have to construct it manually: hypura run \ --prompt "$(printf '<|turn>user\nWrite a short poem about cats.<turn|>\n<|turn>model\n')" \ --max-tokens 100 \ ./gemma-4-26B-A4B-it-Q4_K_M.gguf Note that gemma4 uses <|turn> / <turn|> / <|think|>, *not* the <start_of_turn> markers from gemma3 — those don't exist in gemma4's vocab. Without these markers (or with the wrong ones), even with the fixes above, the model produces nonsense because it never saw raw text in training. Auto-template-application is a worthwhile follow-up. A couple of caveats for anyone finding this thread later: - *Multimodal bundles* (e.g., Ollama's gemma4 pull, which packs the LLM plus vision + audio + projector into one GGUF) currently fail to load with done_getting_tensors: wrong number of tensors; expected 2131, got 720. That's an upstream limitation in llama_model_load — the LLM-only loader doesn't consume the multimodal projector tensors. The text-only lmstudio file in this thread doesn't hit it. - @AlexHarrowell <https://github.com/AlexHarrowell> — re your inspect/estimate showing both GPU placement and CPU-only run, that's a separate concern about boundary cases on smaller GPUs (M3 has ~10 GB Metal working set vs M1 Max's ~22 GB). Once you can confirm the text-only model loads, want to open a follow-up issue with the actual inspect/estimate output? It's probably a budget threshold tweak. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AATJPQWD2XPG3RDWYZMYSYL4UWIVHAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMBSHA4DSNRUHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlexHarrowell]

further follow-up: v0.2.0 has the same error on the Unsloth GGUF. On Wed, Apr 8, 2026 at 8:02 AM Alexander Harrowell ***@***.***> wrote:

…

I am pretty sure none of Finder, unix ls, or TextEdit are subject to hallucinations. The path is: /hypura/vendor/llama.cpp/docs/development/HOWTO-add-model.md and the content is as follows: # Add a new model architecture to `llama.cpp` Adding a model requires few steps: 1. Convert the model to GGUF 2. Define the model architecture in `llama.cpp` 3. Build the GGML graph implementation After following these steps, you can open PR. Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially: - [cli](/tools/cli/) - [completion](/tools/completion/) - [imatrix](/tools/imatrix/) - [quantize](/tools/quantize/) - [server](/tools/server/) ### 1. Convert the model to GGUF This step is done in python with a `convert` script using the [gguf]( https://pypi.org/project/gguf/) library. Depending on the model architecture, you can use either [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for `llama/llama2` models in `.pth` format). The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors. The required steps to implement for an HF model are: 1. Define the model `ModelBase.register` annotation in a new `TextModel` or `MmprojModel` subclass, example: ```python @ModelBase.register("MyModelForCausalLM") class MyModel(TextModel): model_arch = gguf.MODEL_ARCH.MYMODEL ``` or ```python @ModelBase.register("MyModelForConditionalGeneration") class MyModel(MmprojModel): model_arch = gguf.MODEL_ARCH.MYMODEL ``` 2. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py) Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`. Example for `falcon` model: ```python MODEL_ARCH.FALCON: [ MODEL_TENSOR.TOKEN_EMBD, MODEL_TENSOR.OUTPUT_NORM, MODEL_TENSOR.OUTPUT, MODEL_TENSOR.ATTN_NORM, MODEL_TENSOR.ATTN_NORM_2, MODEL_TENSOR.ATTN_QKV, MODEL_TENSOR.ATTN_OUT, MODEL_TENSOR.FFN_DOWN, MODEL_TENSOR.FFN_UP, ] ``` 3. Map the original tensor names to the standardize equivalent in GGUF As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist. Once you have found the GGUF tensor name equivalent, add it to the [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file. If the tensor name is part of a repetitive layer/block, the key word `bid` substitutes it. Example for the normalization tensor in attention layers: ```python block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = { # Attention norm MODEL_TENSOR.ATTN_NORM: ( "gpt_neox.layers.{bid}.input_layernorm", # gptneox "transformer.h.{bid}.ln_1", # gpt2 gpt-j refact qwen "transformer.blocks.{bid}.norm_1", # mpt ... ) } ``` `transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` in GGUF. Depending on the model configuration, tokenizer, code and tensors layout, you will have to override: - `TextModel#set_gguf_parameters` - `MmprojModel#set_gguf_parameters` - `ModelBase#set_vocab` - `ModelBase#modify_tensors` NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is the convention and several tools like `quantize` expect this to proceed the weights. ### 2. Define the model architecture in `llama.cpp` The model params and tensors layout must be defined in `llama.cpp` source files: 1. Define a new `llm_arch` enum value in `src/llama-arch.h`. 2. In `src/llama-arch.cpp`: - Add the architecture name to the `LLM_ARCH_NAMES` map. - Add the list of model tensors to `llm_get_tensor_names` (you may also need to update `LLM_TENSOR_NAMES`) 3. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`. 4. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`. NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions. ### 3. Build the GGML graph implementation This is the funniest part, you have to provide the inference graph implementation of the new model architecture in `src/llama-model.cpp`. Create a new struct that inherits from `llm_graph_context` and implement the graph-building logic in its constructor. Have a look at existing implementations like `llm_build_llama`, `llm_build_dbrx` or `llm_build_bert`. Then, in the `llama_model::build_graph` method, add a case for your architecture to instantiate your new graph-building struct. Some `ggml` backends do not support all operations. Backend implementations can be added in a separate PR. Note: to debug the inference graph: you can use [llama-eval-callback](/examples/eval-callback/). ## GGUF specification https://github.com/ggml-org/ggml/blob/master/docs/gguf.md ## Resources - YaRN RoPE scaling ggml-org/llama.cpp#2268 - support Baichuan serial models ggml-org/llama.cpp#3009 - support attention bias ggml-org/llama.cpp#4283 - Mixtral support ggml-org/llama.cpp#4406 - BERT embeddings ggml-org/llama.cpp#5423 - Grok-1 support ggml-org/llama.cpp#6204 - Command R Plus support ggml-org/llama.cpp#6491 - support arch DBRX ggml-org/llama.cpp#6515 - How to convert HuggingFace model to GGUF format ggml-org/llama.cpp#2948 On Wed, Apr 8, 2026 at 12:57 AM Tate Berenbaum ***@***.***> wrote: > *t8* left a comment (t8/hypura#8) > <#8 (comment)> > > Closed by #9 <#9>. > > *What the fix does:* > > 1. Bumps vendored llama.cpp to current master (66c4f9ded) — that's > where the gemma4 architecture support lives, in > ggml-org/llama.cpp#21309 > <ggml-org/llama.cpp#21309> plus seven > follow-up patches. The vendored copy was just stale (March 13 vs the April > 2 merge). > 2. Three small Hypura adaptations: a GateUp variant for the MoE > expert classifier (gemma4 fuses ffn_gate_up_exps where Mixtral keeps > gate/up separate), array-valued attention.head_count_kv parsing > (gemma4 has per-layer KV head counts because of interleaved sliding-window > attention), and parse_special=true in the tokenize wrapper (was > hardcoded false, which made chat-template tokens like <|turn> get > tokenized as raw bytes and produce gibberish output). > > *Quick correction, @AlexHarrowell <https://github.com/AlexHarrowell>* — > there's no HOWTOAddModel.md file in this repo, and never has been. > Whatever told you that was hallucinating. Hypura doesn't keep its own > per-architecture tensor name lists; it classifies tensors by name patterns ( > blk.N.attn_q, ffn_gate_inp, _exps., etc.) that are consistent across > llama.cpp model architectures. So adding a new model usually doesn't > require Hypura-side work — it requires the architecture to exist in > upstream llama.cpp first. Once it's there, Hypura mostly just works, and > the three adjustments above were the only gaps gemma4 exposed. > > *Verified end-to-end* against the same > lmstudio-community/gemma-4-26B-A4B-it-Q4_K_M.gguf you're using. On an M1 > Max 32GB: > > - Loads cleanly, classified into the SparseMoeMmap path (6.25% > activation → ~1 GB active working set fits in unified memory) > - *51 tok/s decode*, 577 tok/s prompt eval > - Coherent output: The capital of France is **Paris**. > > *One important caveat:* gemma4 is an instruction-tuned reasoning model > and needs chat-formatted prompts to behave well. hypura run --prompt > doesn't auto-apply the chat template yet, so for now you have to construct > it manually: > > hypura run \ > --prompt "$(printf '<|turn>user\nWrite a short poem about cats.<turn|>\n<|turn>model\n')" \ > --max-tokens 100 \ > ./gemma-4-26B-A4B-it-Q4_K_M.gguf > > Note that gemma4 uses <|turn> / <turn|> / <|think|>, *not* the > <start_of_turn> markers from gemma3 — those don't exist in gemma4's > vocab. Without these markers (or with the wrong ones), even with the fixes > above, the model produces nonsense because it never saw raw text in > training. Auto-template-application is a worthwhile follow-up. > > A couple of caveats for anyone finding this thread later: > > - *Multimodal bundles* (e.g., Ollama's gemma4 pull, which packs the > LLM plus vision + audio + projector into one GGUF) currently fail to load > with done_getting_tensors: wrong number of tensors; expected 2131, > got 720. That's an upstream limitation in llama_model_load — the > LLM-only loader doesn't consume the multimodal projector tensors. The > text-only lmstudio file in this thread doesn't hit it. > - @AlexHarrowell <https://github.com/AlexHarrowell> — re your > inspect/estimate showing both GPU placement and CPU-only run, that's a > separate concern about boundary cases on smaller GPUs (M3 has ~10 GB Metal > working set vs M1 Max's ~22 GB). Once you can confirm the text-only model > loads, want to open a follow-up issue with the actual inspect/estimate > output? It's probably a budget threshold tweak. > > — > Reply to this email directly, view it on GitHub > <#8 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AATJPQWD2XPG3RDWYZMYSYL4UWIVHAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMBSHA4DSNRUHE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[AlexHarrowell]

and also on the LMStudio 4-bit gguf. On Wed, Apr 8, 2026 at 12:22 PM Alexander Harrowell ***@***.***> wrote:

…

further follow-up: v0.2.0 has the same error on the Unsloth GGUF. On Wed, Apr 8, 2026 at 8:02 AM Alexander Harrowell ***@***.***> wrote: > I am pretty sure none of Finder, unix ls, or TextEdit are subject to > hallucinations. The path is: > > /hypura/vendor/llama.cpp/docs/development/HOWTO-add-model.md > > and the content is as follows: > > # Add a new model architecture to `llama.cpp` > > Adding a model requires few steps: > > 1. Convert the model to GGUF > 2. Define the model architecture in `llama.cpp` > 3. Build the GGML graph implementation > > After following these steps, you can open PR. > > Also, it is important to check that the examples and main ggml backends > (CUDA, METAL, CPU) are working with the new architecture, especially: > - [cli](/tools/cli/) > - [completion](/tools/completion/) > - [imatrix](/tools/imatrix/) > - [quantize](/tools/quantize/) > - [server](/tools/server/) > > ### 1. Convert the model to GGUF > > This step is done in python with a `convert` script using the [gguf]( > https://pypi.org/project/gguf/) library. > Depending on the model architecture, you can use either > [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or > [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for > `llama/llama2` models in `.pth` format). > > The convert script reads the model configuration, tokenizer, tensor > names+data and converts them to GGUF metadata and tensors. > > The required steps to implement for an HF model are: > > 1. Define the model `ModelBase.register` annotation in a new `TextModel` > or `MmprojModel` subclass, example: > > ```python > @ModelBase.register("MyModelForCausalLM") > class MyModel(TextModel): > model_arch = gguf.MODEL_ARCH.MYMODEL > ``` > > or > > ```python > @ModelBase.register("MyModelForConditionalGeneration") > class MyModel(MmprojModel): > model_arch = gguf.MODEL_ARCH.MYMODEL > ``` > > 2. Define the layout of the GGUF tensors in > [constants.py](/gguf-py/gguf/constants.py) > > Add an enum entry in `MODEL_ARCH`, the model human friendly name in > `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`. > > Example for `falcon` model: > ```python > MODEL_ARCH.FALCON: [ > MODEL_TENSOR.TOKEN_EMBD, > MODEL_TENSOR.OUTPUT_NORM, > MODEL_TENSOR.OUTPUT, > MODEL_TENSOR.ATTN_NORM, > MODEL_TENSOR.ATTN_NORM_2, > MODEL_TENSOR.ATTN_QKV, > MODEL_TENSOR.ATTN_OUT, > MODEL_TENSOR.FFN_DOWN, > MODEL_TENSOR.FFN_UP, > ] > ``` > > 3. Map the original tensor names to the standardize equivalent in GGUF > > As a general rule, before adding a new tensor name to GGUF, be sure the > equivalent naming does not already exist. > > Once you have found the GGUF tensor name equivalent, add it to the > [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file. > > If the tensor name is part of a repetitive layer/block, the key word > `bid` substitutes it. > > Example for the normalization tensor in attention layers: > > ```python > block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = { > # Attention norm > MODEL_TENSOR.ATTN_NORM: ( > "gpt_neox.layers.{bid}.input_layernorm", # > gptneox > "transformer.h.{bid}.ln_1", # > gpt2 gpt-j refact qwen > "transformer.blocks.{bid}.norm_1", # mpt > ... > ) > } > ``` > > `transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` > in GGUF. > > Depending on the model configuration, tokenizer, code and tensors layout, > you will have to override: > - `TextModel#set_gguf_parameters` > - `MmprojModel#set_gguf_parameters` > - `ModelBase#set_vocab` > - `ModelBase#modify_tensors` > > NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is > the convention and several tools like `quantize` expect this to proceed the > weights. > > ### 2. Define the model architecture in `llama.cpp` > > The model params and tensors layout must be defined in `llama.cpp` source > files: > 1. Define a new `llm_arch` enum value in `src/llama-arch.h`. > 2. In `src/llama-arch.cpp`: > - Add the architecture name to the `LLM_ARCH_NAMES` map. > - Add the list of model tensors to `llm_get_tensor_names` (you may > also need to update `LLM_TENSOR_NAMES`) > 3. Add any non-standard metadata loading in the `llama_model_loader` > constructor in `src/llama-model-loader.cpp`. > 4. If the model has a RoPE operation, add a case for the architecture in > `llama_model_rope_type` function in `src/llama-model.cpp`. > > NOTE: The dimensions in `ggml` are typically in the reverse order of the > `pytorch` dimensions. > > ### 3. Build the GGML graph implementation > > This is the funniest part, you have to provide the inference graph > implementation of the new model architecture in `src/llama-model.cpp`. > Create a new struct that inherits from `llm_graph_context` and implement > the graph-building logic in its constructor. > Have a look at existing implementations like `llm_build_llama`, > `llm_build_dbrx` or `llm_build_bert`. > Then, in the `llama_model::build_graph` method, add a case for your > architecture to instantiate your new graph-building struct. > > Some `ggml` backends do not support all operations. Backend > implementations can be added in a separate PR. > > Note: to debug the inference graph: you can use > [llama-eval-callback](/examples/eval-callback/). > > ## GGUF specification > > https://github.com/ggml-org/ggml/blob/master/docs/gguf.md > > ## Resources > > - YaRN RoPE scaling ggml-org/llama.cpp#2268 > - support Baichuan serial models > ggml-org/llama.cpp#3009 > - support attention bias ggml-org/llama.cpp#4283 > - Mixtral support ggml-org/llama.cpp#4406 > - BERT embeddings ggml-org/llama.cpp#5423 > - Grok-1 support ggml-org/llama.cpp#6204 > - Command R Plus support ggml-org/llama.cpp#6491 > - support arch DBRX ggml-org/llama.cpp#6515 > - How to convert HuggingFace model to GGUF format > ggml-org/llama.cpp#2948 > > > > > On Wed, Apr 8, 2026 at 12:57 AM Tate Berenbaum ***@***.***> > wrote: > >> *t8* left a comment (t8/hypura#8) >> <#8 (comment)> >> >> Closed by #9 <#9>. >> >> *What the fix does:* >> >> 1. Bumps vendored llama.cpp to current master (66c4f9ded) — that's >> where the gemma4 architecture support lives, in >> ggml-org/llama.cpp#21309 >> <ggml-org/llama.cpp#21309> plus seven >> follow-up patches. The vendored copy was just stale (March 13 vs the April >> 2 merge). >> 2. Three small Hypura adaptations: a GateUp variant for the MoE >> expert classifier (gemma4 fuses ffn_gate_up_exps where Mixtral keeps >> gate/up separate), array-valued attention.head_count_kv parsing >> (gemma4 has per-layer KV head counts because of interleaved sliding-window >> attention), and parse_special=true in the tokenize wrapper (was >> hardcoded false, which made chat-template tokens like <|turn> get >> tokenized as raw bytes and produce gibberish output). >> >> *Quick correction, @AlexHarrowell <https://github.com/AlexHarrowell>* — >> there's no HOWTOAddModel.md file in this repo, and never has been. >> Whatever told you that was hallucinating. Hypura doesn't keep its own >> per-architecture tensor name lists; it classifies tensors by name patterns ( >> blk.N.attn_q, ffn_gate_inp, _exps., etc.) that are consistent across >> llama.cpp model architectures. So adding a new model usually doesn't >> require Hypura-side work — it requires the architecture to exist in >> upstream llama.cpp first. Once it's there, Hypura mostly just works, >> and the three adjustments above were the only gaps gemma4 exposed. >> >> *Verified end-to-end* against the same >> lmstudio-community/gemma-4-26B-A4B-it-Q4_K_M.gguf you're using. On an >> M1 Max 32GB: >> >> - Loads cleanly, classified into the SparseMoeMmap path (6.25% >> activation → ~1 GB active working set fits in unified memory) >> - *51 tok/s decode*, 577 tok/s prompt eval >> - Coherent output: The capital of France is **Paris**. >> >> *One important caveat:* gemma4 is an instruction-tuned reasoning model >> and needs chat-formatted prompts to behave well. hypura run --prompt >> doesn't auto-apply the chat template yet, so for now you have to construct >> it manually: >> >> hypura run \ >> --prompt "$(printf '<|turn>user\nWrite a short poem about cats.<turn|>\n<|turn>model\n')" \ >> --max-tokens 100 \ >> ./gemma-4-26B-A4B-it-Q4_K_M.gguf >> >> Note that gemma4 uses <|turn> / <turn|> / <|think|>, *not* the >> <start_of_turn> markers from gemma3 — those don't exist in gemma4's >> vocab. Without these markers (or with the wrong ones), even with the fixes >> above, the model produces nonsense because it never saw raw text in >> training. Auto-template-application is a worthwhile follow-up. >> >> A couple of caveats for anyone finding this thread later: >> >> - *Multimodal bundles* (e.g., Ollama's gemma4 pull, which packs the >> LLM plus vision + audio + projector into one GGUF) currently fail to load >> with done_getting_tensors: wrong number of tensors; expected 2131, >> got 720. That's an upstream limitation in llama_model_load — the >> LLM-only loader doesn't consume the multimodal projector tensors. The >> text-only lmstudio file in this thread doesn't hit it. >> - @AlexHarrowell <https://github.com/AlexHarrowell> — re your >> inspect/estimate showing both GPU placement and CPU-only run, that's a >> separate concern about boundary cases on smaller GPUs (M3 has ~10 GB Metal >> working set vs M1 Max's ~22 GB). Once you can confirm the text-only model >> loads, want to open a follow-up issue with the actual inspect/ >> estimate output? It's probably a budget threshold tweak. >> >> — >> Reply to this email directly, view it on GitHub >> <#8 (comment)>, or >> unsubscribe >> <https://github.com/notifications/unsubscribe-auth/AATJPQWD2XPG3RDWYZMYSYL4UWIVHAVCNFSM6AAAAACXNDNQYGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMBSHA4DSNRUHE> >> . >> You are receiving this because you were mentioned.Message ID: >> ***@***.***> >> >

[t8]

You're referencing llama.cpp in the comments above. I think your issue is with that, not Hypura.

More context- everything in the vendor folder is a llama.cpp git submodule.