Add `--grammar-file` argument to `server` (similar to how `main` does it)

[ibehnam]

Feature Description

Similar to the docs for main (https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md), it'd be great to have the --grammar-file=... flag available in server as well.

Motivation

Currently, server can't process long grammars. I don't know if it's a bug but I've noticed that even with the json.gbnf files in the repo. Basically, the content of the file gets too complicated for the grammar parser to read. I think it has to do with how multi-line strings are treated in terminal.

In any case, using grammar-file simplifies API calls a lot.

[Maximilian-Winter]

@ibehnam I'm curious, can you show me the complicated grammar? Because I never had any problems with it.

[ibehnam]

@Maximilian-Winter Sure! For example, in one project I am dealing with a highly-nested Pydantic hierarchy like this:

from typing import Optional, List, Union, Literal
from pydantic import BaseModel, Field

class A(BaseModel):
    a: int = Field(...)
    b: Optional[int] = Field(None)
    c: Optional[int] = Field(None)
    d: Optional[int] = Field(None)
    e: Optional[int] = Field(None)
    f: Optional[int] = Field(None)

class B(BaseModel):
    g: A = Field(...)
    h: str = Field(...)

class C(BaseModel):
    i: str = Field(...)

class D(BaseModel):
    j: str = Field(...)
    k: Optional[List[Union[B, C]]] = Field(None)

class E(BaseModel):
    l: str = Field(...)
    m: Optional[List[Union[B, C]]] = Field(None)

class F(BaseModel):
    n: int = Field(...)
    o: str = Field(...)
    p: str = Field(...)

class G(BaseModel):
    q: int = Field(...)
    r: str = Field(...)
    s: str = Field(...)

class H(str, Enum):
    T = 't'
    U = 'u'
    V = 'v'

class I(BaseModel):
    w: H = Field(...)
    x: Union[List[E], F, G] = Field(...)

class J(BaseModel):
    y: str = Field(...)
    z: Optional[List[I]] = Field(None)

class K(BaseModel):
    aa: str = Field(...)
    ab: Optional[List[I]] = Field(None)
    ac: Optional[List[J]] = Field(None)

class L(BaseModel):
    ad: str = Field(...)
    ae: Optional[List[I]] = Field(None)
    af: Optional[List[K]] = Field(None)

class M(BaseModel):
    ag: Literal['ah', 'ai', 'aj'] = Field(...)
    ak: str = Field(...)
    al: Optional[str] = Field(None)
    am: Optional[List[I]] = Field(None)
    an: Optional[List[L]] = Field(None)
    ao: Optional[str] = Field(None)

class N(BaseModel):
    ap: str = Field(...)
    aq: List[M] = Field(...)
    ar: Optional[List[str]] = Field(None)

I want to create a grammar for any object, including N. I even used your awesome llama-cpp-agent package but even that couldn't handle the hierarchy:

...
generate_gbnf_rule_for_type(model_name, field_name, field_type, is_optional, processed_models, created_rules, field_info)
    293 if isclass(field_type) and issubclass(field_type, BaseModel):
    294     nested_model_name = format_model_and_field_name(field_type.__name__)
--> 295     nested_model_rules, _ = generate_gbnf_grammar(field_type, processed_models, created_rules)
    296     rules.extend(nested_model_rules)
    297     gbnf_type, rules = nested_model_name, rules

ValueError: not enough values to unpack (expected 2, got 0)

It seems like anything above level J throws that error. But even with some simpler (lower level) classes that llama-cpp-agent can handle, the grammar is long and crashes the llama.cpp server.

[Maximilian-Winter]

@ibehnam I will try to find a fix, either by updating my agent framework or the llama.cpp grammar generator I added. Or both.

[Maximilian-Winter]

@ibehnam I have tried your example with the grammar generator I added from my framework to llama.cpp examples. And it works without problems!

from typing import Optional, List, Union, Literal
from pydantic import BaseModel, Field

class A(BaseModel):
    a: int = Field(...)
    b: Optional[int] = Field(None)
    c: Optional[int] = Field(None)
    d: Optional[int] = Field(None)
    e: Optional[int] = Field(None)
    f: Optional[int] = Field(None)

class B(BaseModel):
    g: A = Field(...)
    h: str = Field(...)

class C(BaseModel):
    i: str = Field(...)

class D(BaseModel):
    j: str = Field(...)
    k: Optional[List[Union[B, C]]] = Field(None)

class E(BaseModel):
    l: str = Field(...)
    m: Optional[List[Union[B, C]]] = Field(None)

class F(BaseModel):
    n: int = Field(...)
    o: str = Field(...)
    p: str = Field(...)

class G(BaseModel):
    q: int = Field(...)
    r: str = Field(...)
    s: str = Field(...)

class H(str, Enum):
    T = 't'
    U = 'u'
    V = 'v'

class I(BaseModel):
    w: H = Field(...)
    x: Union[List[E], F, G] = Field(...)

class J(BaseModel):
    y: str = Field(...)
    z: Optional[List[I]] = Field(None)

class K(BaseModel):
    aa: str = Field(...)
    ab: Optional[List[I]] = Field(None)
    ac: Optional[List[J]] = Field(None)

class L(BaseModel):
    ad: str = Field(...)
    ae: Optional[List[I]] = Field(None)
    af: Optional[List[K]] = Field(None)

class M(BaseModel):
    ag: Literal['ah', 'ai', 'aj'] = Field(...)
    ak: str = Field(...)
    al: Optional[str] = Field(None)
    am: Optional[List[I]] = Field(None)
    an: Optional[List[L]] = Field(None)
    ao: Optional[str] = Field(None)

class N(BaseModel):
    ap: str = Field(...)
    aq: List[M] = Field(...)
    ar: Optional[List[str]] = Field(None)

gbnf_grammar, _ = generate_gbnf_grammar_and_documentation([J])
system_message = "You are an advanced AI assistant. You are interacting with the user and with your environment by calling functions. You call functions by writing JSON objects, which represent specific function calls.\nBelow is a list of your available function calls:\n\n" + documentation


text = """Do something random!"""
prompt = f"<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{text}<|im_end|>\n<|im_start|>assistant"

text = create_completion(prompt=prompt, grammar=gbnf_grammar)

json_data = json.loads(text)

print(json_data)

Which results in:

{
  "y": "Function",
  "z": [
    {
      "w": "u",
      "x": {
        "q": 1,
        "r": 
         "AI has chosen to call the 'Calculator' function. It will perform a math operation.",
        "s": "+/-"
      }
    },
    {
      "w": "v",
      "x": {
        "q": 2,
        "r": 
         "AI has chosen to call the 'SendMessageToUser' function. It will send a message.",
        "s": ""
      }
    }]
}

[Maximilian-Winter]

I also tried A

[Maximilian-Winter]

@ibehnam Let me know if you still have problems.

[ibehnam]

@Maximilian-Winter I tried your llama-cpp-agent directly (not the grammar generator here, but I think they're the same?), and yes, it works for A to J, but for anything higher than J it throws an error 🤔

[Maximilian-Winter]

@ibehnam I made some updates that I didn't pushed yet to the agent framework. Because I want to fix a couple of things in the next version of the framework. The problems arose as I worked at MemGPT and this grammar generator for llama.cpp.

[Maximilian-Winter]

@ibehnam But let me check your other models too.

[Maximilian-Winter]

@ibehnam I looked at your examples and the problem is the grammar generator doesn't support literals directly, but you could use a string enum for your example or any other type enum. The LLM will then write the value of the enum. Hopes this helps.

[Maximilian-Winter]

@ibehnam The generator in llama.cpp examples should handle your pydantic model correctly.

[Maximilian-Winter]

Beside from the literals issue.

[ibehnam]

@Maximilian-Winter Thanks for looking into this—Yes, I had a feeling it has to do with Literals, but since your code also took care of non-Pydantic models, I assumed Literals were fine:

# gbnf_grammar_from_pydantic_models.py
...
    if not issubclass(model, BaseModel):
        # For non-Pydantic classes, generate model_fields from __annotations__ or __init__
        if hasattr(model, "__annotations__") and model.__annotations__:
            model_fields = {name: (typ, ...) for name, typ in model.__annotations__.items()}
        else:
            init_signature = inspect.signature(model.__init__)
            parameters = init_signature.parameters
            model_fields = {name: (param.annotation, param.default) for name, param in parameters.items() if
                            name != "self"}
    else:
        # For Pydantic models, use model_fields and check for ellipsis (required fields)
        model_fields = model.__annotations__
...

I'm going to test the one you included in this repo and will update here!

[Maximilian-Winter]

@ibehnam To be honest, I always use enum as literals when working with the grammar generator. It is my fault. Will try to add them soon.