GraphWalks

An OpenAI-compatible proxy that answers graph-structure questions by combining FalkorDBLite (embedded graph database) with a small LLM.

Instead of relying on a large language model to reason about graphs from raw text, GraphWalks:

1. Parses the graph edges and question from the prompt.
2. Loads the graph into an embedded FalkorDBLite instance (no external server needed).
3. Generates a Cypher query via a lightweight LLM (Ollama, vLLM, or any OpenAI-compatible endpoint).
4. Executes the Cypher against FalkorDB and returns the result — formatted as a standard OpenAI chat completion.

Client (OpenAI SDK)
  │
  ▼
POST /v1/chat/completions
  │
  ├─ Parse edges & question
  ├─ Load graph → FalkorDBLite (embedded, ephemeral per-request)
  ├─ LLM → Cypher query
  ├─ Execute Cypher → result
  ├─ Format answer
  └─ Delete graph
  │
  ▼
OpenAI-compatible JSON response

Quick Start

# Install uv (if needed): https://docs.astral.sh/uv/getting-started/installation/
uv sync

# Run the proxy (FalkorDBLite is embedded — no Docker, no external DB)
uv run python main.py

The proxy starts on http://localhost:8000. That's it — no containers, no database setup.

Try It

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "graphwalks-proxy",
    "messages": [{"role": "user", "content": "(A)->(B)\n(B)->(C)\n(C)->(D)\n(A)->(D)\nQuestion: What is the shortest path from A to D?"}]
  }' | python -m json.tool

Configuration

All settings are configured via environment variables (or a .env file):

Variable	Default	Description
FALKORDB_DB_PATH	/tmp/graphwalks.db	Path for the embedded FalkorDBLite database file
LLM_MODEL	ollama/mistral	LiteLLM model ID (e.g. gpt-4o, claude-3-haiku-20240307, ollama/llama3)
LLM_API_BASE	—	Custom API base URL (optional, for self-hosted endpoints)
LLM_TEMPERATURE	0.0	Sampling temperature
LLM_MAX_TOKENS	512	Max tokens for Cypher generation
SERVER_HOST	0.0.0.0	Proxy listen address
SERVER_PORT	8000	Proxy listen port
PROXY_MODEL_NAME	graphwalks-proxy	Model name returned in responses
CYPHER_QUERY_TIMEOUT_MS	5000	Cypher query timeout (ms)
LLM_RETRY_COUNT	1	Retries on failed Cypher generation

LLM provider API keys are set via standard env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.) — see LiteLLM docs.

Project Structure

graphwalks/
├── main.py             # FastAPI app & endpoints
├── prompt_parser.py    # Regex-based edge/question extraction
├── graph_manager.py    # Async FalkorDBLite lifecycle (load, query, delete)
├── llm_client.py       # Text-to-Cypher LLM orchestrator
├── executor.py         # End-to-end pipeline
├── models.py           # Pydantic OpenAI request/response models
├── config.py           # Environment-based configuration
├── pyproject.toml
└── tests/
    └── test_prompt_parser.py

API Endpoints

Method	Path	Description
POST	/v1/chat/completions	OpenAI-compatible chat completion
GET	/v1/models	List available models
GET	/health	Health check

How It Works

Prompt Parsing

The parser extracts directed edges in common formats:

(A)->(B)        # parenthesized
A -> B          # spaced
(A)-[3.5]->(B)  # weighted

The question is identified by Operation: markers (GraphWalks benchmark format), generic markers like Question:, Find, What, etc., or falls back to the last non-edge line.

Benchmark prompts with <begin example>...<end example> sections are handled — example edges are excluded from the parsed graph.

Graph Lifecycle

Each request gets a UUID-named ephemeral graph in the embedded FalkorDBLite instance. Edges are bulk-loaded via a single UNWIND/MERGE Cypher statement. The graph is automatically deleted after the response is sent (even on errors).

Cypher Generation

The LLM receives a strict system prompt constraining output to read-only Cypher. Generated queries are validated — any write keywords (CREATE, DELETE, SET, etc.) are rejected. On failure, the LLM gets one retry with the error message as feedback.

Running the GraphWalks Benchmark

The proxy is designed to be evaluated against the openai/graphwalks HuggingFace dataset.

Using the built-in runner

# Install bench dependencies
uv sync --group bench

# Run against the local proxy (start it first with uv run python main.py)
uv run python bench.py --limit 20

# Filter by problem type
uv run python bench.py --problem-type parents --limit 50

# Run the large dataset (256K–1M char prompts)
uv run python bench.py --size large --limit 10

# Run all sizes
uv run python bench.py --size all --problem-type bfs

# Save results to JSON
uv run python bench.py --limit 100 --output results.json

# Target a different endpoint
uv run python bench.py --base-url http://localhost:8000/v1 --model graphwalks-proxy

The --size flag selects the dataset tier:

• small (default) — prompts ≤128K chars (650 samples: 350 parents + 300 BFS)
• large — prompts >128K chars (500 samples: 250 parents + 250 BFS, graphs up to ~70K edges)
• all — entire dataset (1150 samples)

Using lm-evaluation-harness

pip install lm-eval

lm_eval --model local-completions \
  --model_args model=graphwalks-proxy,base_url=http://localhost:8000/v1 \
  --tasks graphwalks_128k

Testing

uv run pytest tests/ -v

License

See LICENSE.