Bifrost + Langfuse — Local AI Gateway Lab

A local development stack that runs Bifrost as an AI Gateway in front of Amazon Bedrock, with Langfuse for LLM observability — all wired together via OpenTelemetry.

Architecture

┌──────────────┐      ┌──────────────────┐      ┌──────────────────┐
│  Your Code   │─────▶│  Bifrost Gateway  │─────▶│  Amazon Bedrock  │
│  (boto3 /    │      │  :8080            │      │  (Nova, Titan)   │
│   LangChain) │      │                  │      └──────────────────┘
└──────────────┘      │  ┌────────────┐  │
                      │  │ Sem. Cache │  │──▶ Redis (vector store)
                      │  └────────────┘  │
                      │  ┌────────────┐  │
                      │  │   OTel     │  │──▶ Langfuse (traces)
                      │  └────────────┘  │
                      └──────────────────┘

What Bifrost provides:

• Weighted routing across Bedrock regions (EU ↔ US)
• Semantic caching via Redis + Titan Embeddings
• OpenTelemetry traces exported to Langfuse
• Virtual key governance

Prerequisites

• Docker & Docker Compose
• OrbStack (recommended on macOS for .local domains)
• AWS CLI configured with a profile that has Bedrock access
• uv (for Python dependency management)

Quick Start

1. Configure environment

cp .env.example .env
# Edit .env — set your AWS_PROFILE at minimum

2. Start the stack

make up

This brings up:

Service	URL	Description
Bifrost Gateway	http://bifrost.local	AI Gateway (proxy to Bedrock)
Bifrost UI	http://bifrost.local	Built-in dashboard
Langfuse	http://langfuse.local	Observability UI

3. Run the notebook

uv sync
jupyter notebook bifrost-poc.ipynb

Services

Bifrost (docker-compose.bifrost.yaml)

Container	Port	Purpose
bifrost	8080	AI Gateway — proxies requests to Bedrock
bifrost-redis	6380 (host) → 6379	Vector store for semantic cache

Langfuse (docker-compose.langfuse.yaml)

Container	Port	Purpose
langfuse-web	3000	Web UI + API
langfuse-worker	3030 (localhost)	Async background jobs
postgres	5432 (localhost)	Primary data store
clickhouse	8123 (localhost)	Analytics OLAP store
redis	6379 (localhost)	Queue / cache
minio	9090 / 9091 (localhost)	S3-compatible object storage

Bifrost Configuration

The gateway config lives in bifrost_configs/config.json:

Providers — Two Bedrock regions + an embedding key:

• Bedrock-EU → eu-west-1 (Nova Micro, Nova Lite, Titan Embed)
• Bedrock-US → us-east-1 (Nova Micro, Nova Lite)
• Bedrock-Embedding → eu-west-1 (Titan Embed)

Routing Rules:

• regional-router — 50/50 weighted split between EU and US for the benchmark virtual key
• embeddings-router — Routes embedding requests to the dedicated key

Plugins:

• semantic_cache — Caches similar prompts using Titan Embeddings + Redis
• otel — Exports traces to Langfuse via OpenTelemetry

Virtual Keys:

• sk-bf-my-benchmark-key — For chat/completion benchmarks
• sk-bf-my-embedding-key — For embedding requests

Makefile Commands

make up              # Start all services
make down            # Stop all services
make restart         # Restart all services
make logs            # Tail all logs
make logs-bifrost    # Tail Bifrost logs only
make logs-langfuse   # Tail Langfuse logs only
make ps              # Show running containers
make clean           # Stop and remove all volumes
make help            # Show available commands

Usage Examples

boto3 + LangChain (via Bedrock Converse API)

import boto3
from langchain_aws import ChatBedrockConverse

def add_bifrost_headers(request, **kwargs):
    request.headers.add_header("x-bf-vk", "sk-bf-my-benchmark-key")
    request.headers.add_header("x-bf-cache-key", "session-123")
    request.headers.add_header("x-bf-cache-type", "semantic")
    request.headers.add_header("x-bf-cache-ttl", "60s")
    request.headers.add_header("x-bf-cache-threshold", "0.2")

client = boto3.client(
    service_name="bedrock-runtime",
    endpoint_url="http://bifrost.local/bedrock",
)
client.meta.events.register_first("before-sign.bedrock-runtime.*", add_bifrost_headers)

llm = ChatBedrockConverse(client=client, model="amazon.nova-lite-v1:0", temperature=0)
response = llm.invoke("Explain AI Gateways")

Persistent Data

All stateful services use named Docker volumes:

Volume	Service	Survives down?	Removed by clean?
bifrost_redis_data	bifrost-redis	✅	✅
langfuse_postgres_data	postgres	✅	✅
langfuse_clickhouse_data	clickhouse	✅	✅
langfuse_minio_data	minio	✅	✅

Project Structure

.
├── bifrost_configs/
│   └── config.json              # Bifrost gateway configuration
├── docker-compose.bifrost.yaml  # Bifrost + Redis
├── docker-compose.langfuse.yaml # Langfuse full stack
├── .env.example                 # Template — copy to .env
├── .env                         # Your local config (git-ignored)
├── Makefile                     # Dev commands
├── bifrost-poc.ipynb            # Jupyter notebook with examples
├── main.py                      # Python entrypoint
└── pyproject.toml               # Python dependencies (uv)