LiteLLM + Langfuse — Local PoC

Local development stack that runs a LiteLLM proxy in front of Amazon Bedrock models, with Langfuse v3 for observability and tracing via OpenTelemetry.

Architecture

┌────────────┐  OTEL traces   ┌──────────────┐
│  LiteLLM   │───────────────▶│  Langfuse    │
│  Proxy     │                │  Web + Worker│
│  :4000     │                │  :30010      │
└─────┬──────┘                └──────┬───────┘
      │                              │
      ▼                              ▼
┌──────────┐                 ┌────────────┐  ┌───────┐  ┌────────────┐
│ LiteLLM  │                 │ Langfuse   │  │ Redis │  │ ClickHouse │
│ Postgres │                 │ Postgres   │  │       │  │            │
└──────────┘                 └────────────┘  └───────┘  └────────────┘
                                                              │
                                                         ┌────┴────┐
                                                         │  MinIO  │
                                                         └─────────┘

The stack is split into two Compose files joined by a shared Docker network (litellm-shared):

File	Services
docker-compose.langfuse.yaml	langfuse-web, langfuse-worker, langfuse-postgres, clickhouse, redis, minio
docker-compose.litellm.yaml	litellm, litellm-db (postgres)

Prerequisites

• Docker & Docker Compose v2+
• AWS credentials configured in ~/.aws (the LiteLLM container mounts this read-only)
• An AWS profile with access to Amazon Bedrock (default profile: fer-playground-admin)

Quick Start

1. Configure environment variables

All configuration lives in a single .env file at the project root. The defaults work out of the box for local development — just review and adjust as needed:

cp .env .env.backup   # optional: back up before editing

Key variables to review:

Variable	Purpose
LITELLM_MASTER_KEY	Admin API key for the LiteLLM proxy
AWS_PROFILE / AWS_REGION	AWS credentials for Bedrock access (set in compose)
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY	Shared API keys used by both Langfuse init and LiteLLM
LANGFUSE_INIT_USER_EMAIL / PASSWORD	Langfuse admin user created on first boot
NEXTAUTH_SECRET, SALT, ENCRYPTION_KEY	Langfuse auth secrets — change these in production

2. Start the stack

make run

This brings up all services in detached mode. Langfuse starts first since it creates the shared network.

3. Verify

• LiteLLM Proxy → http://localhost:4000
• LiteLLM Health → http://localhost:4000/health/liveliness
• Langfuse UI → http://localhost:30010

To log into the LiteLLM UI, use admin as the username and the value of LITELLM_MASTER_KEY (default: sk-1234) as the password.

If using OrbStack, Langfuse is also available at http://langfuse.litellm.local.

4. Test a request

curl -X 'POST' \
  'https://litellm.litellm-poc.orb.local/chat/completions' \
  -H 'accept: application/json' \
  -H 'x-litellm-api-key: sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "nova-pro-us",
  "messages": [
    {
      "role": "user",
      "content": "Hello, how are you?"
    }
  ],
  "mock_testing_fallbacks": true
}'

The trace should appear in the Langfuse UI within a few seconds.

Available Make Targets

Command	Description
make run	Start all services in detached mode
make down	Stop and remove all containers
make logs	Tail logs from all services

Project Structure

.
├── .env                            # Single source of truth for all env vars
├── docker-compose.langfuse.yaml    # Langfuse v3 stack (web, worker, infra)
├── docker-compose.litellm.yaml     # LiteLLM proxy, postgres
├── litellm_configs/
│   └── config.yaml                 # LiteLLM proxy config (models, routing, callbacks)
├── Makefile                        # Convenience targets
└── README.md

Models

Configured in litellm_configs/config.yaml:

Alias	Bedrock Model	Region
nova-lite	us.amazon.nova-lite-v1:0	us-east-1 (default)
nova-pro-us	us.amazon.nova-pro-v1:0	us-central-1
nova-pro-eu	eu.amazon.nova-pro-v1:0	eu-central-1

Routing uses simple-shuffle with fallback chains defined between the nova-pro-* and nova-lite models.

Notes

• The .env file uses variable interpolation to avoid duplication (e.g. LANGFUSE_INIT_PROJECT_PUBLIC_KEY references LANGFUSE_PUBLIC_KEY).
• Internal infrastructure services (ClickHouse, Redis, MinIO, each Postgres) are bound to 127.0.0.1 and isolated in their own Docker networks.
• Search for CHANGEME in the compose files to find all secrets that should be rotated before any non-local deployment.
• Complete guide in notion