Skip to content

06.4 Runtime Model Switching

Jump to bottom
Nikolay Vyahhi edited this page Feb 19, 2026 · 3 revisions

Runtime Model Switching

Relevant source files

The following files were used as context for generating this wiki page:

This page explains how users can dynamically switch LLM providers and models on a per-sender basis during channel conversations using slash commands. This feature allows different users in the same channel (e.g., Telegram, Discord) to use different models simultaneously without requiring daemon restarts or config changes.

For general channel configuration, see Channel Architecture. For provider configuration and model selection during onboarding, see Built-in Providers.

Purpose and Scope

Runtime model switching enables:

  • Per-sender routing: Each user in a channel can select their own provider and model
  • Zero-downtime switching: Change models without restarting zeroclaw daemon
  • Provider discovery: List available providers and cached model IDs
  • Session isolation: Model preferences persist per sender until explicitly changed or daemon restarts

This feature is channel-specific and only available in channels that support it (currently Telegram and Discord). CLI and gateway modes use the provider/model specified in config.toml or via command-line flags.

Supported Channels

Runtime model switching is enabled for specific channels via the supports_runtime_model_switch() function:

Channel Supported Notes
Telegram ✅ Yes Mention-only mode supported
Discord ✅ Yes Mention-only mode supported
Slack ❌ No Uses default config
Matrix ❌ No Uses default config
Email ❌ No Uses default config
WhatsApp ❌ No Uses default config
CLI ❌ No Use --model flag instead

Sources: src/channels/mod.rs:142-144

Command Syntax

/models - List Providers or Switch Provider

List all providers:

/models

Response shows current provider, current model, and a list of all available providers with their aliases.

Switch provider:

/models <provider-name>

Examples:

/models anthropic
/models openai
/models ollama
/models groq

Provider names are case-insensitive and support aliases (e.g., grok → xai, google → gemini).

/model - Show Current Model or Switch Model

Show current model:

/model

Response shows current provider, current model, and a preview of cached models for that provider (up to 10 entries).

Switch model:

/model <model-id>

Examples:

/model claude-sonnet-4-5-20250929
/model gpt-4o
/model llama3.2
/model anthropic/claude-opus-4-20250514

Model IDs can include backticks (they are stripped automatically). This allows users to copy-paste from the cached model list.

Telegram-Specific: Mention-Only Mode

In Telegram groups with mention_only = true, commands must mention the bot:

/models@botname anthropic
/model@botname gpt-4o

The @botname suffix is stripped automatically by the command parser.

Sources: src/channels/mod.rs:146-184

Architecture Overview

System Components

graph TB
    User["User<br/>(Telegram/Discord)"]
    Parse["parse_runtime_command()<br/>Lines 146-184"]
    Handle["handle_runtime_command_if_needed()<br/>Lines 365-441"]
    Resolve["resolve_provider_alias()<br/>Lines 186-205"]
    GetProvider["get_or_create_provider()<br/>Lines 266-308"]
    
    RouteMap["RouteSelectionMap<br/>Arc&lt;Mutex&lt;HashMap&lt;String, ChannelRouteSelection&gt;&gt;&gt;<br/>Lines 74-80"]
    ProviderCache["ProviderCacheMap<br/>Arc&lt;Mutex&lt;HashMap&lt;String, Arc&lt;dyn Provider&gt;&gt;&gt;&gt;<br/>Line 73"]
    
    ModelCache["models_cache.json<br/>workspace/state/<br/>Lines 243-264"]
    ProcessMsg["process_channel_message()<br/>Lines 556-814"]
    
    User --> Parse
    Parse --> Handle
    Handle --> Resolve
    Resolve --> GetProvider
    GetProvider --> ProviderCache
    Handle --> RouteMap
    
    Handle --> ModelCache
    
    ProcessMsg --> RouteMap
    ProcessMsg --> ProviderCache
    
    style RouteMap fill:#f9f9f9
    style ProviderCache fill:#f9f9f9
    style ModelCache fill:#f9f9f9
Loading

Sources: src/channels/mod.rs:73-123, src/channels/mod.rs:146-441, src/channels/mod.rs:556-814

Command Processing Flow

sequenceDiagram
    participant User
    participant parse_runtime_command
    participant handle_runtime_command_if_needed
    participant resolve_provider_alias
    participant get_or_create_provider
    participant set_route_selection
    participant clear_sender_history
    participant Channel
    
    User->>parse_runtime_command: "/models anthropic"
    parse_runtime_command->>parse_runtime_command: Strip @mention suffix
    parse_runtime_command->>parse_runtime_command: Parse command type
    parse_runtime_command-->>handle_runtime_command_if_needed: ChannelRuntimeCommand::SetProvider
    
    handle_runtime_command_if_needed->>resolve_provider_alias: "anthropic"
    resolve_provider_alias->>resolve_provider_alias: Check canonical names<br/>Check aliases
    resolve_provider_alias-->>handle_runtime_command_if_needed: "anthropic"
    
    handle_runtime_command_if_needed->>get_or_create_provider: provider_name, ctx
    get_or_create_provider->>get_or_create_provider: Check provider_cache
    alt Not in cache
        get_or_create_provider->>get_or_create_provider: create_resilient_provider_with_options()
        get_or_create_provider->>get_or_create_provider: provider.warmup()
        get_or_create_provider->>get_or_create_provider: Insert into cache
    end
    get_or_create_provider-->>handle_runtime_command_if_needed: Arc&lt;dyn Provider&gt;
    
    handle_runtime_command_if_needed->>set_route_selection: sender_key, new route
    set_route_selection->>set_route_selection: Update route_overrides map
    
    handle_runtime_command_if_needed->>clear_sender_history: sender_key
    clear_sender_history->>clear_sender_history: Remove from conversation_histories
    
    handle_runtime_command_if_needed->>Channel: Send success response
    Channel->>User: "Provider switched to `anthropic`..."
Loading

Sources: src/channels/mod.rs:146-184, src/channels/mod.rs:186-205, src/channels/mod.rs:266-308, src/channels/mod.rs:223-234, src/channels/mod.rs:236-241, src/channels/mod.rs:365-441

Data Structures

ChannelRouteSelection

Stores the provider and model selection for a specific sender:

struct ChannelRouteSelection {
    provider: String,  // e.g., "anthropic", "openai"
    model: String,     // e.g., "claude-sonnet-4-5-20250929"
}

Sources: src/channels/mod.rs:76-80

ChannelRuntimeCommand

Enum representing parsed slash commands:

enum ChannelRuntimeCommand {
    ShowProviders,           // /models
    SetProvider(String),     // /models <provider>
    ShowModel,               // /model
    SetModel(String),        // /model <model-id>
}

Sources: src/channels/mod.rs:82-88

ChannelRuntimeContext

Runtime context shared across all channel message processing:

Field Type Purpose
provider Arc<dyn Provider> Default provider from config
default_provider Arc<String> Default provider name
model Arc<String> Default model name
provider_cache ProviderCacheMap Cache of provider instances
route_overrides RouteSelectionMap Per-sender routing overrides
conversation_histories ConversationHistoryMap Per-sender chat history

Sources: src/channels/mod.rs:101-123

Provider Caching

Cache Key Structure

Providers are cached by name in a thread-safe HashMap:

provider_cache: Arc<Mutex<HashMap<String, Arc<dyn Provider>>>>

Cache Behavior

  1. Default provider: Always cached at daemon startup (from config.toml)
  2. Runtime providers: Cached on first use via /models <provider> command
  3. Warmup: Each provider calls provider.warmup() after creation to initialize connection pools
  4. Reuse: Subsequent /models <provider> commands for the same provider reuse the cached instance
  5. Lifetime: Cache persists until daemon restarts

get_or_create_provider() Logic

flowchart TD
    Start["get_or_create_provider(ctx, provider_name)"]
    CheckDefault{"provider_name ==<br/>default_provider?"}
    CheckCache{"In provider_cache?"}
    CreateProvider["create_resilient_provider_with_options()"]
    Warmup["provider.warmup()"]
    InsertCache["Insert into provider_cache"]
    Return["Return Arc&lt;dyn Provider&gt;"]
    
    Start --> CheckDefault
    CheckDefault -->|Yes| Return
    CheckDefault -->|No| CheckCache
    CheckCache -->|Yes| Return
    CheckCache -->|No| CreateProvider
    CreateProvider --> Warmup
    Warmup --> InsertCache
    InsertCache --> Return
Loading

Sources: src/channels/mod.rs:266-308

Model Cache System

Purpose

The model cache stores lists of available models per provider to avoid repeated API calls during /model command usage. This is especially useful for providers with large model catalogs (e.g., OpenRouter has 200+ models).

Cache File Location

~/.zeroclaw/workspace/state/models_cache.json

Cache Structure

{
  "entries": [
    {
      "provider": "openrouter",
      "models": [
        "anthropic/claude-sonnet-4.6",
        "openai/gpt-5.2",
        "google/gemini-3-pro-preview",
        ...
      ]
    },
    {
      "provider": "anthropic",
      "models": [
        "claude-sonnet-4-5-20250929",
        "claude-opus-4-6",
        "claude-haiku-4-5-20251001"
      ]
    }
  ]
}

Sources: src/channels/mod.rs:90-99

Cache Population

The cache is populated by:

  1. zeroclaw models refresh --provider <provider>: CLI command to fetch and cache models
  2. Onboarding wizard: Optionally fetches models during provider setup

The cache is not automatically populated by /model commands. Users see a message prompting them to run zeroclaw models refresh if no cache exists.

Cache Preview

The /model command shows up to 10 cached models by default:

Current provider: `anthropic`
Current model: `claude-sonnet-4-5-20250929`

Switch model with `/model <model-id>`.

Cached model IDs (top 10):
- `claude-sonnet-4-5-20250929`
- `claude-opus-4-6`
- `claude-haiku-4-5-20251001`
- ...

Sources: src/channels/mod.rs:70-71, src/channels/mod.rs:243-264, src/channels/mod.rs:310-338

Per-Sender Routing

Route Key Format

Route overrides are keyed by conversation_history_key():

fn conversation_history_key(msg: &traits::ChannelMessage) -> String {
    format!("{}_{}", msg.channel, msg.sender)
}

Example keys:

  • telegram_123456789
  • discord_987654321

Sources: src/channels/mod.rs:129-131

Route Selection Logic

flowchart TD
    GetRoute["get_route_selection(ctx, sender_key)"]
    CheckMap{"sender_key in<br/>route_overrides?"}
    UseOverride["Return ChannelRouteSelection<br/>from map"]
    UseDefault["Return default_route_selection(ctx)"]
    
    GetRoute --> CheckMap
    CheckMap -->|Yes| UseOverride
    CheckMap -->|No| UseDefault
    
    UseDefault --> DefaultProvider["provider: ctx.default_provider"]
    UseDefault --> DefaultModel["model: ctx.model"]
Loading

When a user switches provider or model:

  1. set_route_selection() updates the route_overrides map for that sender
  2. If the new route equals the default route, the entry is removed (optimization)
  3. Subsequent messages from that sender use get_route_selection() to retrieve their route

Sources: src/channels/mod.rs:207-234

Message Processing with Routing

In process_channel_message(), the route is resolved before LLM calls:

let history_key = conversation_history_key(&msg);
let route = get_route_selection(ctx.as_ref(), &history_key);
let active_provider = get_or_create_provider(ctx.as_ref(), &route.provider).await?;

// Use route.provider and route.model for this sender
run_tool_call_loop(
    active_provider.as_ref(),
    &mut history,
    /* ... */
    route.provider.as_str(),
    route.model.as_str(),
    /* ... */
)

Sources: src/channels/mod.rs:569-586

History Management

Clearing History on Model Switch

When a user switches provider or model, their conversation history is cleared to prevent context contamination:

// After successful provider switch
clear_sender_history(ctx, &sender_key);

// After successful model switch
clear_sender_history(ctx, &sender_key);

This is necessary because:

  1. Different models may have different context window sizes
  2. Different providers may have different system prompt requirements
  3. Tool calling formats may vary between providers

The user's routing preference is preserved, but their chat history starts fresh.

Sources: src/channels/mod.rs:236-241, src/channels/mod.rs:390, src/channels/mod.rs:420

Provider Alias Resolution

Supported Aliases

The resolve_provider_alias() function normalizes provider names and handles common aliases:

Input Resolved To
grok xai
together together-ai
google gemini
google-gemini gemini
nvidia-nim nvidia
build.nvidia.com nvidia

Provider names and aliases are matched case-insensitively.

Resolution Flow

flowchart TD
    Input["resolve_provider_alias(name)"]
    Trim["Trim and validate"]
    CheckEmpty{"name.is_empty()?"}
    ListProviders["providers::list_providers()"]
    SearchExact{"Exact match in<br/>provider.name?"}
    SearchAlias{"Match in<br/>provider.aliases?"}
    ReturnName["Return provider.name"]
    ReturnNone["Return None"]
    
    Input --> Trim
    Trim --> CheckEmpty
    CheckEmpty -->|Yes| ReturnNone
    CheckEmpty -->|No| ListProviders
    ListProviders --> SearchExact
    SearchExact -->|Yes| ReturnName
    SearchExact -->|No| SearchAlias
    SearchAlias -->|Yes| ReturnName
    SearchAlias -->|No| ReturnNone
Loading

Sources: src/channels/mod.rs:186-205

Error Handling

Invalid Provider

If a provider name cannot be resolved:

Unknown provider `<raw_provider>`. Use `/models` to list valid providers.

Provider Initialization Failure

If provider creation fails (e.g., invalid API key):

Failed to initialize provider `<provider_name>`. Route unchanged.
Details: <sanitized_error>

API keys and sensitive data are stripped from error messages via providers::sanitize_api_error().

Empty Model ID

If user sends /model with no argument or empty string:

Model ID cannot be empty. Use `/model <model-id>`.

Sources: src/channels/mod.rs:398-407, src/channels/mod.rs:573-585, src/channels/mod.rs:413-416

Configuration Impact

Channel-Level Settings

Runtime model switching respects channel-specific settings:

Setting Impact on Runtime Switching
mention_only Commands require @botname suffix
allowed_users Only allowlisted users can switch models
stream_mode Applies to all providers (if channel supports it)

Provider-Level Settings

Runtime-switched providers inherit settings from config.toml:

Setting Inherited Behavior
reliability.provider_retries Applied to all providers
reliability.provider_backoff_ms Applied to all providers
reliability.api_keys Key rotation applies
api_url Only used for default provider

Custom api_url overrides (e.g., remote Ollama) do not apply to runtime-switched providers—they use standard endpoints.

Sources: src/channels/mod.rs:284-296

Limitations

  1. Channel support: Only Telegram and Discord support runtime switching
  2. No custom endpoints: Runtime-switched providers use default API endpoints (cannot override api_url)
  3. No config persistence: Route preferences reset on daemon restart
  4. No model validation: Users can specify any model ID—validation happens at LLM API call time
  5. Cache-only model discovery: /model shows cached models only; does not fetch live model lists

Sources: src/channels/mod.rs:142-144, src/channels/mod.rs:266-308

Example Usage

Scenario: Switching from OpenRouter to Anthropic

User (Telegram):

/models anthropic

Bot response:

Provider switched to `anthropic` for this sender session. Current model is `claude-sonnet-4-5-20250929`.
Use `/model <model-id>` to set a provider-compatible model.

User:

/model claude-opus-4-6

Bot response:

Model switched to `claude-opus-4-6` for provider `anthropic` in this sender session.

Now all subsequent messages from this user use Anthropic's Claude Opus 4.6, while other users in the same channel continue using their own provider/model selections.

Integration with Agent Core

Runtime model switching affects the agent turn cycle at the provider and model level:

  1. Provider resolution: get_route_selection() determines which provider to use
  2. Provider instantiation: get_or_create_provider() fetches or creates the provider
  3. Model selection: route.model is passed to run_tool_call_loop()
  4. Tool execution: Tools execute regardless of provider/model (provider-agnostic)
  5. History storage: Per-sender history is maintained separately per route

The agent core (see Agent Turn Cycle) is provider-agnostic—it delegates model execution to the selected provider via the Provider trait.

Sources: src/channels/mod.rs:698-714

Home

  1. Overview

  2. Getting Started

  3. Core Architecture

  4. Configuration

  5. Providers

  6. Channels

  7. Agent Core

  8. Tools

  9. Memory System

  10. Gateway

  11. Scheduler

  12. Development Guide

  13. Deployment

  14. Maturity Framework

Clone this wiki locally
⚡