Skip to content

09.1 Memory Backends

Jump to bottom
Nikolay Vyahhi edited this page Feb 19, 2026 · 3 revisions

Memory Backends

Relevant source files

The following files were used as context for generating this wiki page:

This page documents the pluggable memory backend system in ZeroClaw. It covers the available backend implementations (SQLite, PostgreSQL, Lucid, Markdown, None), their configuration options, and guidelines for selecting the appropriate backend for your deployment.

For information about hybrid search architecture and vector/keyword retrieval, see Hybrid Search. For memory snapshot and export functionality, see Memory Snapshot.

Overview

ZeroClaw implements memory as a trait-based pluggable system. The Memory trait defines the interface for storing, recalling, and managing conversational context. Different backend implementations provide varying trade-offs between performance, features, deployment complexity, and resource usage.

Key Concepts:

  • Backend: The underlying storage mechanism (SQLite, PostgreSQL, Lucid, Markdown, or None)
  • Auto-save: Automatic persistence of conversation turns to memory
  • Hybrid Search: Combined vector similarity + keyword search (SQLite only)
  • Hygiene: Automatic archival and purging of old memory entries (SQLite only)

The memory system is initialized during configuration loading and injected into the agent/channel runtime context as Arc<dyn Memory>.

Sources: src/channels/mod.rs:36, src/config/schema.rs:99-103

Memory Trait Architecture

graph TB
    subgraph "Memory Trait Interface"
        MemoryTrait["Memory trait<br/>(store, recall, forget, list)"]
    end
    
    subgraph "Backend Implementations"
        SQLiteBackend["SqliteMemory<br/>(hybrid search, hygiene,<br/>embedding cache)"]
        PostgresBackend["PostgresMemory<br/>(remote storage)"]
        LucidBackend["LucidMemory<br/>(external CLI bridge)"]
        MarkdownBackend["MarkdownMemory<br/>(file-based)"]
        NoneBackend["NoopMemory<br/>(no persistence)"]
    end
    
    subgraph "Runtime Context"
        ChannelContext["ChannelRuntimeContext"]
        AgentLoop["Agent Turn Loop"]
    end
    
    subgraph "Configuration"
        MemoryConfig["[memory] in config.toml<br/>backend = 'sqlite' | 'postgres' | 'lucid' | 'markdown' | 'none'"]
        StorageConfig["[storage.provider.config]<br/>(PostgreSQL connection)"]
    end
    
    MemoryConfig --> MemoryTrait
    StorageConfig --> PostgresBackend
    
    MemoryTrait --> SQLiteBackend
    MemoryTrait --> PostgresBackend
    MemoryTrait --> LucidBackend
    MemoryTrait --> MarkdownBackend
    MemoryTrait --> NoneBackend
    
    SQLiteBackend --> ChannelContext
    PostgresBackend --> ChannelContext
    LucidBackend --> ChannelContext
    MarkdownBackend --> ChannelContext
    NoneBackend --> ChannelContext
    
    ChannelContext --> AgentLoop
    AgentLoop --> |"recall(query)"| ChannelContext
    AgentLoop --> |"store(key, content)"| ChannelContext
Loading

Diagram: Memory Backend Architecture

The Memory trait provides a uniform interface. Configuration determines which concrete implementation is instantiated at runtime. The agent and channel systems interact only with the trait interface, enabling zero-code backend swapping.

Sources: src/channels/mod.rs:36, src/channels/mod.rs:106, src/config/schema.rs:99-103

Backend Implementations

SQLite Backend

Backend Key: sqlite

The default and most feature-complete backend. Provides full-stack search capabilities with no external dependencies.

Features:

  • Hybrid search: Vector (cosine similarity) + keyword (FTS5 BM25) search
  • Embedding cache: Caches embeddings with LRU eviction to reduce API calls
  • Hygiene: Automatic archival (archive_after_days) and purging (purge_after_days)
  • Response cache: Optional caching of LLM responses with TTL
  • Snapshot support: Export/import memory state for backup and recovery
  • Transactional: ACID guarantees for all operations

Storage Location: ~/.zeroclaw/workspace/memory.db

Use Cases:

  • Default choice for most deployments
  • Single-node agents where local disk persistence is acceptable
  • Scenarios requiring full-text and vector search without external services

Configuration Example:

[memory]
backend = "sqlite"
auto_save = true
hygiene_enabled = true
archive_after_days = 7
purge_after_days = 30
embedding_provider = "openai"  # or "none", "custom:https://..."
embedding_model = "text-embedding-3-small"
embedding_dimensions = 1536
vector_weight = 0.7
keyword_weight = 0.3
embedding_cache_size = 10000
response_cache_enabled = false
response_cache_ttl_minutes = 60
sqlite_open_timeout_secs = 30  # optional timeout for locked DB

Sources: src/onboard/wizard.rs:263-298, src/config/schema.rs:99-103

PostgreSQL Backend

Backend Key: postgres

Remote database backend for multi-agent deployments requiring shared memory state.

Features:

  • Remote storage: Centralized memory across multiple agent instances
  • Schema isolation: Configurable schema and table names
  • Connection pooling: Built-in connection management
  • JSONB storage: Native PostgreSQL JSON support for flexible memory entries

Limitations:

  • No built-in vector search (requires PostgreSQL extensions like pgvector)
  • No FTS5-style keyword search (requires PostgreSQL extensions like pg_trgm)
  • Hygiene and response caching are not implemented

Use Cases:

  • Multi-agent systems with shared memory requirements
  • Cloud deployments where local disk is ephemeral
  • Scenarios requiring centralized audit/compliance logs

Configuration Example:

[memory]
backend = "postgres"
auto_save = true

[storage.provider.config]
provider = "postgres"
db_url = "postgres://user:password@host:5432/zeroclaw"
schema = "public"
table = "memories"
connect_timeout_secs = 15

Aliases: The db_url key also accepts dbURL for backward compatibility.

Sources: src/config/schema.rs:99-103, README.md:355-365

Lucid Backend

Backend Key: lucid

Bridge to an external lucid CLI binary for memory operations. Designed for integration with Lucid AI's memory service.

Features:

  • External CLI bridge: Delegates memory operations to lucid command
  • Local hit threshold: Caches frequently accessed keys locally before querying external service
  • Timeout controls: Separate timeouts for recall (low-latency) and store (async sync)
  • Failure cooldown: Backs off after failures to avoid repeated slow attempts

Limitations:

  • Requires lucid binary in PATH (or set via ZEROCLAW_LUCID_CMD)
  • Network/CLI latency overhead
  • No built-in vector search or hygiene

Use Cases:

  • Integration with Lucid AI's memory infrastructure
  • Scenarios requiring external memory governance or compliance
  • Hybrid deployments where Lucid manages long-term memory

Configuration Example:

[memory]
backend = "lucid"
auto_save = true

Environment Variables:

Variable Default Description
ZEROCLAW_LUCID_CMD lucid Path to lucid binary
ZEROCLAW_LUCID_BUDGET 200 Token budget for context recall
ZEROCLAW_LUCID_LOCAL_HIT_THRESHOLD 3 Local hit count to skip external recall
ZEROCLAW_LUCID_RECALL_TIMEOUT_MS 120 Low-latency recall timeout
ZEROCLAW_LUCID_STORE_TIMEOUT_MS 800 Async store timeout
ZEROCLAW_LUCID_FAILURE_COOLDOWN_MS 15000 Cooldown after failure

Sources: README.md:370-377

Markdown Backend

Backend Key: markdown

File-based backend storing each memory entry as a separate .md file in the workspace.

Features:

  • Human-readable: Plain markdown files in ~/.zeroclaw/workspace/memory/
  • Git-friendly: Easy to version control and inspect
  • No dependencies: No database setup required

Limitations:

  • No search capabilities (recall returns empty results)
  • No hygiene or caching
  • Poor performance at scale (O(n) file scans)
  • No transactional guarantees

Use Cases:

  • Debugging and inspection workflows
  • Simple single-agent setups where search is not required
  • Scenarios prioritizing auditability over performance

Configuration Example:

[memory]
backend = "markdown"
auto_save = true

Storage Location: ~/.zeroclaw/workspace/memory/*.md

Sources: src/onboard/wizard.rs:263-298

None Backend

Backend Key: none

Explicit no-op backend that disables memory persistence entirely.

Features:

  • Zero overhead: No disk I/O or database operations
  • Stateless: Each agent turn starts with empty context
  • Deterministic: No side effects from prior conversations

Limitations:

  • No memory persistence or recall
  • Auto-save is ignored

Use Cases:

  • Stateless API endpoints where context is passed explicitly
  • Testing and benchmarking scenarios requiring deterministic behavior
  • Privacy-sensitive deployments where conversation history must not persist

Configuration Example:

[memory]
backend = "none"
auto_save = false  # ignored, but semantically correct

Sources: src/onboard/wizard.rs:263-298, README.md:354-355

Backend Comparison

Backend Search Hygiene Remote Setup Complexity Performance Use Case
SQLite Hybrid (vector + keyword) Yes No Low (bundled) High (local) Default for most deployments
PostgreSQL No (requires extensions) No Yes Medium (DB setup) Medium (network) Multi-agent, shared memory
Lucid External No Yes Medium (CLI binary) Medium (network) Lucid AI integration
Markdown No No No Low (filesystem) Low (O(n) scans) Debugging, inspection
None No No No None N/A (no-op) Stateless APIs, testing

Sources: src/onboard/wizard.rs:263-298

Configuration Flow

sequenceDiagram
    participant User
    participant Wizard as "Onboarding Wizard"
    participant ConfigLoader as "Config::load_or_init"
    participant MemoryFactory as "memory::create_backend"
    participant Backend as "Concrete Backend<br/>(e.g., SqliteMemory)"
    participant Runtime as "ChannelRuntimeContext"
    
    User->>Wizard: zeroclaw onboard --interactive
    Wizard->>Wizard: setup_memory() prompts user
    Wizard->>ConfigLoader: Save memory.backend = "sqlite"
    
    User->>Runtime: zeroclaw daemon
    Runtime->>ConfigLoader: Config::load_or_init()
    ConfigLoader->>MemoryFactory: create_backend(config.memory)
    
    alt backend = "sqlite"
        MemoryFactory->>Backend: SqliteMemory::new(workspace/memory.db)
        Backend->>Backend: Initialize FTS5, embedding cache
    else backend = "postgres"
        MemoryFactory->>Backend: PostgresMemory::new(storage.provider.config)
        Backend->>Backend: Connect to remote DB
    else backend = "lucid"
        MemoryFactory->>Backend: LucidMemory::new()
        Backend->>Backend: Verify lucid CLI in PATH
    else backend = "markdown"
        MemoryFactory->>Backend: MarkdownMemory::new(workspace/memory/)
        Backend->>Backend: Ensure directory exists
    else backend = "none"
        MemoryFactory->>Backend: NoopMemory::new()
    end
    
    MemoryFactory-->>Runtime: Arc<dyn Memory>
    Runtime->>Runtime: Store in ChannelRuntimeContext
Loading

Diagram: Memory Backend Initialization Flow

The wizard collects user preferences for the memory backend. During runtime initialization, the configuration loader invokes the memory factory to instantiate the appropriate backend. The factory returns an Arc<dyn Memory> trait object, which is stored in the runtime context and shared across all agent/channel operations.

Sources: src/onboard/wizard.rs:94, src/channels/mod.rs:106

Runtime Usage Patterns

Memory Context Enrichment

Before each LLM call, the agent recalls relevant context from memory to enrich the user's message.

Code Flow:

  1. build_memory_context: src/channels/mod.rs:443-469

    • Calls memory.recall(user_msg, 5, None) to retrieve top 5 relevant entries
    • Filters entries by min_relevance_score threshold
    • Formats results as [Memory context]\n- key: content\n\n

  2. process_channel_message: src/channels/mod.rs:589

    • Invokes build_memory_context with user message
    • Prepends memory context to user message before LLM call

Example:

// [src/channels/mod.rs:443-469]
async fn build_memory_context(
    mem: &dyn Memory,
    user_msg: &str,
    min_relevance_score: f64,
) -> String {
    let mut context = String::new();
    if let Ok(entries) = mem.recall(user_msg, 5, None).await {
        let relevant: Vec<_> = entries
            .iter()
            .filter(|e| match e.score {
                Some(score) => score >= min_relevance_score,
                None => true,
            })
            .collect();
        if !relevant.is_empty() {
            context.push_str("[Memory context]\n");
            for entry in &relevant {
                let _ = writeln!(context, "- {}: {}", entry.key, entry.content);
            }
            context.push('\n');
        }
    }
    context
}

Sources: src/channels/mod.rs:443-469, src/channels/mod.rs:589

Auto-Save

When auto_save = true, each channel message is automatically persisted to memory.

Code Flow:

  1. process_channel_message: src/channels/mod.rs:591-602
    • Checks ctx.auto_save_memory flag
    • Generates memory key: conversation_memory_key(&msg) → {channel}_{sender}_{message_id}
    • Calls memory.store(key, content, Conversation, None)

Example:

// [src/channels/mod.rs:591-602]
if ctx.auto_save_memory {
    let autosave_key = conversation_memory_key(&msg);
    let _ = ctx
        .memory
        .store(
            &autosave_key,
            &msg.content,
            crate::memory::MemoryCategory::Conversation,
            None,
        )
        .await;
}

Key Format: {channel}_{sender}_{message_id}

Sources: src/channels/mod.rs:591-602, src/channels/mod.rs:125-127

Backend-Specific Behavior

graph TB
    subgraph "Agent Turn"
        UserMsg["User Message"]
        RecallCtx["Recall Context<br/>(build_memory_context)"]
        LLMCall["LLM Call<br/>(enriched prompt)"]
        AutoSave["Auto-Save<br/>(if enabled)"]
    end
    
    subgraph "SQLite Backend Behavior"
        SQLiteRecall["FTS5 + Vector Search<br/>(hybrid merge)"]
        SQLiteStore["Insert into memories table<br/>(trigger FTS5 + embedding)"]
        SQLiteHygiene["Background hygiene<br/>(archive/purge old entries)"]
    end
    
    subgraph "PostgreSQL Backend Behavior"
        PGRecall["SELECT with JSONB filter<br/>(no vector search)"]
        PGStore["INSERT into remote table<br/>(JSONB payload)"]
    end
    
    subgraph "Lucid Backend Behavior"
        LucidRecall["lucid recall --budget=200<br/>(CLI subprocess)"]
        LucidStore["lucid store<br/>(async timeout)"]
        LucidCooldown["Failure cooldown<br/>(15s after error)"]
    end
    
    subgraph "Markdown Backend Behavior"
        MarkdownRecall["Return empty results<br/>(no search)"]
        MarkdownStore["Write {key}.md file<br/>(append mode)"]
    end
    
    subgraph "None Backend Behavior"
        NoneRecall["Return empty immediately"]
        NoneStore["No-op (discard)"]
    end
    
    UserMsg --> RecallCtx
    RecallCtx --> |"backend = 'sqlite'"| SQLiteRecall
    RecallCtx --> |"backend = 'postgres'"| PGRecall
    RecallCtx --> |"backend = 'lucid'"| LucidRecall
    RecallCtx --> |"backend = 'markdown'"| MarkdownRecall
    RecallCtx --> |"backend = 'none'"| NoneRecall
    
    RecallCtx --> LLMCall
    LLMCall --> AutoSave
    
    AutoSave --> |"backend = 'sqlite'"| SQLiteStore
    AutoSave --> |"backend = 'postgres'"| PGStore
    AutoSave --> |"backend = 'lucid'"| LucidStore
    AutoSave --> |"backend = 'markdown'"| MarkdownStore
    AutoSave --> |"backend = 'none'"| NoneStore
    
    SQLiteStore --> SQLiteHygiene
    LucidStore --> LucidCooldown
Loading

Diagram: Backend-Specific Runtime Behavior

Each backend implements recall and store with different strategies. SQLite performs hybrid search and triggers hygiene. PostgreSQL uses JSONB queries. Lucid delegates to an external CLI. Markdown writes files. None is a complete no-op.

Sources: src/channels/mod.rs:443-469, src/channels/mod.rs:591-602

Backend Selection Guidelines

Choose SQLite if:

  • You need vector + keyword hybrid search
  • Your deployment is single-node (local disk persistence is acceptable)
  • You want automatic hygiene (archival/purging)
  • You require snapshot/export functionality

Choose PostgreSQL if:

  • You need shared memory across multiple agent instances
  • Your deployment is cloud-native with ephemeral local disk
  • You have an existing PostgreSQL infrastructure
  • You can add extensions for vector search (e.g., pgvector)

Choose Lucid if:

  • You are integrating with Lucid AI's memory service
  • You need external memory governance or compliance
  • You want centralized memory management across multiple systems

Choose Markdown if:

  • You need human-readable memory inspection
  • Your use case is single-agent with low message volume
  • You prioritize git versioning and auditability over search performance
  • You are debugging memory issues and need file-level visibility

Choose None if:

  • Your agent is stateless (context passed explicitly per request)
  • You are testing deterministic behavior without side effects
  • Privacy requirements prohibit conversation history persistence
  • You are benchmarking and want to eliminate memory I/O overhead

Sources: src/onboard/wizard.rs:263-298, README.md:330-365

Default Configuration

The wizard and quick setup use backend-specific defaults optimized for each backend's capabilities.

Function: src/onboard/wizard.rs:268-298

fn memory_config_defaults_for_backend(backend: &str) -> MemoryConfig {
    let profile = memory_backend_profile(backend);
    
    MemoryConfig {
        backend: backend.to_string(),
        auto_save: profile.auto_save_default,
        hygiene_enabled: profile.uses_sqlite_hygiene,
        archive_after_days: if profile.uses_sqlite_hygiene { 7 } else { 0 },
        purge_after_days: if profile.uses_sqlite_hygiene { 30 } else { 0 },
        conversation_retention_days: 30,
        embedding_provider: "none".to_string(),
        embedding_model: "text-embedding-3-small".to_string(),
        embedding_dimensions: 1536,
        vector_weight: 0.7,
        keyword_weight: 0.3,
        min_relevance_score: 0.4,
        embedding_cache_size: if profile.uses_sqlite_hygiene { 10000 } else { 0 },
        chunk_max_tokens: 512,
        response_cache_enabled: false,
        response_cache_ttl_minutes: 60,
        response_cache_max_entries: 5_000,
        snapshot_enabled: false,
        snapshot_on_hygiene: false,
        auto_hydrate: true,
        sqlite_open_timeout_secs: None,
    }
}

Backend Profiles:

Backend auto_save hygiene_enabled embedding_cache_size
sqlite true true 10000
postgres true false 0
lucid true false 0
markdown true false 0
none false false 0

Sources: src/onboard/wizard.rs:268-298

Environment Variable Overrides

PostgreSQL backend supports environment variable overrides for connection strings:

Variable Description
DATABASE_URL PostgreSQL connection string
ZEROCLAW_DB_URL ZeroClaw-specific override
ZEROCLAW_POSTGRES_URL PostgreSQL-specific override

Priority: ZEROCLAW_POSTGRES_URL > ZEROCLAW_DB_URL > DATABASE_URL > config.toml

Sources: README.md:355-365

Summary

ZeroClaw's memory system provides five backend implementations, each optimized for different deployment scenarios:

  1. SQLite — Default choice with hybrid search, hygiene, and zero external dependencies
  2. PostgreSQL — Remote storage for multi-agent deployments requiring shared memory
  3. Lucid — External CLI bridge for integration with Lucid AI's memory service
  4. Markdown — File-based backend for human-readable inspection and git versioning
  5. None — Explicit no-op for stateless APIs and deterministic testing

All backends implement the Memory trait, enabling configuration-driven swapping with zero code changes. The runtime context stores Arc<dyn Memory>, providing uniform access across agent and channel subsystems.

Sources: src/channels/mod.rs:36, src/channels/mod.rs:106, src/config/schema.rs:99-103, src/onboard/wizard.rs:263-298

Home

  1. Overview

  2. Getting Started

  3. Core Architecture

  4. Configuration

  5. Providers

  6. Channels

  7. Agent Core

  8. Tools

  9. Memory System

  10. Gateway

  11. Scheduler

  12. Development Guide

  13. Deployment

  14. Maturity Framework

Clone this wiki locally
⚡