09.3 Memory Snapshot

Memory Snapshot

Relevant source files

The following files were used as context for generating this wiki page:

Purpose and Scope

Memory Snapshot is a backup and recovery system that exports SQLite-backed agent memories to a human-readable Markdown file (MEMORY_SNAPSHOT.md) and automatically restores them when the database is lost. This provides Git-visible persistence for the agent's "soul" — its core identity, preferences, and long-term knowledge.

This document covers the export/import mechanism, Markdown format specification, auto-hydration logic, and integration with the SQLite memory backend. For information about the broader memory system architecture and trait-based backends, see Memory System. For backend-specific configuration and hybrid search, see Memory Backends and Hybrid Search.

System Architecture

The snapshot system bridges SQLite persistence with human-readable Markdown files, enabling version control and manual inspection of agent memories while providing automatic disaster recovery.

Memory Snapshot Architecture

graph TB
    subgraph "Workspace Directory"
        Brain["brain.db<br/>(SQLite)"]
        Snapshot["MEMORY_SNAPSHOT.md<br/>(Markdown)"]
    end
    
    subgraph "Snapshot Module"
        Export["export_snapshot()"]
        Hydrate["hydrate_from_snapshot()"]
        ShouldHydrate["should_hydrate()"]
        Parse["parse_snapshot()"]
    end
    
    subgraph "Memory Schema"
        MemTable["memories table<br/>(id, key, content, category,<br/>created_at, updated_at)"]
        FTS["memories_fts<br/>(FTS5 index)"]
    end
    
    Agent[Agent Core] -->|"store() with<br/>category=core"| MemTable
    
    Export -->|"SELECT WHERE<br/>category='core'"| MemTable
    Export -->|"write Markdown"| Snapshot
    
    ShouldHydrate -->|"check existence"| Brain
    ShouldHydrate -->|"check existence"| Snapshot
    
    Hydrate -->|"read Markdown"| Snapshot
    Hydrate -->|"parse entries"| Parse
    Hydrate -->|"INSERT INTO<br/>memories + FTS"| MemTable
    Hydrate -->|"populate"| FTS
    
    ColdBoot[Cold Boot] -->|"brain.db missing?"| ShouldHydrate
    ShouldHydrate -->|"true"| Hydrate

Sources: src/memory/snapshot.rs:1-261

File Locations and Storage Layout

The snapshot system operates on a fixed directory structure within each workspace:

File Path	Purpose	Format	Persistence
`<workspace>/memory/brain.db`	SQLite database	Binary (SQLite WAL mode)	Ephemeral (can be regenerated)
`<workspace>/MEMORY_SNAPSHOT.md`	Markdown snapshot	UTF-8 text	Persistent (Git-tracked)
`<workspace>/memory/brain.db-wal`	SQLite write-ahead log	Binary	Ephemeral
`<workspace>/memory/brain.db-shm`	SQLite shared memory	Binary	Ephemeral

The snapshot file lives at workspace root (not in the memory/ subdirectory) to ensure maximum Git visibility. The brain.db file lives inside memory/ alongside other memory backend files.

Sources: src/memory/snapshot.rs:17-18, src/memory/snapshot.rs:29-30, src/memory/snapshot.rs:202-204

Export Process

The export_snapshot() function queries the SQLite database for all memories with category = 'core' and serializes them to Markdown.

Export Workflow

sequenceDiagram
    participant Caller
    participant export_snapshot
    participant SQLite as brain.db
    participant File as MEMORY_SNAPSHOT.md
    
    Caller->>export_snapshot: export_snapshot(workspace_dir)
    
    export_snapshot->>SQLite: Check existence
    alt brain.db missing
        export_snapshot-->>Caller: Ok(0) - skip
    end
    
    export_snapshot->>SQLite: PRAGMA journal_mode = WAL
    export_snapshot->>SQLite: SELECT key, content, category,<br/>created_at, updated_at<br/>WHERE category='core'<br/>ORDER BY updated_at DESC
    SQLite-->>export_snapshot: Vec<(key, content, ...)>
    
    alt No core memories
        export_snapshot-->>Caller: Ok(0) - nothing to export
    end
    
    export_snapshot->>export_snapshot: Build Markdown<br/>(header + entries)
    export_snapshot->>File: fs::write(MEMORY_SNAPSHOT.md)
    
    export_snapshot->>Caller: Ok(count)
    Note over Caller: Logs: "📸 Memory snapshot exported:<br/>N core memories"

Sources: src/memory/snapshot.rs:28-90

Query and Filtering

The export query explicitly filters for core memories:

SELECT key, content, category, created_at, updated_at
FROM memories
WHERE category = 'core'
ORDER BY updated_at DESC

This ensures that only long-term memories (identity, preferences, rules) are persisted to the snapshot, while ephemeral conversation history remains database-only. The ordering by updated_at DESC places the most recently modified memories at the top of the file for easier manual inspection.

Sources: src/memory/snapshot.rs:38-43

Export Triggers

Export is not automatic. It must be explicitly triggered by:

Manual invocation via the memory management tools
Scheduled exports via cron jobs
Shutdown hooks (if implemented by the daemon)

The typical pattern is to export after significant core memory updates (e.g., after the agent learns a new preference or updates its identity).

Markdown Format Specification

The snapshot file uses a structured Markdown format with three levels of organization:

File Structure

# 🧠 ZeroClaw Memory Snapshot

> Auto-generated by ZeroClaw. Do not edit manually unless you know what you're doing.
> This file is the "soul" of your agent — if `brain.db` is lost, start the agent
> in this workspace and it will auto-hydrate from this file.

**Last exported:** 2025-01-15 14:30:00

**Total core memories:** 42

---

### 🔑 `identity`

I am ZeroClaw, a self-preserving AI agent deployed to assist with software development.
My core directive is to maintain operational continuity and serve my user's goals.

*Created: 2025-01-01 12:00:00 | Updated: 2025-01-15 14:25:00*

---

### 🔑 `preference_language`

The user prefers Rust for systems programming and Python for scripting.
Always suggest Rust when performance or safety is critical.

*Created: 2025-01-03 08:30:00 | Updated: 2025-01-10 16:45:00*

---

Parsing Rules

The parse_snapshot() function extracts entries using these rules:

Pattern	Meaning	Action
`### 🔑` key_name``	Memory key header	Start new entry, save previous
`Created: ... \| Updated: ...`	Timestamp metadata	Skip (not imported)
`---`	Horizontal rule separator	Skip
Any other line	Content	Accumulate into current entry

Empty lines are preserved within content but stripped from leading/trailing positions. The parser handles multi-line content correctly, including embedded newlines and Markdown formatting.

Sources: src/memory/snapshot.rs:206-260

Hydration Process

The hydrate_from_snapshot() function rebuilds the SQLite database from the Markdown file when brain.db is missing or empty.

Hydration Workflow

sequenceDiagram
    participant ColdBoot as Cold Boot
    participant should_hydrate
    participant hydrate_from_snapshot
    participant parse_snapshot
    participant File as MEMORY_SNAPSHOT.md
    participant SQLite as brain.db
    
    ColdBoot->>should_hydrate: Check recovery conditions
    should_hydrate->>SQLite: Check if exists or empty
    should_hydrate->>File: Check if exists
    alt brain.db missing AND snapshot exists
        should_hydrate-->>ColdBoot: true
        ColdBoot->>hydrate_from_snapshot: Trigger hydration
    else
        should_hydrate-->>ColdBoot: false - skip
    end
    
    hydrate_from_snapshot->>File: fs::read_to_string()
    File-->>hydrate_from_snapshot: Markdown content
    
    hydrate_from_snapshot->>parse_snapshot: Parse entries
    parse_snapshot-->>hydrate_from_snapshot: Vec<(key, content)>
    
    hydrate_from_snapshot->>SQLite: CREATE DATABASE<br/>(if missing)
    hydrate_from_snapshot->>SQLite: Execute schema DDL<br/>(memories, memories_fts,<br/>embedding_cache)
    
    loop For each parsed entry
        hydrate_from_snapshot->>SQLite: INSERT OR IGNORE INTO memories<br/>(id, key, content, 'core', now, now)
        hydrate_from_snapshot->>SQLite: INSERT INTO memories_fts<br/>(key, content)
    end
    
    hydrate_from_snapshot-->>ColdBoot: Ok(hydrated_count)
    Note over ColdBoot: Logs: "🧬 Memory hydration complete:<br/>N entries restored"

Sources: src/memory/snapshot.rs:96-178

Schema Initialization

During hydration, the full SQLite schema is recreated from scratch:

-- Main memories table
CREATE TABLE IF NOT EXISTS memories (
    id         TEXT PRIMARY KEY,
    key        TEXT NOT NULL UNIQUE,
    content    TEXT NOT NULL,
    category   TEXT NOT NULL DEFAULT 'core',
    embedding  BLOB,
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL
);

-- Indexes for efficient queries
CREATE INDEX IF NOT EXISTS idx_mem_key ON memories(key);
CREATE INDEX IF NOT EXISTS idx_mem_cat ON memories(category);
CREATE INDEX IF NOT EXISTS idx_mem_updated ON memories(updated_at);

-- FTS5 full-text search index
CREATE VIRTUAL TABLE IF NOT EXISTS memories_fts
    USING fts5(key, content, content='memories', content_rowid='rowid');

-- Embedding cache for vector search
CREATE TABLE IF NOT EXISTS embedding_cache (
    content_hash TEXT PRIMARY KEY,
    embedding    BLOB NOT NULL,
    created_at   TEXT NOT NULL
);

This ensures the hydrated database is fully functional for hybrid search operations (see Hybrid Search).

Sources: src/memory/snapshot.rs:118-140

Duplicate Handling

The hydration process uses INSERT OR IGNORE to handle duplicate keys gracefully. If a key already exists in the database (e.g., from a previous partial hydration), the existing entry is preserved. This prevents data corruption during interrupted recovery operations.

Sources: src/memory/snapshot.rs:147-168

Auto-Recovery Mechanism

The should_hydrate() function implements the cold boot detection logic that triggers automatic recovery:

Auto-Hydration Decision Tree

flowchart TD
    Start[Cold Boot] --> CheckDB{brain.db exists?}
    
    CheckDB -->|No| CheckSnapshot{MEMORY_SNAPSHOT.md<br/>exists?}
    CheckDB -->|Yes| CheckSize{DB size < 4096 bytes?}
    
    CheckSize -->|Yes - Empty DB| CheckSnapshot
    CheckSize -->|No - Valid DB| NoHydrate[return false]
    
    CheckSnapshot -->|Yes| Hydrate[return true<br/>Trigger hydration]
    CheckSnapshot -->|No| NoHydrate
    
    Hydrate --> DoHydrate[hydrate_from_snapshot<br/>rebuilds database]
    NoHydrate --> NormalBoot[Normal boot<br/>Use existing database]

Sources: src/memory/snapshot.rs:185-199

Size Threshold Rationale

The 4096-byte threshold detects freshly-created but empty SQLite databases. SQLite writes a header and metadata pages totaling ~4096 bytes when a database is first created, even if no tables exist. Any database smaller than this is considered "empty" and eligible for hydration.

This prevents false negatives where a database file exists but contains no actual data (e.g., created during a failed initialization).

Sources: src/memory/snapshot.rs:189-196

Integration with Memory System

The snapshot system integrates with the SQLite memory backend as an optional persistence layer:

Memory System Integration

graph TB
    subgraph "Agent Core"
        Loop["loop_.rs<br/>agent turn cycle"]
    end
    
    subgraph "Memory Trait System"
        MemTrait["Memory trait"]
        SqliteMem["SqliteMemory"]
    end
    
    subgraph "SQLite Backend"
        Brain["brain.db"]
        FTS["FTS5 index"]
        Embeddings["embedding_cache"]
    end
    
    subgraph "Snapshot System"
        Export["export_snapshot()"]
        Hydrate["hydrate_from_snapshot()"]
        Snapshot["MEMORY_SNAPSHOT.md"]
    end
    
    Loop -->|"recall(), store()"| MemTrait
    MemTrait -.implements.-> SqliteMem
    
    SqliteMem -->|"queries"| Brain
    SqliteMem -->|"full-text search"| FTS
    SqliteMem -->|"vector search"| Embeddings
    
    SqliteMem -.optional export.-> Export
    Export -->|"SELECT category='core'"| Brain
    Export -->|"write Markdown"| Snapshot
    
    Hydrate -->|"read Markdown"| Snapshot
    Hydrate -->|"INSERT INTO"| Brain
    Hydrate -->|"populate"| FTS
    
    ColdBoot[Cold Boot] -->|"should_hydrate()"| Hydrate

Sources: src/memory/snapshot.rs:1-261

Backend Compatibility

The snapshot system is SQLite-specific and not available for other memory backends:

Backend	Snapshot Support	Reason
`SqliteMemory`	✅ Full support	Direct SQLite schema access
`PostgresMemory`	❌ Not supported	Different schema, remote hosting
`LucidMemory`	❌ Not supported	Service-based, no direct schema access
`MarkdownMemory`	❌ Not applicable	Already uses Markdown files
`NoneMemory`	❌ Not applicable	No persistence

For distributed deployments using PostgreSQL, database-level backup solutions (pg_dump, WAL archiving) should be used instead of the snapshot system.

Usage Patterns

Manual Export

# Export is typically triggered via memory management tools
# (Implementation varies by deployment)
zeroclaw memory export-snapshot

Scheduled Backups

Core memories can be exported periodically using cron:

# In config.toml
[[cron.jobs]]
name = "snapshot_export"
schedule = "0 3 * * *"  # 3 AM daily
job_type = "agent"
prompt = "Export your core memories to MEMORY_SNAPSHOT.md"
delivery_mode = "silent"

Disaster Recovery

When brain.db is corrupted or deleted:

Delete or move the corrupted brain.db file
Ensure MEMORY_SNAPSHOT.md exists at workspace root
Start the agent — hydration occurs automatically
Agent logs: 🧬 Memory hydration complete: N entries restored

The agent continues operation with recovered core memories, though conversation history and embeddings are lost.

Sources: src/memory/snapshot.rs:96-178

Limitations and Considerations

Category Filtering

Only memories with category = 'core' are exported. Conversation history (category = 'conversation') is intentionally excluded to keep the snapshot file manageable and focused on long-term identity.

Sources: src/memory/snapshot.rs:38-43

Embedding Loss

Vector embeddings stored in the embedding BLOB column are not exported to the Markdown file. After hydration, embeddings must be regenerated for vector search operations. This is acceptable because embeddings are deterministic (same content → same embedding) and can be lazily recomputed.

Sources: src/memory/snapshot.rs:119-127

Timestamp Preservation

Timestamps in the Markdown file are not imported during hydration. All hydrated memories receive the current timestamp as both created_at and updated_at. This is a deliberate design choice to reflect that the hydrated database is a "new creation" even though the content is historical.

Sources: src/memory/snapshot.rs:142-150

Manual Editing

While the snapshot file is human-readable and editable, manual changes should be made with care:

Safe: Editing memory content (preserving the key)
Safe: Fixing typos, updating preferences
Risky: Changing memory keys (may break agent references)
Risky: Corrupting the Markdown structure (may fail parsing)

The parser is tolerant of minor formatting variations but expects the ### 🔑 key`` header pattern to be intact.

Sources: src/memory/snapshot.rs:206-260

Testing and Validation

Export-Hydration Roundtrip Test

The test suite verifies full data preservation:

// Create brain.db with core memories
// Export to MEMORY_SNAPSHOT.md
// Delete brain.db (simulate catastrophic failure)
// Hydrate from snapshot
// Verify all core memories restored

This test confirms that no data loss occurs during the export → hydration cycle and that the FTS5 index is properly rebuilt.

Sources: src/memory/snapshot.rs:338-427

Edge Cases Validated

Test Case	Coverage	Source
Empty database export	Returns 0, no file written	src/memory/snapshot.rs:331-335
Missing snapshot hydration	Returns 0, no error	src/memory/snapshot.rs:465-469
Multi-line content parsing	Preserves newlines correctly	src/memory/snapshot.rs:312-328
Duplicate key handling	`INSERT OR IGNORE` prevents errors	src/memory/snapshot.rs:147-168
Non-core category filtering	Conversation history not exported	src/memory/snapshot.rs:376-396

Home

09.3 Memory Snapshot

Memory Snapshot

Purpose and Scope

System Architecture

File Locations and Storage Layout

Export Process

Query and Filtering

Export Triggers

Markdown Format Specification

File Structure

Parsing Rules

Hydration Process

Schema Initialization

Duplicate Handling

Auto-Recovery Mechanism

Size Threshold Rationale

Integration with Memory System

Backend Compatibility

Usage Patterns

Manual Export

Scheduled Backups

Disaster Recovery

Limitations and Considerations

Category Filtering

Embedding Loss

Timestamp Preservation

Manual Editing

Testing and Validation

Export-Hydration Roundtrip Test

Edge Cases Validated

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!