Drive indexing: Add DB auto vacuum

Author: vdavid

AGENTS.md

@@ -63,6 +63,12 @@ Run the smallest set of checks possible for efficiency while maintaining confide
 
 ## Debugging
 
+- **Data directories (IMPORTANT — dev and prod are separate!)**:
+    - **Prod**: `~/Library/Application Support/com.veszelovszki.cmdr/`
+    - **Dev**: `~/Library/Application Support/com.veszelovszki.cmdr-dev/`
+    - This is set by `resolved_app_data_dir()` in `src-tauri/src/config.rs` (appends `-dev` in debug builds).
+      Settings, index DBs, font metrics, AI models, and license data all live here. When debugging data issues,
+      always check the right directory for the mode you're running. Deleting the wrong one is a real risk.
 - **Unified logging**: Frontend and backend logs appear together in the terminal and in a shared log file at
   `~/Library/Logs/com.veszelovszki.cmdr/`. The log file is also accessible from Settings > Logging > "Open log file".
 - **Svelte/TypeScript**: Use LogTape via `getAppLogger('feature')` from `$lib/logging/logger`. Levels: debug, info, warn, error.

apps/desktop/src-tauri/src/indexing/CLAUDE.md

@@ -66,18 +66,22 @@ All writes go through a dedicated `std::thread` via a bounded `sync_channel` (20
 
 Reads happen on separate WAL connections (any thread). The global read-only store (`GLOBAL_INDEX_STORE`) provides enrichment without passing `AppHandle` through the listing pipeline.
 
-### SQLite schema (v3: integer-keyed)
+### SQLite schema (v4: integer-keyed, incremental vacuum)
 
-One DB per volume: `~/Library/Application Support/com.veszelovszki.cmdr/index-{volume_id}.db`
+One DB per volume. **Dev and prod use separate directories** (see AGENTS.md § Debugging):
+- **Prod**: `~/Library/Application Support/com.veszelovszki.cmdr/index-{volume_id}.db`
+- **Dev**: `~/Library/Application Support/com.veszelovszki.cmdr-dev/index-{volume_id}.db`
 
 Three tables:
 - `entries` (id INTEGER PK, parent_id, name COLLATE platform_case, is_directory, is_symlink, size, modified_at) with unique index `idx_parent_name(parent_id, name)`. Root sentinel: id=1, parent_id=0, name="".
 - `dir_stats` (entry_id INTEGER PK, recursive_size, recursive_file_count, recursive_dir_count)
 - `meta` (key TEXT PK, value TEXT) WITHOUT ROWID
 
-WAL mode, 16 MB page cache. Custom `platform_case` collation registered on every connection: case-insensitive + NFD normalization on macOS, binary on Linux. **Opening the DB with the sqlite3 CLI will fail** on queries touching the name column (the collation isn't registered).
+WAL mode, 16 MB page cache, `auto_vacuum = INCREMENTAL` (free pages reclaimed via `PRAGMA incremental_vacuum` after truncation). Custom `platform_case` collation registered on every connection: case-insensitive + NFD normalization on macOS, binary on Linux. **Opening the DB with the sqlite3 CLI will fail** on queries touching the name column (the collation isn't registered).
 
-**Schema v3**: Bumped from v2 to force DB rebuild after fixing orphan entry bug. Scanner, writer, aggregator, reconciler, enrichment, and IPC commands all fully migrated to integer keys. `IndexManager` owns a `PathResolver` for LRU-cached path→ID resolution in IPC commands (`get_dir_stats`, `get_dir_stats_batch`). Enrichment uses integer-keyed fast path: resolve parent once → batch child dir stats by ID. Reconciler sends integer-keyed messages exclusively. Old path-keyed `WriteMessage` variants and backward-compat shims (`ScannedEntry`, `DirStats`) still exist for post-replay verification — cleanup in milestone 6.
+History of changes:
+- **Schema v3**: Bumped from v2 to force DB rebuild after fixing orphan entry bug. Scanner, writer, aggregator, reconciler, enrichment, and IPC commands all fully migrated to integer keys. `IndexManager` owns a `PathResolver` for LRU-cached path→ID resolution in IPC commands (`get_dir_stats`, `get_dir_stats_batch`). Enrichment uses integer-keyed fast path: resolve parent once → batch child dir stats by ID. Reconciler sends integer-keyed messages exclusively. Old path-keyed `WriteMessage` variants and backward-compat shims (`ScannedEntry`, `DirStats`) still exist for post-replay verification — cleanup in milestone 6.
+- **Schema v4**: Bumped from v3 to enable `auto_vacuum = INCREMENTAL` (requires DB rebuild since the pragma must be set before table creation).
 
 ## How to test
 

apps/desktop/src-tauri/src/indexing/store.rs

@@ -19,7 +19,7 @@
 use rusqlite::{Connection, OptionalExtension, params};
 use std::path::{Path, PathBuf};
 
-const SCHEMA_VERSION: &str = "3";
+const SCHEMA_VERSION: &str = "4";
 
 /// Root entry sentinel ID. All top-level entries have `parent_id = ROOT_ID`.
 pub const ROOT_ID: i64 = 1;
@@ -280,7 +280,8 @@ fn ensure_root_sentinel(conn: &Connection) -> Result<(), IndexStoreError> {
 /// Apply WAL-mode pragmas for performance.
 fn apply_pragmas(conn: &Connection) -> Result<(), IndexStoreError> {
     conn.execute_batch(
-        "PRAGMA journal_mode = WAL;
+        "PRAGMA auto_vacuum = INCREMENTAL;
+         PRAGMA journal_mode = WAL;
          PRAGMA synchronous = NORMAL;
          PRAGMA cache_size = -16384;",
     )?;

apps/desktop/src-tauri/src/indexing/writer.rs

@@ -415,6 +415,10 @@ fn process_message(conn: &rusqlite::Connection, msg: WriteMessage, stats: &Write
                         "Writer: truncated entries + dir_stats ({}ms)",
                         t.elapsed().as_millis(),
                     );
+                    // Reclaim free pages from the truncation
+                    if let Err(e) = conn.execute_batch("PRAGMA incremental_vacuum;") {
+                        log::warn!("Writer: incremental_vacuum after truncate failed: {e}");
+                    }
                 }
                 Err(e) => log::warn!("Writer: truncate failed: {e}"),
             }