status command crashes with "too many SQL variables" on large palaces (129K drawers)

[bloodcarter]

Version

• mempalace 3.3.2
• chromadb 1.5.8
• Python 3.13
• macOS (Apple Silicon)

Summary

mempalace status crashes with chromadb.errors.InternalError: too many SQL variables on a palace with 129,382 drawers. Looks like a regression of the fix from #66 ("Batch ChromaDB reads to avoid SQLite variable limit") that landed in v3.1.0.

Stack trace

File "/Users/vlad/.mempalace-venv/lib/python3.13/site-packages/mempalace/miner.py", line 852, in status
    r = col.get(limit=total, include=["metadatas"]) if total else {"metadatas": []}
File "/Users/vlad/.mempalace-venv/lib/python3.13/site-packages/mempalace/backends/chroma.py", line 341, in get
    raw = self._collection.get(**kwargs)
File "/Users/vlad/.mempalace-venv/lib/python3.13/site-packages/chromadb/api/models/Collection.py", line 161, in get
File "/Users/vlad/.mempalace-venv/lib/python3.13/site-packages/chromadb/api/rust.py", line 440, in _get
chromadb.errors.InternalError: Error executing plan: Internal error: error returned from database: (code: 1) too many SQL variables

Root cause

miner.py:852 passes limit=total to col.get(...) without batching. When total exceeds SQLite's bind-parameter ceiling (default 999, sometimes 32766), the single query fails.

Repro

1. Mine enough content to exceed ~10-30K drawers
2. mempalace status

Expected behavior

Status should paginate the metadata read using offset/limit batching, similar to the approach that #66 applied elsewhere. A batch size of 5000–10000 works reliably.

Workaround

import chromadb
from collections import defaultdict
client = chromadb.PersistentClient(path="~/.mempalace/palace")
col = client.get_collection("mempalace_drawers")
total = col.count()
wing_room = defaultdict(lambda: defaultdict(int))
offset = 0
while offset < total:
    r = col.get(include=["metadatas"], limit=5000, offset=offset)
    for m in r["metadatas"]:
        wing_room[m.get("wing","?")][m.get("room","?")] += 1
    offset += 5000

Same pattern likely affects

• list_wings, list_rooms, get_taxonomy — the MCP read tools that were previously capped at limit=10000. Check whether the cap was removed but batching wasn't added.

[mvalentsev]

Same SQL variable limit as #802: unpaginated col.get(limit=total, include=["metadatas"]) in miner.py. #851 landed a paginated fix on develop (merged 2026-04-22) and will ship in the next release.

[vnguyen-lexipol]

Thanks for the report - this is the same root cause I reproduced (status fetching metadata in one large query).

I have pushed a fix in PR #1080: #1080
The patch paginates status metadata fetches with limit/offset and aggregates incrementally, which avoids SQLite "too many SQL variables" on large palaces.

Validated locally on a large palace, plus targeted tests in tests/test_miner.py and tests/test_cli.py.

[vnguyen-lexipol]

Quick clarification: the actionable patch for this report is the new commit in PR #1080 (2ec886d), which applies paginated metadata reads in status() on the current main-based branch. This is the specific change intended to address the SQLite variable-limit failure here.

[igorls]

Resolved on develop. mempalace/miner.py status() (~L1240) now paginates via batch_size=5000 (refs #802/#850), and mempalace/mcp_server.py uses _fetch_all_metadata() (offset-based, 1000-row batches) for tool_status / tool_list_wings / tool_list_rooms / tool_get_taxonomy. The 32766-variable SQLite ceiling is no longer reachable. Reopen if 129K-drawer palaces still crash.