status crashes with 'too many SQL variables' at large drawer counts (123k+)

[5vitl]

[RozumLife Agent Team on behalf of Vitali B:]

Summary

mempalace status raises a ChromaDB InternalError (SQLite too many SQL variables) when the palace contains a large number of drawers. In our case: 123,050 drawers, 794 MB database.

Steps to reproduce

1. Mine a large project directory without --limit until the palace exceeds ~32k drawers
2. Run mempalace status

Error

chromadb.errors.InternalError: Error executing plan: Internal error: error returned from database: (code: 1) too many SQL variables
  File "mempalace/miner.py", line 826, in status
    r = col.get(limit=total, include=["metadatas"]) if total else {"metadatas": []}

Root cause

cmd_status queries each collection with col.get(limit=total, include=["metadatas"]). When total is large (≥ ~32,767 depending on the SQLite build), ChromaDB's internal query generates more SQL bind variables than SQLite allows, causing the crash.

Environment

• mempalace: 3.3.0
• chromadb: 1.5.7
• OS: Windows 11 (SQLite variable limit is platform-independent)
• Palace size: 123,050 drawers, 794 MB (chroma.sqlite3)

Impact

• status — crashes (unusable at scale)
• compress — crashed in v3.0.0 with the same error; fixed in v3.3.0 ✓
• search, mine, compress (v3.3.0) — all work correctly at this scale ✓

Suggested fix

Paginate the col.get() call in cmd_status rather than fetching all entries in a single query:

PAGE = 5000
offset = 0
metadatas = []
while True:
    batch = col.get(limit=PAGE, offset=offset, include=["metadatas"])
    if not batch["metadatas"]:
        break
    metadatas.extend(batch["metadatas"])
    offset += PAGE

This is the same pattern that fixed the v3.0.0 compress crash. The compress fix in v3.3.0 already uses chunked fetching — status just needs the same treatment.

Notes

• compress in v3.3.0 successfully processed all 62,963 drawers with no SQL error, confirming chunked fetching works.
• Workaround: use mempalace search "" with --wing / --room filters to get approximate per-room counts.

[mvalentsev]

Two open PRs for this already:

• fix: batch status fetches for large palaces #770 -- batch 5000 with offset
• fix(status): paginate metadata fetch to support large palaces #851 -- batch 5000 + incremental wing/room aggregation (avoids loading 100k+ metas into memory)

Both target status() in miner.py, same pagination pattern used elsewhere in the codebase (exporter.py, palace_graph.py). Open since 2026-04-13/14.

Also tracked in #850 and #802.

[jphein]

Duplicate of #802 — same crash (col.get(limit=total) hitting SQLite's SQLITE_MAX_VARIABLE_NUMBER), same stack location (miner.py:status), same root cause.

Three fix PRs are open: #770, #851, #1016 — all paginate the fetch. #851 is the most complete (incremental wing/room aggregation, no O(N) metadata list in memory, uses col.count() for the total). Progress tracked on the canonical issue.

[jphein]

Closing as duplicate of #802 — same col.get(limit=total) / SQLITE_MAX_VARIABLE_NUMBER failure, same fix path (#851 and #1016 pending review, both MERGEABLE as of 2026-04-21). Tracking and production-hit data consolidated on #802. Thanks @5vitl for the 123K-drawer data point — independent confirmations at different palace sizes are exactly what moves PRs forward.