perf: paginate miner.status() — fixes SQLITE_MAX_VARIABLE_NUMBER crash past ~32K drawers

[jphein]

Closes #802 (reported 1 week ago) and #1015 (filed today — same crash, same stack location, filed after you merged #999).

The bug

total = col.count()
r = col.get(limit=total, include=["metadatas"])

On palaces with more than ~32,766 drawers, ChromaDB's underlying SQLite query binds total variables and exceeds SQLITE_MAX_VARIABLE_NUMBER (default 32,766), raising:

sqlite3.OperationalError: too many SQL variables

mempalace status crashes before printing anything on any palace over the limit. The fork has been running this paginated version for 9 days (landed on jphein/mempalace main on 2026-04-10) against a 152,682-drawer palace, so I haven't re-triggered the crash directly in that window — but the unpatched code path is reproducible by reverting this patch.

The fix

offset = 0
while offset < total:
    r = col.get(limit=10000, offset=offset, include=["metadatas"])
    metas = r.get("metadatas") or []
    if not metas:
        break
    ...
    offset += len(metas)

Each page stays well under the variable cap. Loop terminates when ChromaDB returns fewer than the requested batch (final page).

Also switches printed counts to thousand-separated decimals (152,682 vs 152682) — easier to read on large palaces.

Test plan

• pytest tests/test_miner.py tests/test_cli.py -q — 62 passed
• Paginated code path exercised on a 152,682-drawer palace in the fork since 2026-04-10 — status completes and prints correct per-wing/per-room counts

Notes

• This is a targeted fix for miner.py::status() only. Same underlying issue may exist in other col.get(limit=total) call sites; happy to file follow-ups if you confirm you want this scope.
• The fork (jphein/mempalace) has been running this patch since 2026-04-10. Filing now because mempalace status crashes on palaces >~32K drawers (SQLite variable limit) #1015 duplicates mempalace status crashes with "too many SQL variables" on large palaces #802 and the fix is simple enough to merge independently.

[Copilot]

Pull request overview

This PR fixes mempalace status crashing on large palaces by avoiding an unbounded col.get(limit=total, ...) call that can exceed SQLite’s SQLITE_MAX_VARIABLE_NUMBER via ChromaDB’s query plan.

Changes:

• Paginate miner.status() metadata retrieval in 10k batches using limit/offset to stay under SQLite bind-variable limits.
• Update status output to use thousands-separated numeric formatting for drawer and room counts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The new pagination/offset loop isn’t covered by tests. There are existing unit tests for status() in tests/test_miner.py, but none assert multi-page behavior (e.g., that col.get() is called with offset advancing and that results across pages are fully tallied). Please add a regression test using a fake collection with count() > batch and a get(limit, offset, ...) that returns deterministic slices, then assert the output totals match the full dataset and multiple get() calls occur.

[Copilot]

The header prints scanned (rows successfully fetched) rather than total (from col.count()). If count() and paged get() ever disagree (concurrent deletes, backend truncation/bug), the command will silently under-report drawers. Consider printing total as the authoritative count, and if scanned != total emit a clear “partial” indicator (similar to how tool_status exposes partial).

Suggested change

	print(f" MemPalace Status — {scanned:,} drawers")
	print(f" MemPalace Status — {total:,} drawers")
	if scanned != total:
	print(f" partial: scanned {scanned:,} of {total:,} drawers")

[jphein]

I should have checked for existing PRs before filing — @eldar702 already filed #1016 yesterday with the same pagination approach, narrower scope (pure bug fix; mine also adds thousand-separator print formatting). #1016 is the focused version and got here first. Happy to close this in favor of #1016, or rescope to just the formatting change as a follow-up — whichever the maintainers prefer.

[jphein]

Further correction — I had actually noted #851 in earlier triage yesterday as the canonical approved fix, and forgot that context when I filed this today. @bensig's comment on #1016 confirms: #851 is approved, MERGEABLE, and additionally addresses #850. Both #1016 and this one are narrower duplicates. Closing this in favor of #851.