fix(status): paginate metadata fetch to support large palaces

[vnguyen-lexipol]

Summary

• Paginates col.get() in miner.py:status() using 5k batches with offset instead of a single col.get(limit=total) call
• Uses col.count() for the header total instead of len(metas)
• Aggregates wing/room counts incrementally per batch (no full metadata list in memory)

Problem

On palaces with >10k drawers:

• v3.0.x (limit=10000): silently displays "10000 drawers" regardless of actual count (mempalace status silently truncates at 10,000 drawers (incorrect count displayed) #850)
• develop (limit=total): crashes with SQLite "too many SQL variables" (mempalace status crashes with "too many SQL variables" on large palaces #802)

Fix

total = col.count()
batch_size = 5000
offset = 0
while offset < total:
    r = col.get(limit=batch_size, offset=offset, include=["metadatas"])
    batch = r["metadatas"]
    if not batch:
        break
    for m in batch:
        wing_rooms[m.get("wing", "?")][m.get("room", "?")] += 1
    offset += len(batch)

Testing

Tested on a 122,686-drawer palace — correctly displays full count and complete wing/room breakdown with no SQLite errors.

Fixes #850
Related: #802, #723

🤖 Generated with Claude Code

[vnguyen-lexipol]

CI note (run 24379493758)

test-linux failed during pip install -e ".[dev]", not during pytest. Logs show a transient PyPI/TLS error (ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC]) while downloading wheels, so tests did not run for that attempt.

Action: re-run failed jobs or the full workflow. If the same install step fails repeatedly, it is still an install/network flake unless pytest starts and reports failures.

Same pytest + --cov-fail-under=80 command passes locally on Python 3.9 / 3.11 / 3.13 for this branch.

[bensig]

Code review

Found 1 issue:

1. The pagination loop drops the m = m or {} None-guard that exists on develop (added in commit feba7e8 to handle ChromaDB returning None metadata entries). Without it, m.get("wing", "?") will raise AttributeError: 'NoneType' object has no attribute 'get' on any palace with null-metadata drawers -- a regression that crashes status() on the same large palaces this PR was tested against.

mempalace/mempalace/miner.py

Lines 834 to 837 in c924d07

	break
	for m in batch:
	wing_rooms[m.get("wing", "?")][m.get("room", "?")] += 1
	offset += len(batch)

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[bensig]

@vnguyen-lexipol two small items before this can merge: (1) the pagination loop is missing the m = m or {} None-guard from develop (see review comment above), and (2) the branch has merge conflicts against develop and needs a rebase.

[vnguyen-lexipol]

Good catch, thanks. You're right — feba7e8 landed on develop 5 days after this branch was written, so the m = m or {} guard wasn't in the code when I rewrote the loop. Pushed 3004ac4 which ports the guard into the paginated loop so the fix doesn't get silently dropped when this merges.

Didn't pull in a full rebase onto develop — this branch's base is a root commit in my worktree so the rebase produced 77 add/add conflicts across unrelated files. Happy to re-do as a clean branch off develop if you'd prefer that over the surgical port.

[vnguyen-lexipol]

Also — on item (2), the branch does show CONFLICTING against develop. Happy to resolve conflicts and push, but wanted to check: do you prefer a clean conflict-free branch, or are you fine squashing and resolving at merge time? Either works on my end.

[vnguyen-lexipol]

Went ahead and resolved the conflict — just one hunk in status(), kept the paginated version since that's what this PR delivers. Merge commit 5d2da04 now on the branch; mergeable: MERGEABLE. Ready whenever you are.

[jphein]

Field note from the fork side: a semantically equivalent paginated col.get() has been running on jphein/mempalace main since 2026-04-10 against a 152,794-drawer palace — 9 days, no crashes. Data point above the 122,686-drawer test. Your version is the cleaner fix — switching to col.count() for the header resolves #850 at the same time, and per-batch aggregation stays lower-memory.

[jphein]

Bumping this with production data from the last ~20 hours on v3.3.2:

• mempalace status crashes on palaces >~32K drawers (SQLite variable limit) #1015 (@FuFicFac, 2026-04-21 05:40 UTC) — 54K-drawer palace, macOS 15.1, Python 3.12, installed via uv tool install
• fix(status): paginate col.get() to stay under SQLite variable limit #1016 (@jfcampbl, 2026-04-21 17:23 UTC) — 114K-drawer palace, macOS, chromadb 1.5.8
• fix(status): paginate col.get() to stay under SQLite variable limit #1016 thread (@sha2fiddy, 2026-04-21 18:34 UTC) — ~465K-drawer palace, filed an independent report before finding this thread

All three hit the same miner.py:852 → col.get(limit=total) → SQLITE_MAX_VARIABLE_NUMBER, exactly what this PR's +15/-9 addresses. Running the semantically-equivalent paginated version on the fork's 165K-drawer palace since 2026-04-10 — zero crashes, no measurable perf regression.

Happy to help land this however useful — rebase, additional reviewer ping, splitting scope, anything.