Skip to content

Commit c924d07

Browse files
fix(status): paginate metadata fetch to support large palaces
`col.get(limit=total)` causes SQLite "too many SQL variables" on palaces with >10k drawers (#802) and on older versions the hardcoded limit=10000 silently truncated the count (#850). Paginate in 5k batches using offset and aggregate wing/room counts incrementally. Also use `col.count()` for the header instead of `len(metas)` so the displayed total is always correct. Tested on a 122,686-drawer palace. Fixes #850 Related: #802, #723
1 parent 5a2f7db commit c924d07

1 file changed

Lines changed: 14 additions & 8 deletions

File tree

mempalace/miner.py

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -821,17 +821,23 @@ def status(palace_path: str):
821821
print(" Run: mempalace init <dir> then mempalace mine <dir>")
822822
return
823823

824-
# Count by wing and room
824+
# Count by wing and room — paginate to avoid SQLite "too many SQL
825+
# variables" error on large palaces (see #802, #850).
825826
total = col.count()
826-
r = col.get(limit=total, include=["metadatas"]) if total else {"metadatas": []}
827-
metas = r["metadatas"]
828-
829-
wing_rooms = defaultdict(lambda: defaultdict(int))
830-
for m in metas:
831-
wing_rooms[m.get("wing", "?")][m.get("room", "?")] += 1
827+
wing_rooms: dict = defaultdict(lambda: defaultdict(int))
828+
batch_size = 5000
829+
offset = 0
830+
while offset < total:
831+
r = col.get(limit=batch_size, offset=offset, include=["metadatas"])
832+
batch = r["metadatas"]
833+
if not batch:
834+
break
835+
for m in batch:
836+
wing_rooms[m.get("wing", "?")][m.get("room", "?")] += 1
837+
offset += len(batch)
832838

833839
print(f"\n{'=' * 55}")
834-
print(f" MemPalace Status — {len(metas)} drawers")
840+
print(f" MemPalace Status — {total} drawers")
835841
print(f"{'=' * 55}\n")
836842
for wing, rooms in sorted(wing_rooms.items()):
837843
print(f" WING: {wing}")

0 commit comments

Comments
 (0)