migrate and repair rebuild trigger unbounded link_lists.bin growth on large palaces

[sha2fiddy]

Summary

Both mempalace migrate and python -m mempalace.repair rebuild re-insert every drawer into a fresh chromadb collection. On a 199,539-drawer / 1.7 GB palace, each path independently produced a multi-hundred-GB sparse link_lists.bin and filled the disk. These are two reproducible entry points for the hnswlib resize-drift bug documented in #344 that are not currently covered by any report, PR, or safeguard.

Root cause is upstream (chroma-core/chroma#2594 — hnswlib link_lists.bin grows unbounded, acknowledged as "by design"), but mempalace is the one calling it, has no disk-space or post-write integrity checks on these paths, and repair rebuild is literally the recommended recovery from HNSW corruption — it reproduces the bug it is meant to fix.

Environment

• mempalace develop @ 32ec74d (v3.3.0), installed via uv tool install --python 3.12
• chromadb 1.5.8
• Python 3.12, macOS 15 (Darwin 25.4.0)
• Palace: 199,539 drawers, 1.7 GB on disk pre-migration, stable on chromadb 0.6.3 for weeks prior

Variant A — mempalace migrate --yes (0.6.3 → 1.5.8 schema)

Migration ran ~1h 45min, wrote the pre-migration backup, and exited with zero errors. The final HNSW write step silently corrupted the new collection:

File	On disk	Logical	State
data_level0.bin	349 MB	349 MB	OK
link_lists.bin	647 GB	5.04 TB	sparse, runaway
index_metadata.pickle	small	small	truncated mid-write

Palace is unopenable: loading the pickle raises _pickle.UnpicklingError: pickle data was truncated. Migrate does not verify HNSW state or pickle integrity before returning success.

Reproduce: on a palace of ~100K+ drawers on chromadb 0.6.3, run mempalace migrate --yes to 1.5.x. Watch du -sh ~/.mempalace/palace/*/link_lists.bin during the final write phase.

Variant B — python -m mempalace.repair rebuild

Runs without any schema migration. After quarantining the corrupt segment from Variant A (rename → .drift-<date>) and invoking repair rebuild, a new collection UUID was created and the upsert phase began:

• link_lists.bin reached 122 GB on disk / 223 GB logical in under 20 min before a disk-usage watchdog tripped.
• pkill -9 -f "mempalace.repair" returned without confirming child-process death. Workers kept writing to the deleted file (open FDs), so disk was not reclaimed until a second pkill pass closed all FDs (~450 GB reclaim).

repair rebuild is the documented recovery path from HNSW corruption — but runs the same hnswlib writer that caused it. No escape hatch.

Reproduce: on any palace of ~50K+ drawers on chromadb 1.5.x, python -m mempalace.repair rebuild. Watch du -sh ~/.mempalace/palace/*/link_lists.bin during upsert.

Related

• HNSW index bloat: link_lists.bin grows to 441GB when mining >10K drawers #344 — same mechanism via mempalace mine at 56K drawers. Documents the resize-drift analysis.
• HNSW link_lists.bin grows to terabytes, causes segfault and APFS orphaned blocks on macOS #525 — closed, 0.6.3-era, add() vs upsert() on re-mine. Different trigger.
• Critical Data write loop, HNSW compaction bug #965 / SIGSEGV in HNSW parallel inserts — missing hnsw:num_threads on collection creation #974 — 10.2 TB sparse link_lists.bin during mine on chromadb 1.x (threaded race).
• fix: mempalace migrate produces incomplete ChromaDB 0.6.x schema #722 — migrate schema incompleteness in the reverse direction (1.5.x → 0.6.x); different symptom.
• PR fix: prevent HNSW index bloat from resize+persist cycles #346 — proposes hnsw:batch_size=50000, hnsw:sync_threshold=50000 for miner.py. Does not touch migrate.py or repair.py.
• PR fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976 — hnsw:num_threads=1 + mine_global_lock. Addresses threading, not resize drift.
• Upstream: [Feature Request]: HNSW index pruning chroma-core/chroma#2594 (unbounded growth acknowledgement), [PERF] Add hnsw:initial_capacity to reduce memory from HNSW index resizing chroma-core/chroma#6621 (hnsw:initial_capacity, unmerged), [Security] Unsafe pickle.load() in PersistentLocalHnswSegment enables arbitrary code execution (CWE-502) chroma-core/chroma#6926 (pickle RCE — confirms pickle still on load path).

Suggested mitigations (mempalace-side, pending upstream fix)

1. Pre-flight disk check in migrate and repair rebuild: abort if free space < 10× palace size.
2. Live size guard during HNSW writes: if link_lists.bin exceeds N× data_level0.bin, stop and restore backup.
3. Post-op integrity check: open the new collection, confirm count() matches source and index_metadata.pickle unpickles, before reporting success.
4. Apply PR fix: prevent HNSW index bloat from resize+persist cycles #346's hnsw:batch_size / hnsw:sync_threshold overrides at the collection-creation sites in migrate.py and repair.py, not only miner.py.
5. Fix watchdog termination in repair.py: after pkill -9, re-pgrep and retry before rm-ing any segment file (open-but-deleted FDs block reclaim).

Current workaround

Restored pre-migration backup, pinned chromadb==0.6.3 in pyproject.toml, reinstalled via uv tool install --force, re-applied the macOS CPU-embedding-provider patch (abandoned branch fix/cpu-embedding-provider-macos). Palace opens, count() == 199540. Will not run migrate or repair rebuild until this is resolved.

[sha2fiddy]

Follow-up: restore + 0.6.3 pin is not sufficient on its own

Restoring a pre-incident backup and pinning chromadb==0.6.3 restored my palace to a reachable state but left the drawer collection unusable. mempalace_status works (sqlite-only), but any HNSW-touching tool (mempalace_search, mempalace_diary_write, mempalace_add_drawer) fails on load with:

RuntimeError: Not enough memory: loadIndex failed to allocate linklist

Root cause

The pre-migrate snapshot captured the palace mid-runaway, so link_lists.bin itself contains garbage: at element 72,019 the 4-byte linkListSize reads as 2,667,577,848 (2.5 GB) — bogus. hnswlib tries to malloc(2.5 GB) for that one element and aborts. A healthy file on the same palace (Apr 12 snapshot, 41K elements) has max linkListSize = 204 bytes. If the blowup occurred during or near a persist, the on-disk snapshot is corrupt and downgrading chromadb doesn't rebuild it.

What is and isn't recoverable

• data_level0.bin — healthy. Element layout for 384-dim: [4-byte level-0 link count][128 bytes level-0 neighbors][1536 bytes float32 vector][8-byte uint64 label] = 1676 bytes. Labels at offset 1668 align 1:1 with index_metadata.pickle: label_to_id.
• index_metadata.pickle — unpickles cleanly (dimensionality, total_elements_added, id_to_label, label_to_id, id_to_seq_id all intact).
• link_lists.bin — unrecoverable. Graph topology for levels ≥ 1 must be rebuilt.
• chroma.sqlite3 — healthy; all 198,837 drawer rows and embedding IDs present.

Vectors survive, graph doesn't. Reading vectors from data_level0.bin and re-inserting them into a fresh hnswlib.Index is enough — no re-embedding, no document access.

Minimal recovery script (hnswlib + pickle, no ChromaDB)

Load fails before any ChromaDB method is reachable, so the existing mempalace.repair.rebuild_index can't help — it calls backend.get_collection(...) first (mempalace/repair.py:225).

import os, pickle, struct, numpy as np, hnswlib

SEG = "<palace>/<vector-segment-uuid>"   # e.g. 5ef0f58c-…
OUT = "/tmp/hnsw-rebuild"
DIM, ELEM, VEC_OFF, LABEL_OFF = 384, 1676, 132, 1668
M, EF_CONSTRUCTION = 16, 100

with open(f"{SEG}/index_metadata.pickle", "rb") as f:
    meta = pickle.load(f)

with open(f"{SEG}/data_level0.bin", "rb") as f:
    raw = f.read()

N = meta.total_elements_added
assert len(raw) == N * ELEM

vecs = np.empty((N, DIM), dtype=np.float32)
labels = np.empty(N, dtype=np.uint64)
for i in range(N):
    off = i * ELEM
    vecs[i] = np.frombuffer(raw[off+VEC_OFF:off+VEC_OFF+DIM*4], dtype=np.float32)
    labels[i] = struct.unpack("<Q", raw[off+LABEL_OFF:off+LABEL_OFF+8])[0]

# dedup (rare duplicate labels appear on some palaces; keep last)
_, rev_first = np.unique(labels[::-1], return_index=True)
keep = np.sort(np.arange(N)[::-1][rev_first])
vecs, labels = vecs[keep], labels[keep]
nonzero = labels != 0
vecs, labels = vecs[nonzero], labels[nonzero]

os.makedirs(OUT, exist_ok=True)
idx = hnswlib.Index(space="l2", dim=DIM)   # match segment's hnsw:space
idx.init_index(
    max_elements=max(int(len(labels) * 1.3), 250_000),
    ef_construction=EF_CONSTRUCTION, M=M,
    is_persistent_index=True, persistence_location=OUT,
)
idx.add_items(vecs, labels)
idx.persist_dirty()

meta.total_elements_added = len(labels)
with open(f"{OUT}/index_metadata.pickle", "wb") as f:
    pickle.dump(meta, f, pickle.HIGHEST_PROTOCOL)

# Move <SEG>/{data_level0.bin,header.bin,link_lists.bin,length.bin,index_metadata.pickle}
# to a sibling <SEG>/.corrupt-<ts>/, then copy OUT/* into SEG/.

Runtime on 166,998 × 384-dim: 8 seconds on M1 Max, peak RSS ~1 GB. Self-query sanity check: top-1 for the first N vectors returns their own labels at distance 0.

After swap, direct chromadb load + query + add + delete all succeed and persist across restarts. Drawers in chroma.sqlite3 but missing from the pickle (~31.8K in my case, from prior #823-class staleness) need re-embedding to be vector-searchable again — separate follow-up.

Proposals

1. Note in the README workaround that restore + downgrade may be insufficient if the backup was taken mid-blowup. A linkListSize > 100KB scan of link_lists.bin catches this cheaply.
2. Add a mempalace repair rebuild-hnsw subcommand that runs the above against a segment directly, bypassing the ChromaDB API path so it works when loadIndex fails. Happy to PR — it's the sister of repair.rebuild_index at a lower layer.