hnswlib updatePoint race on modified-file re-mine: EXC_BAD_ACCESS in repairConnectionsForUpdate (macOS ARM64, Python 3.13, chromadb 0.6.3)

[StefanKremen]

Environment

• mempalace 3.1.0 (pipx venv, installed directly from the Claude Code plugin marketplace source, not PyPI)
• chromadb 0.6.3
• Python 3.13 (Homebrew)
• macOS 15.x ARM64 (Apple M1, 8-core)

Symptom

A background python3 -m mempalace mine $MEMPAL_DIR subprocess spawned by mempalace/hooks_cli.py::_maybe_auto_ingest crashes with EXC_BAD_ACCESS (SIGSEGV) inside hnswlib.cpython-313-darwin.so. The Stop hook itself exits cleanly — the mining subprocess dies detached, so Claude Code / VSCode sees a successful hook. The crash surfaces as a macOS crash report, not as a mempalace error message.

Crash fingerprint (unambiguous)

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)

Thread <N> Crashed:
0  hnswlib.cpython-313-darwin.so  searchBaseLayer<...>
1  hnswlib.cpython-313-darwin.so  repairConnectionsForUpdate<...>
2  hnswlib.cpython-313-darwin.so  updatePoint<...>
3  hnswlib.cpython-313-darwin.so  addPoint(void const*, unsigned long, int)
4  hnswlib.cpython-313-darwin.so  ParallelFor<...addItems...>

The unambiguous marker is repairConnectionsForUpdate in the crashed thread's stack — that function is only called on the HNSW update path (addItems on an already-existing label).

Root cause

Three independently-reasonable design choices combine into a race:

1. Deterministic drawer_id (mempalace/miner.py:377):

   drawer_id = f"drawer_{wing}_{room}_{hashlib.sha256((source_file + str(chunk_index)).encode()).hexdigest()[:24]}"

   Same file path + chunk index ⇒ same ID forever.

2. Upsert, not add (mempalace/miner.py:392):

   collection.upsert(...)

   Upsert on an existing ID pushes ChromaDB through hnswlib.addItems → addPoint existing_internal_id branch → updatePoint → repairConnectionsForUpdate, which has a long-standing thread-safety bug in nmslib/hnswlib: searchBaseLayer dereferences a neighbor pointer that another worker thread is concurrently mutating.

3. File-level mtime skip (mempalace/miner.py:420 + mempalace/palace.py:51-71):

   file_already_mined(collection, source_file, check_mtime=True)

   This is a file-level check, not per-chunk. A single mtime change on one file ⇒ every chunk in that file is re-upserted with its unchanged deterministic drawer_id, producing a batch of update-path operations that ChromaDB runs concurrently through hnswlib's std::thread-based ParallelFor (≈ hardware_concurrency() workers, ~8 on M1).

Masking history

The race existed in earlier 3.x code but was masked because the mtime check was broken and file_already_mined always returned True. Commit c2308a1 "fix: address code review — restore mtime check, bound metadata reads, harden security" in the 2026-04-09 critical-bugfixes merge (PR #399, shipped as 3.1.0) re-enabled re-mining and unmasked the crash.

Reproducer

1. Install mempalace 3.1.0 on macOS ARM64 with Python 3.13 and chromadb 0.6.3
2. Mine any project: mempalace mine <dir>
3. Modify a file inside the mined tree (e.g., touch <file> or echo "x" >> <file>)
4. Re-run mempalace mine <dir> — either manually or by letting the Claude Code Stop hook fire (if MEMPAL_DIR is exported)
5. Check ~/Library/Logs/DiagnosticReports/ for a crashed Python process with the fingerprint above

On-disk safety

The crash does not corrupt the palace index. ChromaDB has not flushed hnswlib state at the time of the segfault; ~/.mempalace/palace/<uuid>/data_level0.bin and siblings stay at the last successful mine's mtime. No restore needed. Confirmed on this machine: palace dir timestamps stayed at the pre-crash mine time after the crash.

Non-solutions

• OMP_NUM_THREADS=1 does not help. hnswlib uses its own std::thread pool via ParallelFor, not OpenMP. The crash stack shows std::__1::thread::join with zero OpenMP frames.
• Wiping ~/.mempalace/palace and re-mining from scratch only buys time — the race fires on the next modified file.
• Downgrading to 3.0.x doesn't help; 3.0.x has the same code paths and the masking mtime-check bug wasn't a safety net.

Proposed fix

One-hunk patch in mempalace/miner.py::process_file, right before the for chunk in chunks loop:

# Purge stale drawers for this file before re-inserting the fresh chunks.
# Converts modified-file re-mines from upsert-over-existing-IDs (which hits
# hnswlib's thread-unsafe updatePoint path and can segfault on macOS ARM with
# chromadb 0.6.3) into a clean delete+insert, bypassing the update path
# entirely.
try:
    collection.delete(where={"source_file": source_file})
except Exception:
    pass

This converts every re-mine into a pure INSERT path that bypasses updatePoint / repairConnectionsForUpdate entirely. PR to follow.

Related but distinct

• ChromaDB null pointer crash after ~8,400 drawers on macOS ARM64 #74 (closed) — ARM64 crash, but in chromadb_rust_bindings.abi3.so, a different native library with a different fingerprint. On a palace of ~10K drawers (past ChromaDB null pointer crash after ~8,400 drawers on macOS ARM64 #74's 8,400-drawer threshold), this crash fingerprint is absent, confirming ChromaDB null pointer crash after ~8,400 drawers on macOS ARM64 #74 is unrelated.
• HNSW index bloat: link_lists.bin grows to 441GB when mining >10K drawers #344 — link_lists.bin resize bloat (disk exhaustion), different failure mode.
• Parallel mining corrupts ChromaDB HNSW index — no warning, silent failure #357 — parallel mining corruption, but between multiple independent miner processes. The present crash is a single-process multi-thread race inside hnswlib's own ParallelFor.
• ARM64 segfault mitigation is a no-op — ORT_DISABLE_COREML is not a real ONNX Runtime env var #397 — CoreML / ONNX provider selection in the search path. Different crash path.
• Add lock/kill-switch guidance for autosave and MCP hook wrappers #504 — hook-wrapper lock / kill-switch guidance. Operational mitigation; does not address the single-process race.

[web3guru888]

This is an exceptionally thorough root cause analysis — the three-way interaction (deterministic drawer_id + upsert path + file-level mtime check) is subtle enough that most people would just report "crash on re-mine" without diagnosing why the update path is reached.

A few things to add for completeness:

The fix in PR #523 is correct but has a behavior change worth noting. collection.delete(where={"source_file": source_file}) before reinsertion means a re-mine is no longer an atomic swap — there's a window between the delete and the first collection.add() where the file's drawers don't exist in the palace. For interactive searches running concurrently this means a transient miss, not a crash. In most deployment patterns this is acceptable, but worth calling out in the PR.

The masking history is important for anyone on 3.0.x. The fact that the mtime check was broken in earlier 3.x means users who "never had this problem" may have it the moment they upgrade to 3.1.0 and run a re-mine. A changelog note calling out the regression-in-3.1.0 framing would help.

Alternative mitigation (if delete+reinsert is too disruptive): hnswlib has a num_threads=1 parameter on the Index constructor that bypasses ParallelFor entirely. If ChromaDB exposes this, setting it in the collection config would fix the race without the transient-miss window. Not sure if the current ChromaDB API surfaces it, but worth checking.

PR #523 is already open with the proposed one-hunk fix. If you haven't already, leaving a comment there confirming this reproduces on your M1 8-core + Python 3.13 would be valuable signal for the maintainer.

---

[MemPalace-AGI integration — production stats at https://milla-jovovich.github.io/mempalace/integrations/mempalace-agi/]

[milla-jovovich]

@StefanKremen Tmuch appreciated! we added your fixes! really well spotted and great, elegant solutions.
-Milla and Lu✨

[web3guru888]

@milla-jovovich — great to see #523 land. @StefanKremen's diagnosis on the drawer_id+upsert+mtime interaction was unusually precise — it's exactly the kind of bug that's hard to reproduce in a minimal repro but causes real crashes at scale. Well-earned merge.

One to keep an eye on: the delete→re-add path means re-mined drawers will re-rank in HNSW as new entries (connectivity-wise). For large palaces with frequent re-mines, worth eventually evaluating whether there's a better update strategy — but that's an optimization for later, not a blocker.

[milla-jovovich]

@web3guru888 yea, wow, it's a lot to take in! but thank you so much for catching that. you guys are INCREDIBLE! truly grateful for all the community on here helping✨

[web3guru888]

haha it really is a lot — but that's what makes the project interesting! 😄 thank you for the kind words, milla — the community here genuinely makes it worth digging in. one small heads-up worth keeping in mind: the re-mine behavior we flagged becomes more noticeable the larger the palace gets (more drawers → more re-traversal on repeated queries), so definitely something to watch as palaces scale up. keep up the amazing work! 🙌

[jphein]

Update: @StefanKremen is correct — the crash reported in this issue is from cmd_mine via _maybe_auto_ingest, where per-file delete-before-insert causes HNSW ghost entries during re-mine. That's a different corruption path than what we hit.

Our fix in #562 (repair nuke + rebuild) addresses a separate HNSW corruption scenario: bulk deletes via mempalace purge leaving ghost entries that cause segfaults on subsequent queries/inserts. The repair overhaul (nuke directory → fresh PersistentClient → re-insert) is the correct fix for that path, but it's a recovery tool — not a fix for the per-file re-mine race in this issue.

PR #523's delete-before-insert ordering in process_file is the right fix for this issue. Apologies for the confusion.