Concurrent writers (hooks + MCP server + CLI) corrupt the palace on chromadb 1.5.8 — sparse-file bloat + SIGSEGV

[AndreyBelyy]

Environment

• mempalace 3.3.2 (installed via pipx, 2026-04-21 release)
• chromadb 1.5.8 (latest)
• Python 3.12.12
• macOS 26.2 (25C56), ARM64 (Apple Silicon / M4)
• Claude Code CLI + Claude desktop both connected to the same palace

Summary

When MemPalace's Claude Code hooks (SessionStart, Stop, PreCompact) spawn mempalace mine processes that run concurrently with an active MCP server (python -m mempalace.mcp_server) and/or an explicit CLI mempalace mine invocation, the chromadb 1.5.8 HNSW vector segment becomes corrupted. The corruption manifests as:

1. Sparse-file bloat: an HNSW segment's link_lists.bin reports a small logical size (~300 KB) but allocates hundreds of gigabytes of disk blocks — in my case 362 GB (337 GiB) allocated for a 302,760-byte file. du -sh ~/.mempalace/palace/ reported 338 GB.
2. Hard SIGSEGV: any subsequent open of the palace by chromadb crashes Python with EXC_BAD_ACCESS / KERN_INVALID_ADDRESS, because chromadb mmaps the full allocated region and reads run off the end of valid memory.

mempalace repair cannot recover because chromadb itself crashes before the repair logic runs, and even after successful surgical recovery + mempalace repair --yes (34905 drawers rebuilt), chromadb still segfaults on the next read. Only a full nuke + re-mine fixes it — and the corruption reproduces within minutes of restoring hooks.

Detection

stat -f "%N: logical=%z blocks=%b" ~/.mempalace/palace/*/link_lists.bin
du -sh ~/.mempalace/palace/

Root cause (hypothesis)

chromaDB 1.5.8's local HNSW segment writer is not safe under concurrent writers across processes. MemPalace currently allows three concurrent writer paths: (1) hooks spawning mempalace mine, (2) MCP server writes, (3) user-invoked CLI mempalace mine. No cross-process lock prevents them from touching the HNSW segment simultaneously.

Suggestions

• Serialize all writers via a cross-process lock (flock on the palace dir).
• Or route all writes through the MCP server (CLI + hooks become MCP clients).
• Add a startup sanity check warning when HNSW *.bin has blocks*512 far > logical size.
• Fsync the HNSW index after each batch so mid-write crashes don't leave sparse garbage.

Happy to share the full Python crash report on request.

[gounthar]

Same setup, WSL2/Linux, chromadb 1.5.8, mempalace 3.3.2 — and somehow I ended up with five MCP servers running concurrently alongside an explicit mempalace mine process. Good times. (WSL2 on Windows, because apparently I enjoy making my life unnecessarily complicated.)

I ran into a few things that weren't in the original report. Sharing them in case they help, though I'll be upfront: I'm not 100% sure my fixes are correct, and your mileage may vary significantly.

data_level0.bin too small is a separate crash

Fixing link_lists.bin doesn't necessarily fix the palace. I had both problems at the same time, which I probably should have figured out faster than I did.

Once link_lists.bin was truncated to its expected size, it still crashed — because data_level0.bin was smaller than max_elements × size_data_per_element. ChromaDB mmaps the full expected range on load regardless of actual file size, so any page access beyond the real end of file gives you a SIGSEGV.

You can check:

import struct, pathlib
hdr = pathlib.Path("~/.mempalace/palace/<seg-id>/header.bin").expanduser().read_bytes()
max_el = struct.unpack_from("<Q", hdr, 12)[0]
size_data = struct.unpack_from("<Q", hdr, 28)[0]
expected = max_el * size_data
actual = pathlib.Path("~/.mempalace/palace/<seg-id>/data_level0.bin").expanduser().stat().st_size
print(f"expected={expected}, actual={actual}, undersized={actual < expected}")

If undersized=True, the fix I used was truncate -s <expected> data_level0.bin. The extension is sparse (zero pages), and ChromaDB treats those as unoccupied slots. I think this is safe — but I'm not certain I fully understand all the implications, so please back things up first.

Getting data out without re-mining

mempalace repair crashes before it can do anything useful because the HNSW load segfaults first.

What actually worked for me — and I'll admit this felt like archaeology more than engineering — was going around HNSW entirely:

1. Export directly from chroma.sqlite3's FTS5 content table, no HNSW involved:

   SELECT e.embedding_id, efs.c0 as document, ...
   FROM embeddings e
   LEFT JOIN embedding_fulltext_search_content efs ON efs.id = e.id
   WHERE e.segment_id = ?

2. Delete chroma.sqlite3 entirely. Truncating or purging tables doesn't work when FTS5 is corrupted — DELETE fails. Also: if you leave sqlite3 in place and only delete the HNSW dirs, chromadb triggers an automatic bulk rebuild on the next PersistentClient() call and crashes around 10K items on 1.5.8.
3. Fresh PersistentClient() + upsert from the JSON export. Filter out metadata keys starting with chroma: before upserting — chroma:document is stored redundantly in metadata and throws a ValueError on re-insert.

~43K drawers back in about 27 seconds, no re-embedding needed. I was honestly surprised this worked.

Local workaround (probably imperfect)

Two things I added locally. I'm not confident in either — they feel more like band-aids than real fixes:

• mine-safe.sh: a wrapper around mempalace mine that holds flock -x palace/mine.lock for the duration
• A patch to tool_add_drawer in mcp_server.py to check for the lock before upserting, backing off up to 30s if mine is running

It doesn't help when you have multiple MCP servers racing each other, but it at least stops mine and the MCP server from stepping on each other. Which is something.

Would I trust this setup long-term? Not yet. If anyone has a cleaner solution I'd genuinely like to hear it.

[guilhermefriol]

Adding another case (closing #1272 as duplicate of this one).

Environment

• mempalace 3.3.3 (pipx)
• chromadb 1.5.8
• Python 3.13.13
• macOS 26.4.1, Apple Silicon (M5 Max), 1.8 TB SSD

What hit me

Specifically the compress path, not the mine path: mempalace compress ran twice in one day (once manually, once via org.mempalace.compress.plist), with the MCP server and the save/precompact hooks active in parallel as described in the original report.

The mempalace_compressed collection's link_lists.bin ended up at:

• du -sh: 1.7 TB physical
• ls -lh: 17 TB logical (sparse)

For ~13K compressed entries. Disk went from 1.8 TB free to 3.7 GB free silently in hours.

Localized workaround

drop + recreate of mempalace_compressed at the start of each compress run, before the upsert loop. Prevents the resize loop entirely on the compress path; trade-off is re-vectorizing ~10K entries each run (a few minutes on the local ONNX backend).

The miner already has the equivalent guard for the same hnswlib behavior (delete-by-source_file before re-insert, miner.py:718). My PR brings the compress path in line. Doesn't fix the underlying chromadb behavior — that probably needs an upstream report in chroma-core/chroma — but it eliminates one practical disk-fill vector.

PR with the patch coming next.