@@ -4,150 +4,198 @@ Fork-ahead changes that aren't yet in upstream `MemPalace/mempalace`.
44Upstream's release history lives in [ ` CHANGELOG.md ` ] ( CHANGELOG.md ) ;
55this file is the supplement.
66
7+ > ** This file is generated.** Edit ` docs/fork-changes.yaml ` and run
8+ > ` scripts/render-docs.py ` to regenerate. Hand-edits will be
9+ > overwritten on the next render.
10+
711Date-based sections, not semver — the fork tracks ` upstream/develop ` and
812doesn't cut its own release tags. When a fork-ahead row lands upstream,
9- the entry is moved to the ** Merged into upstream** section at the
10- bottom (kept for ~ 2 weeks , then trimmed).
13+ move the entry to the ** Merged into upstream** section at the bottom
14+ (kept ~ 30 days , then trimmed).
1115
1216The format is based on [ Keep a Changelog] ( https://keepachangelog.com/en/1.1.0/ ) .
1317
1418---
1519
20+
1621## [ 2026-04-26]
1722
23+
1824### Added
1925
20- - ** ` mempalace_session_recovery_read ` MCP tool** — reads the dedicated
21- ` mempalace_session_recovery ` collection by optional ` session_id ` ,
22- ` agent ` , ` since ` , ` until ` , ` wing ` , ` limit ` filters; returns entries
23- newest-first. Used for hook auditing and "what was I doing 2 hours
24- ago" recovery. Registered in the ` TOOLS ` dict and documented in
25- ` website/reference/mcp-tools.md ` . ([ ` e266365 ` ] ( https://github.com/jphein/mempalace/commit/e266365 ) )
26- - ** ` mempalace repair --mode reorganize ` ** — explicit operator command
27- to migrate existing ` topic=checkpoint ` drawers from the main
28- collection to ` mempalace_session_recovery ` . Idempotent, ID and
29- metadata preserving. ([ ` 42817d7 ` ] ( https://github.com/jphein/mempalace/commit/42817d7 ) )
30- - ** ` scripts/deploy.sh ` ** — one-command push + Syncthing-aware redeploy
31- to the canonical disks daemon (` systemctl --user restart palace-daemon `
32- + post-restart import check that today\' s fork-ahead surface is
33- loaded). ([ ` 8252025 ` ] ( https://github.com/jphein/mempalace/commit/8252025 ) )
34- - ** ` drawer_id ` field** on ` mempalace_search ` , ` mempalace_diary_read ` ,
35- and ` mempalace_session_recovery_read ` payloads — chromadb\' s primary
36- key was always returned by ` query() ` / ` get() ` but never plumbed into
37- the result dicts; consumers (e.g. citation popovers) can now follow
38- a hit back to the underlying drawer via ` mempalace_get_drawer ` .
39- ([ ` 9a8bb77 ` ] ( https://github.com/jphein/mempalace/commit/9a8bb77 ) )
40-
41- ### Changed
42-
43- - ** ` tool_diary_write ` routes ` topic in _CHECKPOINT_TOPICS ` to a
44- dedicated collection** (` mempalace_session_recovery ` ). Everything else
45- stays in ` mempalace_drawers ` . The main collection is now the
46- * verbatim store* — chats, tool calls, mined files — and is no longer
47- polluted by Stop-hook auto-save fragments dominating vector top-N.
48- ([ ` e266365 ` ] ( https://github.com/jphein/mempalace/commit/e266365 ) )
49- - ** ` hook_precompact ` writes a session-recovery marker** before mining
50- the transcript and allowing compaction. Mirrors ` hook_stop ` \' s
51- ` _save_diary_direct ` call so a context-compaction event leaves a
52- queryable timestamp in the recovery collection rather than nothing.
53- ([ ` 42817d7 ` ] ( https://github.com/jphein/mempalace/commit/42817d7 ) )
54- - ** ` tool_diary_write ` accepts an optional ` session_id ` ** parameter,
55- stored in metadata when a checkpoint is being written so the new
56- recovery-read tool can filter by Claude Code session.
57- ([ ` e266365 ` ] ( https://github.com/jphein/mempalace/commit/e266365 ) )
26+
27+ - ** Phase D migration + PreCompact recovery write** ([ ` 42817d7 ` ] ( https://github.com/jphein/mempalace/commit/42817d7 ) )
28+ `` migrate_checkpoints_to_recovery(palace_path, batch_size=1000) `` walks
29+ the main collection in pages, filters drawers with topic in
30+ `` _CHECKPOINT_TOPICS `` in Python (avoids the chromadb 1.5.x `` $in `` /`` $nin ``
31+ filter-planner bug), copies them to the recovery collection
32+ (preserving IDs + metadata), then deletes from main. Idempotent —
33+ re-running on a fully-reorganized palace returns 0. Add-then-delete
34+ order: a crash mid-migration leaves a duplicate, not a loss.
35+ Wired into `` mempalace repair --mode reorganize `` for explicit operator
36+ runs. PreCompact incorporated — `` hook_precompact `` now writes a
37+ session-recovery marker mirroring Stop, so context-compaction events
38+ leave a queryable timestamp in the recovery collection rather than
39+ nothing. Failures are non-fatal (logged; mining + compaction still
40+ proceed).
41+
42+ * Tests:* 6 in TestMigrateCheckpointsToRecovery + 1 in test_hooks_cli
43+ * Files:* ` mempalace/migrate.py ` , ` mempalace/cli.py ` , ` mempalace/hooks_cli.py ` , ` tests/test_migrate.py `
44+
45+
46+ - ** Surface drawer_id in search/diary/recovery payloads** ([ ` 9a8bb77 ` ] ( https://github.com/jphein/mempalace/commit/9a8bb77 ) )
47+ ChromaDB's primary key was always returned by `` query() `` and `` get() ``
48+ but never plumbed into result-building loops; consumers (e.g.
49+ familiar.realm.watch's citation-popover loop) couldn't link a hit
50+ back to the underlying drawer. Three call sites updated for parity:
51+ `` searcher.search_memories `` (vector path + sqlite BM25 fallback),
52+ `` mcp_server.tool_session_recovery_read `` , `` mcp_server.tool_diary_read `` .
53+ Defensive zip with id-pad: production chromadb always returns ids,
54+ but several test mocks omit them — pad with `` None `` when absent so
55+ existing fixtures keep working without touching N tests.
56+
57+ * Tests:* 1 integration + 1 inline assertion
58+ * Files:* ` mempalace/searcher.py ` , ` mempalace/mcp_server.py ` , ` website/reference/mcp-tools.md `
59+
60+
61+ - ** scripts/deploy.sh — one-command Syncthing-aware redeploy** ([ ` 8252025 ` ] ( https://github.com/jphein/mempalace/commit/8252025 ) )
62+ Single command does the right shape: push fork main → wait for
63+ Syncthing to reach `` /mnt/raid/projects/memorypalace `` on the deploy
64+ host → `` systemctl --user restart palace-daemon `` → poll `` /health `` →
65+ ssh-import-check that today's fork-ahead surface is loaded.
66+ Replaces a three-step manual ritual that was easy to get wrong
67+ (e.g. `` pip install --upgrade `` was a no-op on the editable install).
68+
69+ * Files:* ` scripts/deploy.sh `
70+
5871
5972### Fixed
6073
61- - ** ` quarantine_stale_hnsw ` no longer destroys healthy indexes on cold
62- start.** Two-stage gate: (1) mtime gap > threshold (existing) AND
63- (2) ` _segment_appears_healthy ` integrity sniff-test on
64- the chromadb segment metadata file (new — checks for chromadb\' s
65- protocol/terminator bytes without deserializing). Production case
66- 2026-04-26 06:56:45 had three healthy 253MB segments quarantined on
67- cold start by mtime alone (chromadb 1.5.x flushes HNSW asynchronously;
68- clean shutdown does not force-flush, so the on-disk gap is the steady
69- state, not corruption). The integrity gate distinguishes flush-lag
70- from corruption. ([ ` 645ba20 ` ] ( https://github.com/jphein/mempalace/commit/645ba20 ) )
71- - ** ` make_client() ` only invokes ` quarantine_stale_hnsw ` once per palace
72- per process.** Previously, every reconnect under steady write load
73- re-fired the proactive check, racking up ` .drift-* ` directories every
74- 10–30 minutes. The cold-start gate (` ChromaBackend._quarantined_paths ` )
75- caps it to one fire on first open; runtime drift detection still
76- works via palace-daemon\' s ` _auto_repair ` , which calls
77- ` quarantine_stale_hnsw ` directly. ([ ` 70c4bc6 ` ] ( https://github.com/jphein/mempalace/commit/70c4bc6 ) )
74+
75+ - ** Integrity gate prevents quarantine_stale_hnsw from destroying healthy indexes** ([ ` 645ba20 ` ] ( https://github.com/jphein/mempalace/commit/645ba20 ) )
76+ Previous behavior fired whenever `` sqlite_mtime - hnsw_mtime `` exceeded
77+ the (lowered, in #1173 ) 300s threshold. ChromaDB 1.5.x flushes HNSW
78+ asynchronously and a clean shutdown does not force-flush, so the
79+ on-disk HNSW is always meaningfully older than `` chroma.sqlite3 `` —
80+ that's the steady state, not corruption. Quarantine renamed valid
81+ HNSW segments on every cold-start; chromadb created empty replacements;
82+ vector recall went to 0/N until rebuild. Confirmed in production on
83+ the disks daemon journal 2026-04-26 06:56:45: three of three healthy
84+ 253MB segments quarantined on cold-start with 538-557s gaps. Fix:
85+ stage 2 integrity gate sniffs the chromadb segment metadata file
86+ for its protocol/terminator bytes (PROTO `` \x80 `` head, STOP `` \x2e ``
87+ tail) and a non-trivial size, ** without deserializing** . Healthy
88+ segment with mtime drift → keep in place; truncated/zero-filled →
89+ quarantine.
90+
91+ * Tests:* 4 in test_backends.py (renames-corrupt, leaves-healthy-with-drift, leaves-no-metadata, renames-truncated)
92+ * Upstream:* [ PR #1173 ] ( https://github.com/MemPalace/mempalace/pull/1173 ) (OPEN)
93+ * Files:* ` mempalace/backends/chroma.py ` , ` tests/test_backends.py `
94+
7895
7996### Performance
8097
81- - ** Cherry-picked upstream PR [ #1085 ] ( https://github.com/MemPalace/mempalace/pull/1085 ) **
82- (@midweste , OPEN as of 2026-04-26) — batch ChromaDB inserts in
83- ` miner.process_file() ` . New ` _build_drawer() ` helper + ` add_drawers() `
84- batch-insert path; one ` collection.upsert ` + one ONNX embedding pass
85- per sub-batch instead of per-chunk. Hoists ` datetime.now() ` and
86- ` os.path.getmtime() ` to file-level (2 syscalls per file instead of
87- 2N). Reported 10–30× mining speedup upstream. Fork-side resolution
88- preserved fork\' s existing ` DRAWER_UPSERT_BATCH_SIZE=1000 ` ; aliased
89- upstream\' s ` CHROMA_BATCH_LIMIT ` to it. ([ ` 6be6fff ` ] ( https://github.com/jphein/mempalace/commit/6be6fff ) )
90- * Becomes a no-op when #1085 merges to develop and we next sync.*
9198
92- ---
99+ - ** Cherry-pick #1085 — batch ChromaDB inserts in miner (10–30× faster)** ([ ` 6be6fff ` ] ( https://github.com/jphein/mempalace/commit/6be6fff ) )
100+ Cherry-picked from upstream PR
101+ [ #1085 ] ( https://github.com/MemPalace/mempalace/pull/1085 ) (@midweste ,
102+ OPEN as of 2026-04-26). New `` _build_drawer() `` helper + `` add_drawers() ``
103+ batch-insert path; `` process_file `` hands the full chunk list to
104+ `` add_drawers `` instead of looping per-chunk. Hoists `` datetime.now() ``
105+ and `` os.path.getmtime() `` to file-level (2 syscalls per file instead
106+ of 2N). Reported 10–30× mining speedup upstream. Fork-side resolution
107+ preserved fork's existing `` DRAWER_UPSERT_BATCH_SIZE=1000 `` ; aliased
108+ upstream's `` CHROMA_BATCH_LIMIT `` to it. Becomes a no-op when #1085
109+ merges to develop and we next sync.
110+
111+ * Upstream:* [ PR #1085 ] ( https://github.com/MemPalace/mempalace/pull/1085 ) (OPEN)
112+ * Files:* ` mempalace/miner.py `
113+
93114
94115## [ 2026-04-25]
95116
117+
96118### Added
97119
98- - ** Phases A–C of the checkpoint collection split** — new
99- ` mempalace_session_recovery ` collection adapter
100- (` _SESSION_RECOVERY_COLLECTION ` + ` get_session_recovery_collection `
101- in ` palace.py ` ); ` tool_diary_write ` routes ` topic in _CHECKPOINT_TOPICS `
102- to it. Promoted from "future work" to "necessary" by the same-day
103- Cat 9 A/B (` kind=all ` 632 tokens/Q vs ` kind=content ` 3 tokens/Q on
104- the canonical 151K-drawer palace). 12 new tests across
105- ` tests/test_session_recovery.py ` + ` TestCheckpointRouting ` +
106- ` TestSessionRecoveryRead ` . Design doc:
107- ` docs/superpowers/specs/2026-04-25-checkpoint-collection-split.md ` ;
108- TDD plan:
109- ` docs/superpowers/plans/2026-04-25-checkpoint-collection-split-impl.md ` .
110- ([ ` e266365 ` ] ( https://github.com/jphein/mempalace/commit/e266365 ) )
120+
121+ - ** Phases A–C of the checkpoint collection split** ([ ` e266365 ` ] ( https://github.com/jphein/mempalace/commit/e266365 ) )
122+ New `` mempalace_session_recovery `` collection adapter
123+ (`` _SESSION_RECOVERY_COLLECTION `` + `` get_session_recovery_collection ``
124+ in `` palace.py `` ); `` tool_diary_write `` routes `` topic in _CHECKPOINT_TOPICS ``
125+ to it. New `` mempalace_session_recovery_read `` MCP tool reads recovery
126+ collection only with optional filters (session_id, agent, since,
127+ until, wing, limit). Promoted from "future work" to "necessary" by
128+ the same-day Cat 9 A/B (`` kind=all `` 632 tokens/Q vs `` kind=content ``
129+ 3 tokens/Q on the canonical 151K-drawer palace). Design doc at
130+ `` docs/superpowers/specs/2026-04-25-checkpoint-collection-split.md `` .
131+
132+ * Tests:* 12 across test_session_recovery.py + TestCheckpointRouting + TestSessionRecoveryRead
133+ * Files:* ` mempalace/palace.py ` , ` mempalace/mcp_server.py ` , ` tests/test_session_recovery.py ` , ` tests/test_mcp_server.py ` , ` website/reference/mcp-tools.md `
134+
111135
112136### Fixed
113137
114- - ** ` palace_graph.build_graph ` skips ` None ` metadata.**
115- ` palace_graph.py:95 ` was calling ` meta.get("room", "") `
116- unconditionally; ChromaDB returns ` None ` for legacy/partial-write
117- drawers, taking out every consumer of ` build_graph ` (graph_stats,
118- find_tunnels, traverse, the daemon\' s ` /stats ` ). Caught by
119- palace-daemon\' s ` verify-routes.sh ` smoke test. Filed as upstream
120- [ #1201 ] ( https://github.com/MemPalace/mempalace/pull/1201 ) .
121- ([ ` 5fd15db ` ] ( https://github.com/jphein/mempalace/commit/5fd15db ) )
122- - ** ` kind= ` filter on ` search_memories ` excludes Stop-hook
123- checkpoints by default** — surgical fix while the structural split
124- was being designed. Three values: ` "content" ` (default, excludes),
125- ` "checkpoint" ` (recovery/audit only), ` "all" ` (no filter). Two
126- same-day architecture corrections: (a) the where-clause filter
127- (` topic $nin [...] ` ) tripped a chromadb 1.5.x filter-planner bug
128- that returned ` Internal error: Error finding id ` on every
129- ` kind=content ` vector query, so the exclusion moved to post-filter
130- only ([ ` 398f42f ` ] ( https://github.com/jphein/mempalace/commit/398f42f ) );
138+
139+ - ** Gate quarantine_stale_hnsw to once-per-palace-per-process** ([ ` 70c4bc6 ` ] ( https://github.com/jphein/mempalace/commit/70c4bc6 ) )
140+ `` make_client() `` previously invoked `` quarantine_stale_hnsw `` on every
141+ reconnect; under steady write load the proactive check kept firing,
142+ racking up `` .drift-* `` directories every 10–30 minutes. New
143+ `` ChromaBackend._quarantined_paths: set[str] `` caps it to one fire on
144+ first open per palace per process. Real cold-start drift still caught
145+ (replicated/restored palace); real runtime errors still caught via
146+ palace-daemon's `` _auto_repair `` , which calls `` quarantine_stale_hnsw ``
147+ directly and bypasses this gate.
148+
149+ * Tests:* 2 in test_backends.py (single-fire-per-palace, per-palace independence)
150+ * Upstream:* [ PR #1173 ] ( https://github.com/MemPalace/mempalace/pull/1173 ) (OPEN)
151+ * Files:* ` mempalace/backends/chroma.py ` , ` tests/test_backends.py ` , ` tests/conftest.py `
152+
153+
154+ - ** palace_graph.build_graph skips None metadata** ([ ` 5fd15db ` ] ( https://github.com/jphein/mempalace/commit/5fd15db ) )
155+ `` palace_graph.py:95 `` was calling `` meta.get("room", "") `` unconditionally;
156+ ChromaDB returns `` None `` for legacy/partial-write drawers, taking out
157+ every consumer of `` build_graph `` (graph_stats, find_tunnels, traverse,
158+ the daemon's `` /stats `` ). Caught by palace-daemon's `` verify-routes.sh ``
159+ smoke test. Same family as upstream's #999 None-metadata audit, in a
160+ read path the audit didn't reach.
161+
162+ * Upstream:* [ PR #1201 ] ( https://github.com/MemPalace/mempalace/pull/1201 ) (OPEN)
163+ * Files:* ` mempalace/palace_graph.py `
164+
165+
166+ - ** kind= filter on search_memories excludes Stop-hook checkpoints (transitional)** ([ ` f9f5cc4 ` ] ( https://github.com/jphein/mempalace/commit/f9f5cc4 ) )
167+ Three values: `` "content" `` (default, excludes), `` "checkpoint" ``
168+ (recovery/audit only), `` "all" `` (no filter). Two same-day architecture
169+ corrections: (a) the where-clause filter (`` topic $nin [...] `` ) tripped
170+ a chromadb 1.5.x filter-planner bug; the exclusion moved to post-filter
171+ only ([ 398f42f] ( https://github.com/jphein/mempalace/commit/398f42f ) );
131172 (b) vector top-N is dominated by checkpoints on this palace, so
132- post-filter alone empties the result set without aggressive
133- over-fetch — pull size raised to ` max(n*20, 100) ` for ` kind != "all" `
134- ([ ` f9f5cc4 ` ] ( https://github.com/jphein/mempalace/commit/f9f5cc4 ) ).
135- 9 tests in ` TestCheckpointFilter ` . * This is the safety net during
136- the transition; once Phase D ships and existing checkpoints
137- migrate, the post-filter and over-fetch hack become deletable.*
173+ post-filter alone empties the result set without aggressive over-fetch
174+ — pull size raised to `` max(n*20, 100) `` for `` kind != "all" `` (this commit).
175+ Safety net during the transition; once Phase D ships and existing
176+ checkpoints migrate, the post-filter and over-fetch hack become
177+ deletable.
178+
179+ * Tests:* 9 in TestCheckpointFilter
180+ * Files:* ` mempalace/searcher.py ` , ` mempalace/mcp_server.py ` , ` tests/test_searcher.py `
181+
138182
139183---
140184
141185## Merged into upstream (recent)
142186
143- Trimmed monthly. See [ ` CHANGELOG.md ` ] ( CHANGELOG.md ) for the full
144- released history.
145-
146- - ** PR #659 ** — diary ` wing ` parameter (merged 2026-04-23)
147- - ** PR #661 ** — graph cache with write-invalidation (merged 2026-04-22)
148- - ** PR #673 ** — deterministic hook saves (merged 2026-04-22)
149- - ** PR #1021 ** — Claude Code 2.1.114 stdout/silent_save fixes (merged 2026-04-22)
150- - ** PR #999 ** — ` None ` -metadata guards across read paths (merged 2026-04-18)
151- - ** PR #1000 ** — ` quarantine_stale_hnsw ` (shipped in v3.3.2)
152- - ** PR #1023 ** — PID file guard (shipped in v3.3.2)
153- - ** PR #681 ** — Unicode checkmark → ASCII (shipped in v3.3.2)
187+
188+ * Trim entries from this list once they're more than ~ 30 days old.*
189+
190+
191+ * See CHANGELOG.md (upstream) for the full released history.*
192+
193+
194+ - [ PR #659 ] ( https://github.com/MemPalace/mempalace/pull/659 ) — diary ` wing ` parameter — 2026-04-23
195+ - [ PR #661 ] ( https://github.com/MemPalace/mempalace/pull/661 ) — graph cache with write-invalidation — 2026-04-22
196+ - [ PR #673 ] ( https://github.com/MemPalace/mempalace/pull/673 ) — deterministic hook saves — 2026-04-22
197+ - [ PR #1021 ] ( https://github.com/MemPalace/mempalace/pull/1021 ) — Claude Code 2.1.114 stdout/silent_save fixes — 2026-04-22
198+ - [ PR #999 ] ( https://github.com/MemPalace/mempalace/pull/999 ) — None-metadata guards across read paths — 2026-04-18
199+ - [ PR #1000 ] ( https://github.com/MemPalace/mempalace/pull/1000 ) — quarantine_stale_hnsw shipped — v3.3.2
200+ - [ PR #1023 ] ( https://github.com/MemPalace/mempalace/pull/1023 ) — PID file guard prevents stacking mine processes — v3.3.2
201+ - [ PR #681 ] ( https://github.com/MemPalace/mempalace/pull/681 ) — Unicode checkmark → ASCII — v3.3.2
0 commit comments