docs: mark MemPalace#661/MemPalace#673/MemPalace#1021 merged upstream (2026-04-22)

jphein · claude · jphein · commit d53745a15d03 · 2026-04-21T19:07:43.000-07:00
Ben's batched queue-clear pass merged four PRs at 00:38 UTC: graph cache (MemPalace#661), deterministic hook saves (MemPalace#673), Claude Code 2.1.114 hook stdout + silent_save guard (MemPalace#1021), and upstream's own MemPalace#851 pagination fix (closing MemPalace#1036 as superseded). Moved four rows out of the "Fork Changes" / "Fork change queue" tables into their respective merged-upstream history sections. Intro sentence PR count reduced from 7 → 4 open. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -32,18 +32,18 @@ Ruff for linting (`ruff check`), line length 100, target Python 3.9.
 
 1. **feat: bulk_check_mined()** — paginated pre-fetch of all source_file/mtime pairs for concurrent mining (fork-only; independent of the mtime comparison fix, which has since been upstreamed)
 2. **feat: similarity threshold** — `max_distance` parameter in search, default 1.5 cosine distance in MCP
-3. **feat: hooks_cli silent save** — stop hook saves directly via Python API with systemMessage notification, deterministic, zero data loss
+3. ~~**feat: hooks_cli silent save**~~ — **merged upstream via #673 on 2026-04-22.** No longer fork-ahead.
 4. **feat: `mempal_save_hook.sh` Python auto-detection** — checks `MEMPAL_PYTHON` env var → repo venv → system `python3`; no hardcoded path required
 5. **fix: convo_miner wing assignment** — `_wing_from_transcript_path()` extracts project name from Claude Code transcript path
-6. **perf: graph cache** — `build_graph()` cached module-level with 60s TTL, invalidated on writes via `invalidate_graph_cache()`
+6. ~~**perf: graph cache**~~ — **merged upstream via #661 on 2026-04-22.** No longer fork-ahead.
 7. **perf: L1 importance pre-filter** — `_fetch_drawers()` tries `importance >= 3` first, falls back to full scan only if < 15 results
 8. **fix: MCP stale HNSW index** — `_get_client()` detects external writes via mtime (not just inode), `mempalace_reconnect` MCP tool
 9. **fix: diary wing assignment** — `tool_diary_write()` accepts optional `wing` param, stop hook derives project wing from transcript path
 10. **fix: `.blob_seq_ids_migrated` marker** — skip Python `sqlite3.connect()` against a live ChromaDB 1.5.x DB after first successful migration; opening the sqlite file from Python corrupts the next `PersistentClient` call
 11. ~~**feat: `quarantine_stale_hnsw()`**~~ — **merged upstream via #1000 in v3.3.2.** No longer fork-ahead.
 12. **feat: search warnings + sqlite BM25 top-up** — `search_memories()` returns `warnings: [...]` and `available_in_scope: N` whenever the vector path underdelivers (sparse HNSW after repair, `#951` filter-planner failure, drift). Fallback promotes BM25-ranked sqlite candidates tagged `matched_via: "sqlite_bm25_fallback"`. Closes the "silent 0-hit when data is in sqlite" failure mode. CLI `search()` delegates to `search_memories()` so both paths share the fallback.
-13. **fix: stop_hook_active guard** — guard only applies in block mode; silent mode skips it so Claude Code 2.1.114's plugin dispatch (which sets `stop_hook_active:true` on every fire after the first) doesn't suppress subsequent auto-saves
-14. **fix: `_output()` stdout routing** — uses `sys.modules.get()` to find an already-loaded `mcp_server` and reuse its `_REAL_STDOUT_FD`; otherwise writes directly to fd 1. Avoids importing `mcp_server` cold (which would trigger its stdout→stderr redirect as a side effect). Write-all loop handles partial `os.write()` returns.
+13. ~~**fix: stop_hook_active guard**~~ — **merged upstream via #1021 on 2026-04-22.** No longer fork-ahead.
+14. ~~**fix: `_output()` stdout routing**~~ — **merged upstream via #1021 on 2026-04-22.** No longer fork-ahead.
 15. **fix: `_get_client()` get-then-create guard** — `get_or_create_collection` segfaults ChromaDB 1.5.x when existing collection metadata differs; fork tries `get_collection` first, falls back to `create_collection` only on `InvalidCollectionException`.
 16. **perf: `miner.status()` paginated `col.get()`** — upstream's single `col.get(limit=total)` hits SQLite's max-variable limit on palaces with many thousands of drawers; fork paginates in 10 K-drawer batches.
 17. **feat: configurable chunking parameters** — `chunk_size` (800), `chunk_overlap` (100), `min_chunk_size` (50) written to `config.json` and exposed via `MempalaceConfig` properties.
@@ -56,6 +56,9 @@ Ruff for linting (`ruff check`), line length 100, target Python 3.9.
 - Unicode checkmark → ASCII for Windows encoding (#681, shipped in v3.3.2)
 - `quarantine_stale_hnsw()` for HNSW/sqlite drift (#1000, shipped in v3.3.2)
 - PID file guard prevents stacking mine processes, with Windows cross-platform `os.kill` fix (#1023, shipped in v3.3.2)
+- Graph cache with write-invalidation — `build_graph()` module-level cache with 60s TTL, `threading.Lock`, `invalidate_graph_cache()` on writes (#661, merged 2026-04-22)
+- Deterministic hook saves — silent mode via direct Python API call to `tool_diary_write()`, plain-text save, marker advances only after confirmed write, `systemMessage` terminal notification (#673, merged 2026-04-22). Replaces the block-mode "ask AI to save" pattern that could silently drop entries.
+- Hook `silent_save` guard + `_output()` stdout routing — silent-mode skips `stop_hook_active` guard so Claude Code 2.1.114 plugin dispatch keeps firing; `_output()` reuses already-loaded `mcp_server`'s `_REAL_STDOUT_FD` or writes directly to fd 1 to avoid cold-import side effects (#1021, merged 2026-04-22)
 
 ### Merged into upstream v3.3.0
 
@@ -88,17 +91,17 @@ Ruff for linting (`ruff check`), line length 100, target Python 3.9.
 
 ## Upstream PRs
 
-As of 2026-04-21: 10 merged, 7 open, 7 closed. PRs target `develop`. Fork `main` tracks `upstream/develop`.
+As of 2026-04-22: 13 merged, 4 open, 7 closed. PRs target `develop`. Fork `main` tracks `upstream/develop`.
 
 | PR | Status | Description |
 |----|--------|-------------|
 | #659 | open (`MERGEABLE`, waiting review) | Diary wing parameter |
 | #660 | open (`MERGEABLE`, waiting review) | L1 importance pre-filter |
-| #661 | open (`CHANGES_REQUESTED` but feedback addressed, waiting `@bensig` re-review — GitHub holds state until reviewer dismisses) | Graph cache with write-invalidation |
-| #673 | open (externally approved 2026-04-12, rebased + squashed on 2026-04-21, `MERGEABLE`) | Deterministic hook saves (broader than upstream's #966) |
 | #1005 | open (CI green all platforms, waiting maintainer) | Warnings + sqlite BM25 top-up when vector underdelivers (never silent miss) |
-| #1021 | open (CI green all platforms, waiting maintainer) | Hook stdout routing + silent_save guard fixes for Claude Code 2.1.114 |
 | #1024 | open (CI green all platforms, waiting maintainer) | Configurable chunk_size, chunk_overlap, min_chunk_size |
+| #661 | **merged** 2026-04-22 | Graph cache with write-invalidation |
+| #673 | **merged** 2026-04-22 | Deterministic hook saves (broader than upstream's #966) — config-flag-gated, strictly safer save semantics |
+| #1021 | **merged** 2026-04-22 | Hook stdout routing + `silent_save` guard fixes for Claude Code 2.1.114 |
 | #681 | **merged** in v3.3.2 (2026-04-21) | Unicode checkmark → ASCII (#535) |
 | #1000 | **merged** in v3.3.2 (2026-04-21) | `quarantine_stale_hnsw()` for HNSW/sqlite drift |
 | #1023 | **merged** in v3.3.2 (2026-04-21) | PID file guard prevents stacking mine processes + Windows `os.kill` cross-platform fix |
diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@
 
 ---
 
-Fork of [MemPalace](https://github.com/milla-jovovich/mempalace), tracking `upstream/develop` through the 2026-04-21 sync. Upstream shipped [v3.3.2](https://github.com/MemPalace/mempalace/releases/tag/v3.3.2) on 2026-04-21 — contains our #681, #1000, #1023. Running in production since 2026-04-09 — currently 165,632 drawers across 68 rooms in 28 wings, 7 open PRs upstream (#999 merged 2026-04-18; #681/#1000/#1023 released in v3.3.2; #1036 closed as duplicate of approved #851). See upstream README for full feature docs.
+Fork of [MemPalace](https://github.com/milla-jovovich/mempalace), tracking `upstream/develop` through the 2026-04-21 sync. Upstream shipped [v3.3.2](https://github.com/MemPalace/mempalace/releases/tag/v3.3.2) on 2026-04-21 — contains our #681, #1000, #1023. Running in production since 2026-04-09 — currently 165,632 drawers across 68 rooms in 28 wings, 4 open PRs upstream (#999 merged 2026-04-18; #681/#1000/#1023 in v3.3.2; #661/#673/#1021 merged 2026-04-22; #851 merged 2026-04-22 closing #1036). See upstream README for full feature docs.
 
 What this fork adds that you won't get from upstream yet: a **deterministic silent-save hook architecture** (zero data loss, `systemMessage` notification), **ChromaDB 1.5.x hardening** (`quarantine_stale_hnsw` drift recovery, segfault-trigger guards, 8-site `None`-metadata safety), and **search that never silently misses** (`search_memories` returns warnings + sqlite BM25 top-up + `available_in_scope` so callers can see what they aren't getting). Full list below.
 
@@ -32,14 +32,10 @@ Size (lines of diff) and Risk (maintainer-appetite + chance of a rework request)
 | **Performance** | `bulk_check_mined()` paginated pre-fetch + `--workers` ThreadPoolExecutor concurrent mining | [Issue #1088](https://github.com/MemPalace/mempalace/issues/1088) filed 2026-04-21; [cross-ref comment](https://github.com/MemPalace/mempalace/issues/1088#issuecomment-4292570126) ties it to [#357](https://github.com/MemPalace/mempalace/issues/357) (parallel-mining corruption we could fix) and gates the PR on [#1071](https://github.com/MemPalace/mempalace/pull/1071) landing first (ORT thread cap, for bounded parallelism). | medium | medium | `palace.py`, `miner.py` |
 | **Reliability** | `_get_client()` tries `get_collection` before `create_collection` — `get_or_create_collection` segfaults ChromaDB 1.5.x when the existing collection's metadata differs from the call-site metadata | [Issue #1089](https://github.com/MemPalace/mempalace/issues/1089) filed 2026-04-21 — documented the crash + fork workaround, cross-referenced [#974](https://github.com/MemPalace/mempalace/issues/974) / [#1071](https://github.com/MemPalace/mempalace/pull/1071) interaction (metadata drift risk post-merge), offered three paths: interim guard PR, chroma-core bug report, or close as covered. | small | medium | `backends/chroma.py` |
 | **Reliability** | Skip `_fix_blob_seq_ids` sqlite open after first successful migration via `.blob_seq_ids_migrated` marker — opening sqlite3 against a live ChromaDB 1.5.x file corrupts the next PersistentClient | [Issue #1090](https://github.com/MemPalace/mempalace/issues/1090) filed 2026-04-21 — documented the post-migration re-open crash pattern, fork's sentinel workaround, adjacent issues ([#722](https://github.com/MemPalace/mempalace/issues/722), [#832](https://github.com/MemPalace/mempalace/issues/832), [#1035](https://github.com/MemPalace/mempalace/issues/1035)), asked whether other palaces repro this. | small | medium | `backends/chroma.py` |
-| **Performance** | Graph cache — 60s TTL, invalidated on writes | [#661](https://github.com/milla-jovovich/mempalace/pull/661) | small | medium | `palace_graph.py` |
 | **Performance** | L1 importance pre-filter — `importance >= 3` first, full scan fallback | [#660](https://github.com/milla-jovovich/mempalace/pull/660) | small | low | `layers.py` |
-| **Performance** | `miner.status()` paginates `col.get()` in 10 K-drawer batches — upstream's single `col.get(limit=total)` hits SQLite's max-variable limit on palaces with many thousands of drawers | tracked upstream in [#851](https://github.com/milla-jovovich/mempalace/pull/851) (approved, MERGEABLE, also fixes #850); fork's paginated version has been running since 2026-04-10 | small | low | `miner.py` |
+| **Performance** | `miner.status()` paginates `col.get()` in 10 K-drawer batches — upstream's single `col.get(limit=total)` hits SQLite's max-variable limit on palaces with many thousands of drawers | tracked upstream in [#851](https://github.com/milla-jovovich/mempalace/pull/851) (merged 2026-04-22, also fixes #850 and #1015); fork's paginated version has been running since 2026-04-10 | small | low | `miner.py` |
 | **Config** | Configurable chunking parameters — `chunk_size` (default 800 chars), `chunk_overlap` (100), `min_chunk_size` (50) written to `config.json` and exposed via `MempalaceConfig` properties | [#1024](https://github.com/milla-jovovich/mempalace/pull/1024) · addresses [#390](https://github.com/MemPalace/mempalace/issues/390) (default 800 exceeds MiniLM's 256-token cap; this lets users override) | small | low | `config.py`, `miner.py`, `convo_miner.py` |
 | **Search** | Warnings + sqlite BM25 top-up when vector underdelivers — `search_memories` returns `warnings: [...]` and `available_in_scope: N` so callers see why recall was partial; fallback hits tagged `matched_via: "sqlite_bm25_fallback"`. The palace never silently returns fewer results than the scope contains (sibling of #951, addresses read-side of #823) | [#1005](https://github.com/milla-jovovich/mempalace/pull/1005) | medium | low | `searcher.py` |
-| **Hooks** | Silent save mode — direct Python API, deterministic, zero data loss; extracts 2–3 topic words from recent messages for the diary title; optional desktop toast via `notify-send` | [#673](https://github.com/milla-jovovich/mempalace/pull/673) · approved externally 2026-04-12, rebased + squashed 2026-04-21, `MERGEABLE` · closes [#854](https://github.com/MemPalace/mempalace/issues/854) (silent_save flag never read) | medium | low | `hooks_cli.py` |
-| **Hooks** | Honor silent_save when `stop_hook_active:true` — Claude Code 2.1.114 sets the flag on every plugin-dispatched Stop fire after the first, and the legacy block-mode loop guard was suppressing every subsequent auto-save (silent, no log entry, marker stuck). Fixed to only skip on the flag in block mode | [#1021](https://github.com/milla-jovovich/mempalace/pull/1021) | small | low | `hooks_cli.py` |
-| **Hooks** | Write hook JSON to real stdout via `sys.modules` lookup — `mempalace.mcp_server` redirects stdout→stderr at import to protect MCP stdio from ChromaDB C-level noise; `_output()` checks `sys.modules` for an already-loaded `mcp_server` and reuses its `_REAL_STDOUT_FD`, otherwise writes directly to fd 1. Avoids triggering the redirect as a side effect. | [#1021](https://github.com/milla-jovovich/mempalace/pull/1021) | small | low | `hooks_cli.py` |
 | **Features** | Diary wing routing — derive project wing from transcript path; `tool_diary_write` and `tool_diary_read` accept an optional `wing` parameter | [#659](https://github.com/milla-jovovich/mempalace/pull/659) | small | low | `hooks_cli.py`, `mcp_server.py` |
 
 ## What this looks like in practice
@@ -188,6 +184,15 @@ Merged history below. For what's in flight or pending, see the top-of-README "Fo
 
 ### Merged upstream (post-v3.3.1)
 
+**Merged 2026-04-22 (Ben's batched queue-clear pass at 00:38 UTC):**
+
+- Graph cache with write-invalidation — `build_graph()` module-level cache with 60s TTL, `threading.Lock`, `invalidate_graph_cache()` on writes ([#661](https://github.com/milla-jovovich/mempalace/pull/661))
+- Deterministic hook saves — silent mode via direct Python API call to `tool_diary_write()`, plain-text save, marker advances only after confirmed write, `systemMessage` terminal notification; config-flag-gated, strictly safer save semantics than the legacy block-mode "ask AI to save" pattern ([#673](https://github.com/milla-jovovich/mempalace/pull/673), closes #854)
+- Hook `silent_save` guard + `_output()` stdout routing — silent-mode skips `stop_hook_active` guard so Claude Code 2.1.114 plugin dispatch keeps firing; `_output()` reuses already-loaded `mcp_server`'s `_REAL_STDOUT_FD` or writes directly to fd 1 to avoid cold-import side effects ([#1021](https://github.com/milla-jovovich/mempalace/pull/1021))
+- `miner.status()` pagination — upstream's own fix for the `SQLITE_MAX_VARIABLE_NUMBER` crash on large palaces; triage sweep surfaced three-user confirmation data (#1015, #1016, sha2fiddy thread) that prioritized the merge over the superseded #1016 ([#851](https://github.com/milla-jovovich/mempalace/pull/851), closes #802/#850/#1015)
+
+**Merged earlier:**
+
 - `None`-metadata guards across 8 read-path loops — `searcher.py` (CLI + API + closet-boost), `miner.status()`, and 4 MCP handlers ([#999](https://github.com/milla-jovovich/mempalace/pull/999), merged 2026-04-18)
 
 **Released in [v3.3.2](https://github.com/MemPalace/mempalace/releases/tag/v3.3.2) on 2026-04-21:**
@@ -330,15 +335,12 @@ Tools and patterns we're evaluating for the two open problems above. Not competi
 |---|---|---|
 | [#659](https://github.com/milla-jovovich/mempalace/pull/659) | `MERGEABLE`, bensig review addressed 2026-04-19 (wing_ prefix + agent filter) | Diary wing parameter |
 | [#660](https://github.com/milla-jovovich/mempalace/pull/660) | `MERGEABLE`, waiting review | L1 importance pre-filter |
-| [#661](https://github.com/milla-jovovich/mempalace/pull/661) | feedback addressed (threading.Lock in 8adf35a), pinged 2026-04-18, waiting `@bensig` re-review. GitHub holds the `CHANGES_REQUESTED` state until the reviewer dismisses it — this does not mean the PR owes a response. | Graph cache with write-invalidation |
-| [#673](https://github.com/milla-jovovich/mempalace/pull/673) | APPROVED externally 2026-04-12, rebased fresh on `upstream/develop` + squashed to 1 commit 2026-04-21, `MERGEABLE` | Deterministic hook saves (broader than upstream's narrower #966) |
 | [#1005](https://github.com/milla-jovovich/mempalace/pull/1005) | CI green (all platforms), Copilot + Dialectician review addressed, waiting maintainer review | Warnings + sqlite BM25 top-up — never silently return fewer results than scope contains |
-| [#1021](https://github.com/milla-jovovich/mempalace/pull/1021) | bensig review addressed 2026-04-19 (`silent_guard` default), CI green | Hook stdout routing + silent_save guard fixes for Claude Code 2.1.114 |
 | [#1024](https://github.com/milla-jovovich/mempalace/pull/1024) | CI green, filed 2026-04-18 | Configurable `chunk_size` / `chunk_overlap` / `min_chunk_size` |
 
-Merged since v3.3.1: [#999](https://github.com/milla-jovovich/mempalace/pull/999) (2026-04-18), plus [#681](https://github.com/milla-jovovich/mempalace/pull/681), [#1000](https://github.com/milla-jovovich/mempalace/pull/1000), [#1023](https://github.com/milla-jovovich/mempalace/pull/1023) all shipped in [v3.3.2](https://github.com/MemPalace/mempalace/releases/tag/v3.3.2) (2026-04-21).
+Merged since v3.3.1: [#999](https://github.com/milla-jovovich/mempalace/pull/999) (2026-04-18), [#681](https://github.com/milla-jovovich/mempalace/pull/681) / [#1000](https://github.com/milla-jovovich/mempalace/pull/1000) / [#1023](https://github.com/milla-jovovich/mempalace/pull/1023) in [v3.3.2](https://github.com/MemPalace/mempalace/releases/tag/v3.3.2) (2026-04-21), and [#661](https://github.com/milla-jovovich/mempalace/pull/661) / [#673](https://github.com/milla-jovovich/mempalace/pull/673) / [#1021](https://github.com/milla-jovovich/mempalace/pull/1021) (2026-04-22).
 
-Closed: [#626](https://github.com/milla-jovovich/mempalace/pull/626), [#633](https://github.com/milla-jovovich/mempalace/pull/633), [#662](https://github.com/milla-jovovich/mempalace/pull/662) (superseded by BM25), [#663](https://github.com/milla-jovovich/mempalace/pull/663) (upstream wrote [#757](https://github.com/milla-jovovich/mempalace/pull/757)), [#738](https://github.com/milla-jovovich/mempalace/pull/738) (docs stale), [#629](https://github.com/milla-jovovich/mempalace/pull/629) (superseded — upstream shipped batching + file locking), [#632](https://github.com/milla-jovovich/mempalace/pull/632) (superseded — `--version`, `purge`, `repair` all shipped in v3.3.0), [#1036](https://github.com/milla-jovovich/mempalace/pull/1036) (superseded by #851 which was already approved, also fixes #850).
+Closed: [#626](https://github.com/milla-jovovich/mempalace/pull/626), [#633](https://github.com/milla-jovovich/mempalace/pull/633), [#662](https://github.com/milla-jovovich/mempalace/pull/662) (superseded by BM25), [#663](https://github.com/milla-jovovich/mempalace/pull/663) (upstream wrote [#757](https://github.com/milla-jovovich/mempalace/pull/757)), [#738](https://github.com/milla-jovovich/mempalace/pull/738) (docs stale), [#629](https://github.com/milla-jovovich/mempalace/pull/629) (superseded — upstream shipped batching + file locking), [#632](https://github.com/milla-jovovich/mempalace/pull/632) (superseded — `--version`, `purge`, `repair` all shipped in v3.3.0), [#1036](https://github.com/milla-jovovich/mempalace/pull/1036) (superseded by #851 which merged 2026-04-22, also fixes #850).
 
 ## Setup
 
