feat: progressive disclosure + <private> tag filter

mempalace/instructions/help.md

@@ -28,8 +28,9 @@ AI memory system. Store everything, find anything. Local, free, no API key.
 - mempalace_get_aaak_spec -- Get the AAAK specification
 
 ### Palace (write)
-- mempalace_add_drawer -- Add a new memory (drawer)
+- mempalace_add_drawer -- Add a new memory (drawer). See "Privacy" below.
 - mempalace_delete_drawer -- Delete a memory (drawer)
+- mempalace_get_drawer -- Fetch full content for one or many drawer_ids
 
 ### Knowledge Graph
 - mempalace_kg_query -- Query the knowledge graph
@@ -66,6 +67,37 @@ AI memory system. Store everything, find anything. Local, free, no API key.
 
 ---
 
+## Privacy: `<private>` Tags
+
+Any content wrapped in `<private>...</private>` tags is stripped before
+storage. This applies to:
+
+- `mempalace_add_drawer` (content field)
+- `mempalace_diary_write` (entry field)
+
+Behavior:
+- Partial: `"public <private>secret</private> tail"` -> stored as
+  `"public  tail"`.
+- Full wrap: `"<private>all of this</private>"` -> write is refused with
+  `{success: false, reason: "content fully marked private"}`.
+- Case-insensitive, multi-line (DOTALL), non-greedy.
+
+Use this when capturing conversations or transcripts that contain passwords,
+tokens, personal data, or anything the user flags as sensitive. Existing
+drawers are not affected; this is a write-time filter only.
+
+---
+
+## Progressive Disclosure (search tool)
+
+`mempalace_search` returns **summaries by default** (drawer_id + ~30 char
+preview) to conserve tokens. Fetch full content only for relevant hits via
+`mempalace_get_drawer(drawer_id=[...])`. Pass `full=true` to `mempalace_search`
+to bypass and get verbatim text in one call. See `mempalace instructions
+search` for the full pattern.
+
+---
+
 ## Auto-Save Hooks
 
 - Stop hook -- Automatically saves memories every 15 messages. Counts human

mempalace/instructions/search.md

@@ -20,8 +20,16 @@ can discover the taxonomy first if needed.
 
 If MCP tools are available, use them in this priority order:
 
-- mempalace_search(query, wing, room) -- Primary search tool. Pass the semantic
-  query and any wing/room filters.
+- mempalace_search(query, wing, room, full=false) -- Primary search tool.
+  **Returns a SUMMARY by default** (drawer_id + ~30 char text preview per hit)
+  to conserve tokens. Follow up with `mempalace_get_drawer` using the
+  drawer_ids to fetch full verbatim content only for the hits you actually
+  need. Pass `full=true` to get verbatim text in one shot (bypasses progressive
+  disclosure; use only when auditing or when every hit matters).
+- mempalace_get_drawer(drawer_id) -- Fetch full verbatim content for one or
+  many drawer_ids. Accepts a single string **or an array of strings** for
+  batch fetch. Use this after `mempalace_search` to pull details for the
+  relevant hits.
 - mempalace_list_wings -- Discover all available wings. Use when the user asks
   what categories exist or you need to resolve a wing name.
 - mempalace_list_rooms(wing) -- List rooms within a specific wing. Use to help
@@ -34,6 +42,28 @@ If MCP tools are available, use them in this priority order:
   between two wings. Use when the user asks about relationships between
   different knowledge domains.
 
+### Progressive Disclosure Pattern (token-efficient)
+
+For most search tasks, follow this two-step flow:
+
+1. `mempalace_search(query="...")` → returns N hits, each with `drawer_id`
+   and a truncated `text` summary (marked `summary: true`). Cost: ~50-100
+   tokens total.
+2. Inspect summaries, pick the relevant drawer_ids, then call
+   `mempalace_get_drawer(drawer_id=["id1", "id2", ...])` to fetch full
+   content only for those. Cost: ~500-1000 tokens per drawer.
+
+This is **~10x cheaper** than loading full content for every hit, and keeps
+the context window lean when the search returns many weakly-relevant
+candidates.
+
+When to use `full=true` instead:
+- You know every hit will be needed (e.g., summarizing all memories on a
+  topic).
+- You are auditing search quality and want to see raw scores against full
+  text.
+- The query is narrow enough that you expect 1-3 hits.
+
 ## 4. CLI Fallback
 
 If MCP tools are not available, fall back to the CLI:

mempalace/mcp_server.py

@@ -433,6 +433,7 @@ def tool_search(
     max_distance: float = 1.5,
     min_similarity: float = None,
     context: str = None,
+    full: bool = False,
 ):
     limit = max(1, min(limit, _MAX_RESULTS))
     try:
@@ -465,6 +466,17 @@ def tool_search(
         }
     if context:
         result["context_received"] = True
+    # Progressive disclosure: by default return a compact summary so the
+    # caller can decide which drawers to fetch in full. Callers that need
+    # verbatim text pass full=True (or follow up with mempalace_get_drawer).
+    if not full and isinstance(result.get("results"), list):
+        from .privacy import summarize_for_search
+
+        for hit in result["results"]:
+            text = hit.get("text", "")
+            hit["text"] = summarize_for_search(text, 30)
+            hit["summary"] = True
+            hit.pop("closet_preview", None)
     return result
 
 
@@ -614,6 +626,19 @@ def tool_add_drawer(
     except ValueError as e:
         return {"success": False, "error": str(e)}
 
+    # Strip <private>...</private> blocks before they hit the embedding,
+    # the drawer ID hash, or the WAL. Refuse writes that are entirely
+    # wrapped in <private> tags.
+    from .privacy import redact_private
+
+    content, fully_private = redact_private(content)
+    if fully_private:
+        return {"success": False, "reason": "content fully marked private"}
+    try:
+        content = sanitize_content(content)
+    except ValueError as e:
+        return {"success": False, "error": str(e)}
+
     col = _get_collection(create=True)
     if not col:
         return _no_palace()
@@ -695,11 +720,44 @@ def tool_delete_drawer(drawer_id: str):
         return {"success": False, "error": str(e)}
 
 
-def tool_get_drawer(drawer_id: str):
-    """Fetch a single drawer by ID. Returns full content and metadata."""
+def tool_get_drawer(drawer_id):
+    """Fetch one or more drawers by ID.
+
+    ``drawer_id`` accepts a single string (returns ``{drawer_id, content,
+    wing, room, metadata}`` as before) or a list of strings (returns
+    ``{"drawers": [...]}`` with one entry per requested ID, including
+    ``{"drawer_id": ..., "error": "..."}`` for IDs that weren't found).
+    """
     col = _get_collection()
     if not col:
         return _no_palace()
+
+    # Batch form — caller passed a list/tuple of IDs.
+    if isinstance(drawer_id, (list, tuple)):
+        ids = [str(d) for d in drawer_id]
+        try:
+            result = col.get(ids=ids, include=["documents", "metadatas"])
+        except Exception as e:
+            return {"error": str(e)}
+        found = {}
+        for did, doc, meta in zip(
+            result.get("ids", []) or [],
+            result.get("documents", []) or [],
+            result.get("metadatas", []) or [],
+        ):
+            found[did] = {
+                "drawer_id": did,
+                "content": doc,
+                "wing": meta.get("wing", ""),
+                "room": meta.get("room", ""),
+                "metadata": meta,
+            }
+        drawers = [
+            found.get(did, {"drawer_id": did, "error": f"Drawer not found: {did}"}) for did in ids
+        ]
+        return {"drawers": drawers}
+
+    # Single-ID form — preserve the original response shape.
     try:
         result = col.get(ids=[drawer_id], include=["documents", "metadatas"])
         if not result["ids"]:
@@ -932,6 +990,18 @@ def tool_diary_write(agent_name: str, entry: str, topic: str = "general"):
     except ValueError as e:
         return {"success": False, "error": str(e)}
 
+    # Strip <private>...</private> before embedding/logging. Refuse
+    # diary entries that are entirely private.
+    from .privacy import redact_private
+
+    entry, fully_private = redact_private(entry)
+    if fully_private:
+        return {"success": False, "reason": "content fully marked private"}
+    try:
+        entry = sanitize_content(entry)
+    except ValueError as e:
+        return {"success": False, "error": str(e)}
+
     wing = f"wing_{agent_name.lower().replace(' ', '_')}"
     room = "diary"
     col = _get_collection(create=True)
@@ -1372,6 +1442,10 @@ def tool_reconnect():
                     "type": "string",
                     "description": "Background context for the search (optional). NOT used for embedding — only for future re-ranking.",
                 },
+                "full": {
+                    "type": "boolean",
+                    "description": "If true, return verbatim drawer text. Default false returns a ~30-char summary per hit plus drawer_id; fetch full content via mempalace_get_drawer.",
+                },
             },
             "required": ["query"],
         },
@@ -1425,11 +1499,15 @@ def tool_reconnect():
         "handler": tool_delete_drawer,
     },
     "mempalace_get_drawer": {
-        "description": "Fetch a single drawer by ID — returns full content and metadata.",
+        "description": "Fetch one or more drawers by ID — returns full content and metadata. Pass a single string for one drawer (existing shape) or a list of strings for batch fetch (returns {drawers: [...]}).",
         "input_schema": {
             "type": "object",
             "properties": {
-                "drawer_id": {"type": "string", "description": "ID of the drawer to fetch"},
+                "drawer_id": {
+                    "type": ["string", "array"],
+                    "items": {"type": "string"},
+                    "description": "Drawer ID, or list of IDs for batch fetch.",
+                },
             },
             "required": ["drawer_id"],
         },

mempalace/privacy.py

@@ -0,0 +1,104 @@
+"""
+privacy.py — Progressive disclosure + <private> tag handling.
+
+Two responsibilities:
+
+1. ``redact_private(text)`` — strip ``<private>...</private>`` blocks
+   (case-insensitive, multiline) from memory content before it gets
+   stored or embedded. Returns the cleaned string and a boolean
+   indicating whether the *entire* input was private (i.e. after
+   stripping + whitespace trim, nothing remains).
+
+2. ``summarize_for_search(text, n_chars)`` — collapse internal
+   whitespace to single spaces and truncate to ``n_chars`` code
+   points, backing up to the last space on ASCII boundaries so
+   words aren't chopped. Appends an ellipsis when truncated.
+
+These helpers are used by ``mcp_server.tool_search`` (progressive
+disclosure — default responses show a short summary), and by
+``tool_add_drawer`` / ``tool_diary_write`` (private-tag filtering
+before sanitize + embed).
+"""
+
+from __future__ import annotations
+
+import re
+
+# ``<private>...</private>`` — case-insensitive, dot matches newline,
+# non-greedy so multiple blocks in one string are each stripped
+# individually rather than merged.
+PRIVATE_TAG_RE = re.compile(r"<private>.*?</private>", re.DOTALL | re.IGNORECASE)
+
+# Whitespace run — used by summarize_for_search to collapse runs of
+# spaces/tabs/newlines to a single space before truncation.
+_WS_RUN_RE = re.compile(r"\s+")
+
+
+def redact_private(text: str) -> tuple[str, bool]:
+    """Strip ``<private>...</private>`` blocks from ``text``.
+
+    Returns a ``(cleaned, is_fully_private)`` tuple:
+
+    - ``cleaned`` — input with every ``<private>...</private>`` block
+      removed. Whitespace adjacent to stripped blocks is left intact
+      (callers can re-sanitize if they need tight whitespace rules).
+    - ``is_fully_private`` — True iff after stripping and calling
+      ``.strip()`` the remaining string is empty. Callers typically
+      reject the write when this is True so that fully-private
+      entries don't leak via metadata, drawer ID, or embedding.
+
+    The regex is case-insensitive (``<PRIVATE>`` works too) and
+    multiline (tags can span newlines).
+    """
+    if not isinstance(text, str):
+        raise TypeError("redact_private expects a str")
+
+    cleaned = PRIVATE_TAG_RE.sub("", text)
+    is_fully_private = cleaned.strip() == ""
+    return cleaned, is_fully_private
+
+
+def summarize_for_search(text: str, n_chars: int = 30) -> str:
+    """Collapse whitespace and truncate ``text`` to ``n_chars`` code points.
+
+    Behaviour:
+
+    - All internal whitespace runs (spaces, tabs, newlines) become a
+      single space.
+    - Leading/trailing whitespace is stripped.
+    - If the collapsed text is <= ``n_chars`` code points long, it is
+      returned as-is (no ellipsis appended).
+    - Otherwise it is truncated to ``n_chars`` code points. If the
+      truncation point sits mid-word on ASCII text, we back up to
+      the last space so words aren't chopped. For Chinese / CJK and
+      other scripts that don't use spaces between words, truncation
+      happens on the code-point boundary.
+    - A single ellipsis character (U+2026) is appended to signal
+      truncation.
+
+    ``n_chars`` must be a positive integer. Zero or negative values
+    return an empty string.
+    """
+    if not isinstance(text, str):
+        raise TypeError("summarize_for_search expects a str")
+    if n_chars <= 0:
+        return ""
+
+    collapsed = _WS_RUN_RE.sub(" ", text).strip()
+
+    if len(collapsed) <= n_chars:
+        return collapsed
+
+    truncated = collapsed[:n_chars]
+
+    # If the character immediately after the truncation point is an
+    # ASCII word character, we chopped mid-word — back up to the last
+    # space inside our window. Only applies when the surrounding text
+    # is ASCII; CJK scripts have no space-based word boundaries.
+    next_char = collapsed[n_chars]
+    if next_char.isascii() and (next_char.isalnum() or next_char == "_"):
+        last_space = truncated.rfind(" ")
+        if last_space > 0:
+            truncated = truncated[:last_space]
+
+    return truncated.rstrip() + "\u2026"

mempalace/searcher.py

@@ -387,7 +387,8 @@ def search_memories(
     CLOSET_DISTANCE_CAP = 1.5  # cosine dist > 1.5 = too weak to use as signal
 
     scored: list = []
-    for doc, meta, dist in zip(
+    for drawer_id, doc, meta, dist in zip(
+        _first_or_empty(drawer_results, "ids"),
         _first_or_empty(drawer_results, "documents"),
         _first_or_empty(drawer_results, "metadatas"),
         _first_or_empty(drawer_results, "distances"),
@@ -410,6 +411,10 @@ def search_memories(
 
         effective_dist = dist - boost
         entry = {
+            # drawer_id lets callers fetch full verbatim text via
+            # tool_get_drawer when search returns a progressive-disclosure
+            # summary instead of the raw drawer text.
+            "drawer_id": drawer_id,
             "text": doc,
             "wing": meta.get("wing", "unknown"),
             "room": meta.get("room", "unknown"),