feat(init): optional LLM-assisted entity refinement + Claude Code convo scanner (phase 2)

[igorls]

Summary

Implements #1149 — phase-2 of the init entity detection work, stacked on top of #1148.

Adds an opt-in LLM refinement step (mempalace init --llm) that takes the candidate set produced by phase-1 detection and reclassifies each entity as PERSON / PROJECT / TOPIC / COMMON_WORD / AMBIGUOUS. Default behaviour is unchanged — no LLM, no network, no API keys required.

Also adds a deterministic convo_scanner that parses ~/.claude/projects/ session directories into project entities by reading each session's cwd metadata, avoiding the lossy slug-decoding problem.

Why stacked on #1148

The two PRs build incrementally:

• feat(init): scan manifests and git authors for real entity signal (v1) #1148 replaces regex-only detection with manifest + git-author signal
• feat(init): optional LLM-assisted entity classification (phase 2) #1149 adds LLM refinement on top of that candidate set

Reviewing them together is easier than reviewing #1149 in isolation. Merge order: #1148 first, then this one rebases onto develop.

What's in this PR

mempalace/convo_scanner.py — Parse Claude Code conversation directories into ProjectInfo.

• Reads cwd from session JSONL records for accurate project names (slug decoding is lossy — can't distinguish foo-bar one-segment from foo/bar two-segments).
• Falls back to slug-decoding only if JSONL is malformed.
• Dedup by name, prefers the entry with more sessions.

mempalace/llm_client.py — Pluggable provider abstraction, no external SDKs (stdlib urllib only).

• ollama (default local, zero-API).
• openai-compat for OpenRouter, LM Studio, llama.cpp server, vLLM, Groq, Together, Fireworks.
• anthropic for the Messages API.
• JSON-mode plumbing normalized across providers.
• Fast check_available() probe before first use.

mempalace/llm_refine.py — The refinement step.

• Batches 25 candidates per call, collects up to 3 context lines each capped at 240 chars — bounds total input to ~50-100K tokens on any corpus size.
• Interactive progress on stderr (overwrite-line bar with current candidate name).
• Ctrl-C cancels cleanly and returns whatever was classified so far; partial result is safe to pass straight to confirm_entities.
• Defensive response parser: handles label/type keys, case variants, top-level list vs wrapped object, markdown code fences. Unknown labels become AMBIGUOUS so the user reviews them.
• Manifest-backed projects (conf >= 0.95) and git-authored people are not sent to the LLM — they're already authoritative.

CLI flags (opt-in, default zero-API):

mempalace init <dir> --llm
mempalace init <dir> --llm --llm-provider ollama --llm-model gemma4:e4b
mempalace init <dir> --llm --llm-provider openai-compat \
                     --llm-endpoint http://localhost:1234/v1 \
                     --llm-model <model>
mempalace init <dir> --llm --llm-provider anthropic \
                     --llm-model claude-haiku-4-5

API keys fall back to $OPENAI_API_KEY / $ANTHROPIC_API_KEY when not passed explicitly.

Tests

76 new unit tests across the three new modules, all running offline with mocked HTTP and a FakeProvider. Covers:

• Manifest/JSONL parsing with malformed-input tolerance
• Every provider's JSON-mode plumbing and error paths
• Prompt construction and context collection
• Response parser variants (label vs type, code fences, canonical casing, top-level list)
• End-to-end refine + Ctrl-C partial-result + error-tolerant batching

Full suite: all existing tests still pass; ruff clean.

Known limits / future work

• Response time on local models: Small models (4B) take ~5-10s per batch; a corpus with 100+ candidates takes several minutes. The progress bar mitigates the UX impact; user can Ctrl-C if the wait is unwanted.
• No streaming: Providers are called in non-streaming mode to keep the JSON parser simple. Fine for structured output; streaming would only help UX, not correctness.
• Single-turn only: No multi-turn refinement where the LLM asks clarifying questions. Not needed at init time — the regex pass has already narrowed the space.
• Context is line-based: For transcripts where one "line" is a whole message, we may truncate aggressively. Acceptable tradeoff for token bounds; an improvement would be token-window sampling.

Test plan

• uv run pytest tests/ --ignore=tests/benchmarks — full suite passes
• ruff check mempalace/ tests/ — clean
• ruff format --check mempalace/ tests/ — clean
• CLI flags parse: mempalace init --help shows all --llm-* options
• Local-provider smoke test: classify a small candidate set against Ollama and verify JSON round-trip
• Reviewer verification with a different model and/or provider (e.g. openai-compat to LM Studio or OpenRouter)
• Soak test on a large prose corpus (transcripts) to validate scale assumptions and tune batch size if needed

[Copilot]

Pull request overview

Adds an opt-in, provider-pluggable LLM refinement step to improve mempalace init entity classification for prose-heavy corpora, and introduces a deterministic Claude Code conversation scanner to extract project names from ~/.claude/projects/ sessions (using per-session cwd metadata rather than lossy slug decoding).

Changes:

• Add mempalace init --llm with provider/model/endpoint/api-key flags and provider availability checks.
• Introduce mempalace/llm_client.py (Ollama, OpenAI-compatible, Anthropic) and mempalace/llm_refine.py (batching, context sampling, robust parsing, Ctrl‑C partial results).
• Add mempalace/convo_scanner.py and wire it into discover_entities() for .claude/projects/ roots; expand unit tests to cover new modules.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
uv.lock	Adds tomli marker dependency for Python < 3.11.
mempalace/cli.py	Wires new --llm* flags into init and constructs/checks the provider before scanning.
mempalace/project_scanner.py	Extends discover_entities() to optionally scan Claude projects roots and run LLM refinement.
mempalace/convo_scanner.py	New scanner for ~/.claude/projects/ sessions, extracting project names from JSONL cwd.
mempalace/llm_client.py	New minimal HTTP-only provider abstraction (ollama/openai-compat/anthropic) + availability probes.
mempalace/llm_refine.py	New refinement pass: batching, context collection, response parsing, merge logic, and corpus sampling.
tests/test_llm_client.py	Unit tests for provider factory, HTTP wrapper, and provider request/response handling.
tests/test_llm_refine.py	Unit tests for prompt/context building, response parsing variants, merging, batching, and Ctrl‑C partials.
tests/test_convo_scanner.py	Unit tests for Claude projects root detection, cwd extraction, slug fallback, and dedup/ranking.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

_resolve_project_name() sorts session files by p.stat().st_mtime without handling OSError. If a .jsonl file is unreadable, broken, or permission-restricted, this will raise during sorting and can abort scanning the entire Claude projects tree. Consider wrapping stat() in a safe helper (e.g., defaulting mtime to 0 on error) or filtering out paths that fail stat before sorting.