fix(searcher): tolerate None documents in BM25 reranker

[jphein]

Problem

searcher._tokenize calls text.lower() unconditionally. When ChromaDB returns a drawer with documents containing None, the hybrid-rerank path raises AttributeError mid-search, dropping the entire query response.

Observed in production on 2026-04-24 21:07:05:

File "mempalace/searcher.py", line 81, in _bm25_scores
    tokenized = [_tokenize(d) for d in documents]
File "mempalace/searcher.py", line 52, in _tokenize
    return _TOKEN_RE.findall(text.lower())
AttributeError: 'NoneType' object has no attribute 'lower'

This closes the gap left by #999's None-metadata audit, which fixed metadata read loops but did not extend to BM25 reranker helpers.

Fix

_tokenize short-circuits to [] when text is falsy (None or ""). A None doc gets BM25 score 0.0; the rest of the corpus reranks normally.

Tests

Three regression tests in TestBM25NoneSafety:

1. test_tokenize_handles_none — direct unit guard
2. test_tokenize_handles_empty_string — covers the empty-string case at the same boundary
3. test_bm25_scores_does_not_crash_on_none_documents — load-bearing: mixed corpus with a None mid-list, asserts score 0.0 for that doc and finite scores for the rest. This is the exact production failure shape.

All existing searcher and hybrid_search tests still pass (pytest tests/test_searcher.py tests/test_hybrid_search.py → 30/30 against develop after cherry-pick).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens the hybrid BM25 re-ranker against None documents returned by ChromaDB so searches don’t crash mid-rerank in production.

Changes:

• Add a None/empty guard in searcher._tokenize so missing documents produce no tokens (and thus BM25 score 0.0).
• Add regression tests covering _tokenize(None), _tokenize(""), and _bm25_scores with a mixed corpus containing a None.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
mempalace/searcher.py	Makes _tokenize tolerate falsy input to prevent AttributeError during BM25 scoring.
tests/test_searcher.py	Adds regression tests to ensure BM25 helpers don’t crash when Chroma returns None documents.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

_tokenize now intentionally accepts None (and tests cover it), but the type annotation still says text: str. Please update the signature/return typing to reflect the supported inputs (e.g., str | None) and consider narrowing the guard to the intended cases (None/empty string) so unexpected non-string falsy values don’t get silently treated as empty text.

[Copilot]

This regression test claims “finite scores for the rest”, but it only asserts on scores[0] and the None doc. Consider also asserting that the other score(s) are finite (and/or the expected value, e.g. 0.0 for the non-matching doc) to make the test match its stated guarantee and catch potential NaN/inf regressions.

[bensig]

Approve, no asks. Defensive single-line fix with a real production trace driving it.

What landed

`_tokenize` now returns `[]` for falsy input instead of raising `AttributeError` on `None.lower()`. The trace in the test docstring is genuine (daemon log 2026-04-24 21:07:05) — Chroma was returning `None` for drawers without text content during a hybrid-rerank pass, and the BM25 path crashed mid-search.

Test coverage

The new `TestBM25NoneSafety` class covers both the `None` and empty-string paths and includes the failure trace in the docstring so a future reader knows exactly what symptom this prevents. `pytest tests/test_searcher.py` — 27 passed on this branch.

Minor

Could also have hardened `_bm25_scores` upstream of `_tokenize` to filter `None` documents before iterating. The chosen approach (defend at the leaf function) is more defensive and catches any other call site that might pass `None` — better choice for a small surface like this.

Ship it.

[jphein]

Thanks @bensig — ready for merge whenever convenient. Tests stable on the fork's daily-driver palace + smoke pass on the develop branch.

[igorls]

LGTM