Chunk size (800) exceeds embedding model token limit (256 tokens / ~512 chars)

[sidonsoft]

Chunk size (800) exceeds embedding model token limit (256 tokens / ~512 chars)

Problem

The default CHUNK_SIZE = 800 in miner.py exceeds the token limit of the default embedding model.

Details:

• Chunk size: 800 characters (line 56 in miner.py)
• Embedding model: all-MiniLM-L6-v2 (ChromaDB default via ONNX)
• Model token limit: 256 tokens (~512 characters)
• Result: Content beyond ~512 chars is silently truncated before embedding

Impact

1. Lost context: Important information in positions 512-800 of each chunk is not included in the embedding
2. Reduced recall: Semantic search may miss relevant content in truncated portions
3. Silent degradation: No errors or warnings — users don't know this is happening

Suggested Fixes

Option 1: Reduce default chunk size

# miner.py line 56-57
CHUNK_SIZE = 400  # Was 800 - safer for 256 token limit
CHUNK_OVERLAP = 50  # Was 100

Option 2: Add CLI configuration

mempalace mine <path> --chunk-size 400 --chunk-overlap 50

Option 3: Auto-detect model limits
Query the embedding function for its max tokens and adjust chunking accordingly.

Environment

• mempalace version: 3.0.0
• ChromaDB version: 1.5.7
• Embedding: Default (all-MiniLM-L6-v2 via ONNX)

Priority

Medium — search works but quality could be significantly improved.

[web3guru888]

This is a real issue — the 800-char default silently truncates embeddings for the most common local embedding model. The MiniLM models have a 256-token limit, which is roughly 512 characters for typical English prose and much less for dense technical content (code, formulas, citations).

A few data points from our experience running MemPalace on research content (astrophysics, cryptography, economics papers):

The truncation is asymmetric — it silently affects dense content (papers, code) more than casual notes. A research note with a formula in the second half gets the formula silently dropped from its embedding. You get no error, just subtly worse retrieval on the content you care most about.

What actually helps:

• CHUNK_SIZE = 400 is a safer default for all-MiniLM-L6-v2 — leaves headroom for Unicode and tokenization overhead
• For paraphrase-multilingual-mpnet-base-v2 (128-token limit), you'd want to go even lower (~300 chars)
• The right fix is a model-aware chunk size computed from the tokenizer, but that adds complexity

PR #492 (jphein) makes CHUNK_SIZE user-configurable via config — that's a good interim fix since users who know their model's limits can set the right value. PR #488 (EndeavorYen) flagged an implicit CHUNK_SIZE change in the multilingual PR as a potential breaking change.

A warning when CHUNK_SIZE > (model_token_limit * avg_chars_per_token) would also help — silent truncation is the worst kind of correctness bug.