Enhance robustness of memory search with jieba-next CJK tokenizer and native Markdown parsing

[KagaJiankui]

Problem Statement

There are two severe design flaws that carries a high risk of compromising information retrieval effectiveness and degrading overall reliability, especially in CJK (specifically Chinese) contexts:

• The current lack of robust tokenization and semantic parsing forms a critical bottleneck for practical usage in multilingual (notably CJK) environments.
• LLMs may produce malformed or inconsistent Markdown output: their markdown dialect is shaped by black-box pre-training/RLHF, making formats unstable even with explicit prompts. And infrastructure issues (network fluctuations, cluster load balancing) often cause errors in complex structures like tables.
• Plain regex chunking and standard tokenizers (as in current store.py/chunker.py) lead to poor recall, fuzzy chunking, and ineffective BM25 search in Chinese.

Proposal: Tokenizer and Markdown Parser Recommendations

• Chinese tokenizer: Prioritize jieba-next (most API-rich, good Windows support) for CJK tokenization. Alternatively, jieba-rs offers higher speed but narrower API coverage for basic use cases.
• Markdown parser: Favor Mistune, as it is actively maintained and provides a structured AST in Python. Alternatively, md4c is very fast, but the SAX parser design can be fragile in engineering for precise chunking.

Potential Consequences

Failure to address these design flaws will:

• Severely limit semantic search quality, especially for Chinese/CJK agents or knowledge bases
• Hinder adoption as a memory solution in multilingual/global settings
• Brittle heading/paragraph detection increases risk of false negatives/positives during recall

Request: Please consider prioritizing a robust chunker and BM25 tokenizer/pluggable analyzer as a core architectural improvement.

[zc277584121]

Thanks for the detailed analysis! You're right that CJK tokenization is a real gap.

Re: Chinese tokenization — Good news: Milvus 2.5+ natively supports configurable BM25 analyzers, including a built-in jieba tokenizer. So instead of adding jieba as a Python dependency, we can configure it at the Milvus collection schema level via analyzer_params. This keeps the tokenization server-side and avoids duplicating work in Python. We'll look into exposing this as a config option (e.g. bm25_analyzer = "chinese") so users can opt in.

Re: Markdown parsing — The current heading-based chunker works well for most real-world markdown (daily logs, notes, docs). LLM-generated malformed markdown is a valid concern, but in practice memsearch primarily indexes human-written or lightly-edited files. We'll keep Mistune in mind if we see real breakage, but for now the regex approach is intentionally simple and dependency-free.

Keeping this open as a roadmap item for CJK analyzer support.

[KagaJiankui]

As the regex-based markdown tokenizer works well by your statement, the AST based markdown chunker can be put to a low precedence.

There is one concern left: Is the built-in jieba of Milvus 2.5+ python based or native? The original python based jieba tokenizer can experience severe performance issue with large chunk input.