Merge pull request #817 from krisk/feat/token-search

feat: token search — per-term fuzzy matching with IDF scoring

Author: krisk

README.md

@@ -7,48 +7,185 @@
 [![Contributors](https://img.shields.io/github/contributors/krisk/fuse.svg)](https://github.com/krisk/Fuse/graphs/contributors)
 ![License](https://img.shields.io/npm/l/fuse.js.svg)
 
-## Supporting Fuse.js
+Fuse.js is a lightweight, zero-dependency fuzzy-search library written in TypeScript. It works in the browser and on the server, and is designed for searching small-to-medium datasets on the client side where you can't rely on a dedicated search backend.
+
+## ✨ What's New: Token Search
+
+Multi-word fuzzy search with relevance ranking. Type `"javascrpt paterns"` and find `"JavaScript Patterns"` — typo tolerance, multiple words, and smart ranking all at once.
+
+```js
+const fuse = new Fuse(docs, {
+  useTokenSearch: true,
+  keys: ['title', 'author', 'description']
+})
+
+fuse.search('javascrpt paterns')
+// → [{ item: { title: 'JavaScript Patterns', ... } }]
+```
+
+See [Token Search](#token-search) below for details.
+
+## Installation
+
+```bash
+npm install fuse.js
+```
+
+```bash
+yarn add fuse.js
+```
+
+Or include directly via CDN:
+
+```html
+<script src="https://cdn.jsdelivr.net/npm/fuse.js/dist/fuse.min.mjs"></script>
+```
+
+## Quick Start
+
+```js
+import Fuse from 'fuse.js'
+
+const books = [
+  { title: "Old Man's War", author: 'John Scalzi' },
+  { title: 'The Lock Artist', author: 'Steve Hamilton' },
+  { title: 'HTML5', author: 'Remy Sharp' },
+  { title: 'JavaScript: The Good Parts', author: 'Douglas Crockford' }
+]
+
+const fuse = new Fuse(books, {
+  keys: ['title', 'author']
+})
+
+fuse.search('javscript')
+// → [{ item: { title: 'JavaScript: The Good Parts', ... }, ... }]
+```
+
+## Features
+
+### Fuzzy Search
+
+The core of Fuse.js. Uses the [Bitap algorithm](https://en.wikipedia.org/wiki/Bitap_algorithm) for approximate string matching — handles typos, misspellings, and partial matches out of the box.
+
+```js
+fuse.search('javscript')
+// → [{ item: { title: 'JavaScript: The Good Parts', author: 'Douglas Crockford' } }]
+```
+
+### Weighted Keys
+
+Search across multiple fields with different importance levels. Title matches can rank higher than description matches.
+
+```js
+const fuse = new Fuse(docs, {
+  keys: [
+    { name: 'title', weight: 2 },
+    { name: 'description', weight: 1 }
+  ]
+})
+```
+
+### Extended Search
+
+Use operators for precise control: exact match (`=`), prefix (`^`), suffix (`!`), and more. Enable with `useExtendedSearch: true`.
+
+```js
+const fuse = new Fuse(list, {
+  useExtendedSearch: true,
+  keys: ['title']
+})
 
-Through contributions, donations, and sponsorship, you allow Fuse.js to thrive. Also, you will be recognized as a beacon of support to open-source developers.
-
-- [Become a backer or sponsor on **GitHub**.](https://github.com/sponsors/krisk)
-- [Become a backer or sponsor on **Patreon**.](https://patreon.com/fusejs)
-- [One-time donation via **PayPal**.](https://www.paypal.me/kirorisk)
-
----
-
-<h3 align="center">Sponsors</h3>
-<table>
-<tbody>
-    <tr>
-      <td align="center" valign="middle">
-        <a href="https://www.worksome.com" target="_blank">
-          <img width="222px" src="https://raw.githubusercontent.com/krisk/Fuse/7a0d77d85ac90063575613b6a738f418b624357f/docs/.vuepress/public/assets/img/sponsors/worksome.svg">
-        </a>
-      </td>
-      <td align="center" valign="middle">
-        <a href="https://www.bairesdev.com/sponsoring-open-source-projects/" target="_blank">
-          <img width="222px" src="https://github.com/krisk/Fuse/blob/gh-pages/assets/img/sponsors/bairesdev.png?raw=true">
-        </a>
-      </td>
-      <td align="center" valign="middle">
-        <a href="https://litslink.com/" target="_blank">
-          <img width="222px" src="https://github.com/krisk/Fuse/blob/gh-pages/assets/img/sponsors/litslink.svg?raw=true">
-        </a>
-      </td>
-    </tr>
-</body>
-</table>
-
----
-
-## Introduction
-
-Fuse.js is a lightweight fuzzy-search library, written in TypeScript, with zero dependencies.
+fuse.search('=exact match')   // exact match
+fuse.search('^prefix')        // starts with
+fuse.search('!term')          // does not include
+```
+
+### Token Search
+
+Splits multi-word queries into individual terms, fuzzy-matches each independently, and ranks results using BM25-style IDF weighting. Enable with `useTokenSearch: true`.
+
+```js
+const fuse = new Fuse(docs, {
+  useTokenSearch: true,
+  keys: ['title', 'body']
+})
+
+fuse.search('express midleware rout')
+// Finds "Express Middleware" and "Express Routing Guide" despite typos
+```
+
+- **Typo tolerance per word** — each term is fuzzy-matched independently
+- **Relevance ranking** — rare terms are weighted higher than common ones
+- **Word order independent** — `"patterns javascript"` and `"javascript patterns"` return identical results
+- **No query length limit** — long multi-word queries work naturally since each term is searched separately
+
+Available in the full build. See [TOKEN_SEARCH.md](TOKEN_SEARCH.md) for details and performance benchmarks.
+
+### Logical Search
+
+Combine conditions with `$and` and `$or` for complex queries. Available in the full build.
+
+```js
+fuse.search({
+  $and: [
+    { title: 'javascript' },
+    { author: 'crockford' }
+  ]
+})
+```
+
+### Match Highlighting
+
+Get character-level match indices for highlighting search results in your UI.
+
+```js
+const fuse = new Fuse(list, {
+  includeMatches: true,
+  keys: ['title']
+})
+
+const result = fuse.search('javscript')
+// result[0].matches[0].indices → [[0, 9]]
+```
+
+### Single String Matching
+
+Use `Fuse.match()` to fuzzy-match a pattern against a single string without creating an index. Useful for one-off comparisons or custom filtering.
+
+```js
+const result = Fuse.match('javscript', 'JavaScript: The Good Parts')
+// → { isMatch: true, score: 0.04, indices: [[0, 9]] }
+```
+
+### Dynamic Collections
+
+Add and remove documents from a live index without rebuilding.
+
+```js
+fuse.add({ title: 'New Book', author: 'New Author' })
+fuse.remove((doc) => doc.title === 'Old Book')
+```
+
+## Builds
+
+Fuse.js ships in two variants:
+
+| Build | Includes | Min + gzip |
+|---|---|---|
+| **Full** | Fuzzy + Extended + Logical + Token search | ~8 kB |
+| **Basic** | Fuzzy search only | ~6.5 kB |
+
+Use the basic build if you only need fuzzy search and want the smallest bundle size.
 
 ## Documentation
 
-To check out a [live demo](https://fusejs.io/demo.html) and docs, visit [fusejs.io](https://fusejs.io).
+For the full API reference, configuration options, scoring theory, and interactive demos, visit **[fusejs.io](https://fusejs.io)**.
+
+## Supporting Fuse.js
+
+- [Become a backer or sponsor on **GitHub**](https://github.com/sponsors/krisk)
+- [Become a backer or sponsor on **Patreon**](https://patreon.com/fusejs)
+- [One-time donation via **PayPal**](https://www.paypal.me/kirorisk)
 
 ## Develop
 

TOKEN_SEARCH.md

@@ -0,0 +1,56 @@
+# Token Search
+
+Token search splits multi-word queries into individual terms, fuzzy-matches each term independently using the Bitap algorithm, and ranks results using BM25-style IDF weighting. This combines Fuse.js's typo tolerance with relevance ranking — a query like `"javascrpt paterns"` will find `"JavaScript Patterns"`.
+
+## Usage
+
+```js
+const fuse = new Fuse(docs, {
+  useTokenSearch: true,
+  keys: ['title', 'author', 'description']
+})
+
+fuse.search('javascrpt paterns')
+// → [{ item: { title: 'JavaScript Patterns', ... }, score: 0.12 }]
+```
+
+All existing options work as before: `includeScore`, `includeMatches`, `keys` with weights, `threshold`, `limit`, `shouldSort`, etc.
+
+## How it works
+
+1. **Tokenization** — The query is split into individual words. Each word becomes a separate fuzzy search.
+
+2. **Per-term fuzzy matching** — Each term is matched against each field using the Bitap algorithm with `ignoreLocation: true`, so terms can appear anywhere in the field. This means multi-word queries are no longer limited by the 32-character Bitap pattern cap.
+
+3. **IDF weighting** — An inverted index is built at construction time. Rare terms (appearing in fewer documents) are weighted higher than common terms. This means a match on a distinctive word contributes more to the score than a match on a word that appears everywhere.
+
+4. **Score combination** — Per-term scores are combined additively with IDF weights, then normalized to Fuse's 0–1 range (0 = perfect match).
+
+## Key behaviors
+
+- **Partial matching** — A document matching 2 of 3 query terms is still returned, but ranks lower than one matching all 3.
+- **Word order independence** — `"patterns javascript"` and `"javascript patterns"` produce identical results.
+- **Typo tolerance per term** — Each term is fuzzy-matched independently, so typos in any word are tolerated.
+- **Long queries work** — A 6-word query runs 6 independent Bitap searches, each well within the 32-char limit.
+
+## Tips
+
+- Use `threshold` to control fuzziness. The default `0.6` is permissive — for tighter matching, try `0.3` or `0.4`.
+- Use `limit` when you only need the top N results. This also improves performance via heap-based selection.
+- Key weights still apply on top of token search scoring, so you can boost title matches over body matches.
+
+## Availability
+
+Token search is available in the **full build** (`fuse.js` / `fuse.mjs`). It is not included in the basic build to keep bundle size small. If you use the basic build with `useTokenSearch: true`, an error is thrown.
+
+## Performance
+
+Benchmarked on a corpus of randomly generated documents with 2 keys (title + body):
+
+| Metric | 100 docs | 1,000 docs | 5,000 docs |
+|---|---|---|---|
+| Index creation overhead | 2.5x | 5.2x | 5.5x |
+| Single-term search | 1.8x | 1.8x | 1.7x |
+| Multi-term search | 1.3x | 1.3x | 1.2x |
+
+Index creation is a one-time cost (46ms for 5,000 docs). Search overhead is 1.2–1.8x depending on query complexity, primarily because each query term runs its own Bitap search. The inverted index lookup itself is O(1) per term.

bench/token-search.mjs

@@ -0,0 +1,108 @@
+/**
+ * Benchmark: Token search vs default Bitap search
+ *
+ * Measures:
+ * 1. Index creation time (with and without inverted index)
+ * 2. Single-term query (both modes — ensures no regression)
+ * 3. Multi-term query (token search vs Bitap on full string)
+ * 4. Scaling: 100, 1000, 5000 documents
+ */
+
+import Fuse from '../dist/fuse.mjs'
+
+// --- Helpers ---
+
+function time(fn, iterations = 100) {
+  // Warmup
+  for (let i = 0; i < 5; i++) fn()
+
+  const start = performance.now()
+  for (let i = 0; i < iterations; i++) fn()
+  const elapsed = performance.now() - start
+  return elapsed / iterations
+}
+
+function generateDocs(n) {
+  const words = [
+    'javascript', 'python', 'rust', 'golang', 'typescript',
+    'programming', 'algorithms', 'patterns', 'design', 'systems',
+    'database', 'network', 'security', 'performance', 'testing',
+    'framework', 'library', 'compiler', 'runtime', 'deployment',
+    'functional', 'reactive', 'concurrent', 'distributed', 'embedded',
+    'machine', 'learning', 'neural', 'optimization', 'architecture'
+  ]
+
+  const docs = []
+  for (let i = 0; i < n; i++) {
+    const titleLen = 2 + Math.floor(Math.random() * 3)
+    const bodyLen = 8 + Math.floor(Math.random() * 15)
+    const pick = (len) => Array.from({ length: len }, () => words[Math.floor(Math.random() * words.length)]).join(' ')
+    docs.push({ title: pick(titleLen), body: pick(bodyLen) })
+  }
+  return docs
+}
+
+function fmt(ms) {
+  if (ms < 1) return `${(ms * 1000).toFixed(0)}µs`
+  return `${ms.toFixed(2)}ms`
+}
+
+// --- Benchmarks ---
+
+const sizes = [100, 1000, 5000]
+const keys = ['title', 'body']
+
+console.log('Token Search Benchmark')
+console.log('='.repeat(60))
+
+for (const n of sizes) {
+  const docs = generateDocs(n)
+
+  console.log(`\n--- ${n} documents ---\n`)
+
+  // Index creation
+  const defaultCreateTime = time(() => new Fuse(docs, { keys }), 20)
+  const tokenCreateTime = time(() => new Fuse(docs, { keys, useTokenSearch: true }), 20)
+  console.log(`Index creation (default):      ${fmt(defaultCreateTime)}`)
+  console.log(`Index creation (tokenSearch):   ${fmt(tokenCreateTime)}`)
+  console.log(`  overhead: ${((tokenCreateTime / defaultCreateTime - 1) * 100).toFixed(0)}%`)
+
+  // Create instances once for search benchmarks
+  const fuseDefault = new Fuse(docs, { keys })
+  const fuseToken = new Fuse(docs, { keys, useTokenSearch: true })
+
+  // Single-term query
+  const singleTerm = 'javascript'
+  const defaultSingleTime = time(() => fuseDefault.search(singleTerm), 200)
+  const tokenSingleTime = time(() => fuseToken.search(singleTerm), 200)
+  console.log(`\nSingle-term search (default):   ${fmt(defaultSingleTime)}`)
+  console.log(`Single-term search (token):     ${fmt(tokenSingleTime)}`)
+  console.log(`  overhead: ${((tokenSingleTime / defaultSingleTime - 1) * 100).toFixed(0)}%`)
+
+  // Multi-term query
+  const multiTerm = 'javascript design patterns'
+  const defaultMultiTime = time(() => fuseDefault.search(multiTerm), 200)
+  const tokenMultiTime = time(() => fuseToken.search(multiTerm), 200)
+  console.log(`\nMulti-term search (default):    ${fmt(defaultMultiTime)}`)
+  console.log(`Multi-term search (token):      ${fmt(tokenMultiTime)}`)
+  console.log(`  ratio: ${(tokenMultiTime / defaultMultiTime).toFixed(2)}x`)
+
+  // Multi-term with typos
+  const typoTerm = 'javascrpt desgn paterns'
+  const defaultTypoTime = time(() => fuseDefault.search(typoTerm), 200)
+  const tokenTypoTime = time(() => fuseToken.search(typoTerm), 200)
+  console.log(`\nTypo query search (default):    ${fmt(defaultTypoTime)}`)
+  console.log(`Typo query search (token):      ${fmt(tokenTypoTime)}`)
+  console.log(`  ratio: ${(tokenTypoTime / defaultTypoTime).toFixed(2)}x`)
+
+  // Result quality comparison
+  const fuseDefaultQ = new Fuse(docs, { keys, includeScore: true })
+  const fuseTokenQ = new Fuse(docs, { keys, useTokenSearch: true, includeScore: true })
+  const qResults = fuseDefaultQ.search(typoTerm)
+  const tResults = fuseTokenQ.search(typoTerm)
+  console.log(`\nResult count (default):         ${qResults.length}`)
+  console.log(`Result count (token):           ${tResults.length}`)
+}
+
+console.log('\n' + '='.repeat(60))
+console.log('Done.')