BM25 should not consider repeated query tokens

[Witiko]

@dorianbrown In the seminal paper for this package, the Okapi at TREC-3 paper, and most other places, BM25 is defined over query terms rather than tokens, which would indicate that repeated query tokens should not impact the score. However, that does not seem to be the case in the rank-bm25 library:

rank_bm25/rank_bm25.py

Line 117 in 329b794

for q in query:

This can be easily solved by the user by passing set(query)¹ rather than query to the get_scores() method, but it seems as something that the user would expect to happen automatically. At the very least, we may want to document this.

---

¹ Alternatively, list(dict.fromkeys(query)) for reproducible ordering, since floating point summation is not always associative.

[dzieciou]

In Pyterrier/Terrier implementation of BM25 the number of times a query term is repeated matters. It is often used to upweight certain query terms. Look how the scoring changes:

[Witiko]

@dzieciou Different implementations of BM25 take different liberties with the original algorithm, including (Py)Terrier. If the rank_bm25 library is to implement the algorithm as it was originally described, then it should treat the query terms as a set, not as a multiset.

However, I am satisfied with the behavior being documented, at least in an open issue on GitHub if not elsewhere.