BM25Okapi#

class scikitplot.rank_bm25.BM25Okapi(corpus, tokenizer=None, k1=1.5, b=0.75, epsilon=0.25, doc_ids=None, workers=None)[source]#
RECIPE_ID: str = 'okapi-epsilon-floor/1'#

The scoring formula this class implements. Subclasses override it; the class name is not the recipe, because “BM25Okapi” implies a standard the implementation makes specific choices within.

property analyzer: Analyzer | None#

The declared analyzer, or None when tokens were supplied directly.

candidate_count(query)[source]#

Number of documents that contain at least one query term.

Parameters:
querysequence of str

Query tokens.

Returns:
int

Documents the sparse path will score.

Notes

User. This is what a query actually costs. get_scores returns one value per document in the corpus whatever the query, which is a diagnostic rather than a retrieval path.

property doc_ids#

Row identities, in row order.

Type:

Tuple of str or None

get_batch_scores(query, doc_ids)[source]#

Calculate bm25 scores between query and subset of all docs.

get_scores(query)[source]#

ATIRE BM25 variant uses an idf function which uses a log(idf) score.

To prevent negative idf scores, this algorithm also adds a floor to the idf value of epsilon. See [Trotman, A., X. Jia, M. Crane, Towards an Efficient and Effective Search Engine] for more info :param query: :return:

get_top_ids(query, n=5)[source]#

Return the n best (doc_id, score) pairs.

Parameters:
querysequence of str

Query tokens.

nint, optional

Maximum number of hits.

Returns:
list of tuple

(doc_id, score) in descending score, ties in row order.

Raises:
ValueError

If the index holds no identities, or n is not a valid count.

Notes

User. This is the binding to prefer. get_top_n maps row offsets into a sequence you pass at query time, which is only correct while that sequence is in the same order as the corpus the index was built from – a precondition nothing can check for you.

get_top_n(query, documents, n=5)[source]#
property identity: str#

recipe, analyzer and parameters.

Notes

User. Two indexes sharing this value were built the same way. It does not cover the documents: that is what the corpus records, and folding it in here would make a re-tokenisation and a new document look like the same kind of change.

Type:

Digest of what produced this index

classmethod load(directory)[source]#

Load a published index, or a generation directory directly.

Parameters:
directorystr or path-like

Index root, or a generation directory.

Returns:
BM25

The reloaded index, ready to query.

Raises:
ValueError

If the root has no readable pointer, or the state is not from this recipe. A root with no pointer is refused rather than guessed at.

property parameters: dict#

The scoring parameters that belong to this index’s identity.

property recipe: Recipe#

The named, versioned scoring formula this index used.

save(directory)[source]#

Publish this index as a generation behind a pointer.

Parameters:
directorystr or path-like

Index root. Created if absent.

Returns:
pathlib.Path

The generation directory written.

Raises:
OSError

If the generation could not be written. Nothing is published in that case, and any previously published generation is untouched.

Notes

User. A saved index reloads without the corpus: the statistics are what get published, not the documents.

Developer. Same shape as the corpus artifact and the Annoy bundle: candidate, verify by reloading, then move the pointer. The three carry their own implementation rather than a shared one, which is the cost of each submodule standing alone; the shape is deliberately identical so a reader of one recognises the others.