Analyzer#

class scikitplot.rank_bm25.Analyzer(name, version='1', language=None, stopwords=frozenset({}), stemmer=None, lowercase=True)[source]#

How text becomes tokens, declared so a build can record it.

Parameters:
namestr

Analyzer name, e.g. "whitespace".

versionstr, optional

Version of this analyzer’s behaviour. Bump it whenever the tokens it produces change, so an index built with the old behaviour is visibly a different build rather than silently the same one.

languagestr or None, optional

Language tag the analysis assumes, when it assumes one.

stopwordsfrozenset of str, optional

Tokens removed after splitting.

stemmerstr or None, optional

Name of the stemmer applied, if any. A name rather than a callable: a callable has no identity that survives the process.

lowercasebool, optional

Whether text is case-folded before splitting.

Parameters:

Notes

User. Two indexes built with analyzers that share a fingerprint hold comparable frequencies. Two that do not, do not – whatever their class names say.

property fingerprint: str#

Full SHA-256 digest of the declared behaviour.

Returns:
str

A 64-character hex digest, stable across processes because every part is a declared string and sets are sorted before hashing.

language: str | None = None#
lowercase: bool = True#
name: str[source]#
stemmer: str | None = None#
stopwords: frozenset[str] = frozenset({})#
tokenize(text)[source]#

Split text into tokens, applying what this analyzer declares.

Parameters:
textstr

Text to analyse.

Returns:
list of str

Tokens, in order.

Parameters:

text (str)

Return type:

list[str]

Notes

Developer. The declaration drives the behaviour rather than describing it from a distance. A declared field that did not affect the tokens would make the fingerprint a claim about nothing.

stemmer is recorded but not applied here: this module carries no stemming implementation, and pretending to apply one would be worse than declaring that a build used it.

version: str = '1'#