make_corpus_scorer#

scikitplot.corpus.make_corpus_scorer(score_func, *, response_method='predict', greater_is_better=True, id_getter=None, on_degraded='raise', **kwargs)[source]#

Make a scikit-learn compatible scorer from a corpus metric.

Parameters:

score_funccallable: score_func(y_true, y_pred, **kwargs) -> float, for example recall_at_k or token_f1_score.
response_method{“predict”, “search”}, default=”predict”: How predictions are obtained (see the module documentation).
greater_is_betterbool, default=True: False for a loss; the scorer then negates it, as scikit-learn does.
id_gettercallable or None, default=None: response_method="search" only: id_getter(result) maps a search result to the identifier used by y_true (for example a chunk to its source document). None applies the rule of ranked_ids.
on_degraded{“raise”, “ignore”}, default=”raise”: response_method="search" only: what a response whose requested leg did not run does.
**kwargsdict: Passed to score_func on every call, such as k=10.

Returns:

scorer: A _BaseScorer subclass instance (scikit-learn’s, or the vendored copy; see scorer_backend_info). It pickles.

Raises:

TypeError: If score_func or id_getter is not callable, or kwargs do not fit the signature of score_func.
ValueError: For an unknown response_method or on_degraded, or id_getter with response_method="predict".

Parameters:

score_func (Callable[[...], Any])
response_method (str)
greater_is_better (bool)
id_getter (Callable[[Any], Any] | None)
on_degraded (str)
kwargs (Any)

Return type:

Any

See also

get_corpus_scorer: The same from a name such as "ndcg@10".
sklearn.metrics.make_scorer: The scikit-learn original.

Examples

>>> from scikitplot.corpus import make_corpus_scorer, recall_at_k
>>> scorer = make_corpus_scorer(recall_at_k, response_method="search", k=2)
>>> class Index:
...     def search(self, query):
...         return [("d1", 0.9), ("d2", 0.5)]
>>> scorer(Index(), ["first query", "second query"], [["d2"], ["d3"]])
0.5