evaluate_retrieval#

scikitplot.corpus.evaluate_retrieval(retriever, queries, qrels, *, metrics=('hit_rate@10', 'precision@10', 'recall@10', 'mrr@10', 'ndcg@10'), id_getter=None, relevance_threshold=1, duplicates='raise', zero_division='warn', gain='linear', on_degraded='raise', keep_rankings=True, metadata=None, clock=None)[source]#

Run retriever on every query and score the rankings.

Parameters:

retrieverobject

Anything with search(query) (RetrievalIndex, CustomRetrievalIndex) or a callable retriever(query). The result is read by ranked_ids.

queriesmapping

Query identifier to query (usually text), passed to the retriever.

qrelsmapping

Query identifier to judgments ({doc_id: grade} or a collection of relevant identifiers), with exactly the keys of queries. Its order is the evaluation order.

metricsiterable of str, default=DEFAULT_METRICS

Names as in ranking_metrics.

id_gettercallable or None, default=None

Maps a result to its identifier (see ranked_ids).

relevance_thresholdfloat, default=1

Grade from which a judged identifier is relevant.

duplicates{“raise”, “first”}, default=”raise”

What a repeated identifier in a ranking does; "first" when several chunks map to one document through id_getter.

zero_division{“warn”, 0.0, 1.0, np.nan}, default=”warn”

Value of an undefined metric.

gain{“linear”, “exponential”}, default=”linear”

nDCG gain.

on_degraded{“raise”, “warn”, “ignore”}, default=”raise”

What a degraded or failed response does (see the module notes).

keep_rankingsbool, default=True

Keep each ranking in the report, so it can be re-scored later.

metadatamapping or None, default=None

Description stored with the report (system name, corpus version…).

clockcallable or None, default=None

clock() -> seconds; None uses time.perf_counter.

Returns:

RetrievalEvaluation

Raises:

TypeError

For a retriever that cannot be called, or malformed inputs.

ValueError

If queries and qrels have different keys, a ranking repeats an identifier under duplicates="raise", or a response is degraded under on_degraded="raise".

Parameters:

• retriever (Any)

• queries (Mapping[Any, Any])

• qrels (Mapping[Any, Any])

• metrics (Iterable[str])

• id_getter (Callable[[Any], Any] | None)

• relevance_threshold (float)

• duplicates (str)

• zero_division (Any)

• gain (str)

• on_degraded (str)

• keep_rankings (bool)

• metadata (Mapping[str, Any] | None)

• clock (Callable[[], float] | None)

Return type:

RetrievalEvaluation

See also

ranking_metrics

The same metrics over precomputed rankings.

make_corpus_scorer

One metric as a scikit-learn scorer.

Examples

Try it in your browser!

>>> from scikitplot.corpus import evaluate_retrieval
>>> def retriever(query):
...     return [("d1", 0.9), ("d2", 0.4)] if "cat" in query else [("d3", 0.7)]
>>> report = evaluate_retrieval(
...     retriever,
...     queries={"q1": "cat food", "q2": "dog toys"},
...     qrels={"q1": ["d2"], "q2": ["d4"]},
...     metrics=["recall@2", "mrr"],
... )
>>> report.metrics
{'recall@2': 0.5, 'mrr': 0.25}

Go BackOpen In Tab