RetrievalEvaluation#

class scikitplot.corpus.RetrievalEvaluation(metrics, queries, settings, metadata=<factory>)[source]#

Result of evaluate_retrieval.

Parameters:
metricsmapping

Metric name to the mean over queries (undefined values excluded under zero_division=np.nan).

queriestuple of QueryEvaluation

Per-query results, in evaluation order.

settingsmapping

The options the evaluation ran with.

metadatamapping, default={}

Free-form, JSON-compatible description supplied by the caller.

Parameters:

Notes

User note. write picks the format from the file suffix: .json, .jsonl/.ndjson (one row per query), .yaml/.yml or .toml. read restores a report from JSON, YAML or TOML.

compare(baseline, *, n_resamples=10000, seed=0)[source]#

Compare this evaluation with baseline over the same queries.

Parameters:
baselineRetrievalEvaluation

Evaluation of the same query identifiers.

n_resamplesint, default=10000

Random sign flips when the pairs exceed EXACT_SIGN_FLIP_LIMIT; below it every flip is enumerated.

seedint, default=0

Non-negative seed of the random flips, so the p-value is reproducible.

Returns:
EvaluationComparison
Raises:
TypeError

If baseline is not a RetrievalEvaluation.

ValueError

If the query sets differ or no metric is shared.

Parameters:
Return type:

EvaluationComparison

Notes

Per metric, only queries defined in both evaluations are paired. The statistic is |sum(d)| over the per-query differences d (the mean up to the constant number of pairs); the p-value counts sign flips whose statistic reaches the observed one (+1 in numerator and denominator when flips are sampled). A flip within 4 * n * eps * sum(|d|) of the observed statistic counts as reaching it, so flips equal for the values the metrics represent are not lost to floating-point rounding; the p-value errs on the conservative side.

dumps(format='json')[source]#

Serialise to "json", "jsonl", "yaml" or "toml".

Parameters:

format (str)

Return type:

str

classmethod from_dict(data)[source]#

Rebuild a report from to_dict output.

Raises:
ValueError

If the schema or its version is not this one.

Parameters:

data (Mapping[str, Any])

Return type:

RetrievalEvaluation

property latency: dict[str, float]#

mean, p50, p95, max and total.

Type:

Latency summary in seconds

metadata: Mapping[str, Any][source]#
metrics: Mapping[str, float][source]#
property n_queries: int#

Number of evaluated queries.

per_query(metric)[source]#

Return one metric’s per-query values, in evaluation order.

Raises:
KeyError

If the evaluation did not compute metric.

Parameters:

metric (str)

Return type:

ndarray

queries: tuple[source]#
classmethod read(path, format=None)[source]#

Read a report written by write in JSON, YAML or TOML.

Raises:
ValueError

For JSON Lines, which holds the per-query rows only.

Parameters:

  • path (Any)

  • format (str | None)

Return type:

RetrievalEvaluation

settings: Mapping[str, Any][source]#
property status_counts: dict[str, int]#

Number of queries per response status ("unreported" for none).

to_dict()[source]#

Return the full report as a mapping (schema versioned).

Return type:

dict[str, Any]

to_rows()[source]#

Return one flat row per query: identifier, each metric, counts, status.

Returns:
list of dict

The JSON Lines payload.

Return type:

list[dict[str, Any]]

write(path, format=None)[source]#

Write the report atomically.

Parameters:
pathstr or os.PathLike

Destination; its suffix names the format unless format does.

format{“json”, “jsonl”, “yaml”, “toml”} or None, default=None

Explicit format.

Returns:
pathlib.Path
Parameters:

  • path (Any)

  • format (str | None)

Return type:

Any