AnswerEvaluation#

class scikitplot.corpus.AnswerEvaluation(metrics, items, judges, agreement, usage, errors, settings, metadata=<factory>)[source]#

Result of evaluate_answers.

Parameters:

metricsmapping

Metric name to the mean over items where it is defined.

itemstuple of AnswerItem

Per-item results, in dataset order.

judgesmapping

Judge name to identity.

agreementmapping

Judge metric to Krippendorff’s alpha (interval) between the judges, when there are at least two; NaN when undefined.

usagemapping

Judge name to responses (model responses received), cached_responses, input_tokens and output_tokens (summed over responses that were not cached, so a cached rerun costs 0) and responses_without_token_counts.

errorsmapping

Failures by kind: "system" (items without an answer), then one count per judge (failed judgements), whether or not verdicts are kept.

settingsmapping

The options the evaluation ran with.

metadatamapping, default={}

Free-form, JSON-compatible description supplied by the caller.

Parameters:

• metrics (Mapping[str, float])

• items (tuple)

• judges (Mapping[str, Any])

• agreement (Mapping[str, float])

• usage (Mapping[str, Mapping[str, int]])

• errors (Mapping[str, int])

• settings (Mapping[str, Any])

• metadata (Mapping[str, Any])

agreement: Mapping[str, float][source]#

compare(baseline, *, n_resamples=10000, seed=0)[source]#

Compare with baseline over the same items (paired sign-flip test).

Parameters:

baselineAnswerEvaluation

Evaluation of the same item identifiers (another system, or another prompt of the same model).

n_resamples, seedint

As in RetrievalEvaluation.compare.

Returns:

EvaluationComparison

Raises:

TypeError

If baseline is not an AnswerEvaluation.

ValueError

If the item sets differ or no metric is shared.

Parameters:

• baseline (AnswerEvaluation)

• n_resamples (int)

• seed (int)

Return type:

EvaluationComparison

Notes

Compare evaluations scored by the same judges: a judge identity that differs between the two reports makes the difference partly the judges’. compare does not refuse it; check judges.

property coverage: dict[str, int]#

Metric name to the number of items where it is defined.

dumps(format='json')[source]#

Serialise to "json", "jsonl" (one row per item), "yaml" or "toml".

Parameters:

format (str)

Return type:

str

errors: Mapping[str, int][source]#

classmethod from_dict(data)[source]#

Rebuild a report from to_dict output.

Raises:

ValueError

If the schema or its version is not this one.

Parameters:

data (Mapping[str, Any])

Return type:

AnswerEvaluation

items: tuple[source]#

judges: Mapping[str, Any][source]#

metadata: Mapping[str, Any][source]#

metrics: Mapping[str, float][source]#

property n_items: int#

Number of evaluated items.

per_item(metric)[source]#

Return one metric’s per-item values, in dataset order.

Raises:

KeyError

If the evaluation did not compute metric.

Parameters:

metric (str)

Return type:

ndarray

classmethod read(path, format=None)[source]#

Read a report written by write in JSON, YAML or TOML.

Parameters:

• path (Any)

• format (str | None)

Return type:

AnswerEvaluation

settings: Mapping[str, Any][source]#

to_dict()[source]#

Return the full report as a mapping (schema versioned).

Return type:

dict[str, Any]

to_rows()[source]#

One flat row per item: identifier, answer, each score, latency, error.

Return type:

list[dict[str, Any]]

usage: Mapping[str, Mapping[str, int]][source]#

write(path, format=None)[source]#

Write atomically; the format defaults to the path suffix.

Parameters:

• path (Any)

• format (str | None)

Return type:

Any