evaluate_preference#

scikitplot.corpus.evaluate_preference(dataset, answers_a, answers_b, *, judges, names=('a', 'b'), on_error='record', n_jobs=1, metadata=None)[source]#

Judge two systems’ answers head to head, in both orders.

Parameters:

datasetsequence of mapping

Items with "question" and optional "id" and "reference", shown to the judges when the item has exactly one.

answers_a, answers_bsequence of str or mapping

Each system’s answer per item: a sequence in dataset order, or a mapping from item id.

judgesmapping, LLMJudge, LLM or callable

At least one judge (see evaluate_answers).

namestuple of str, default=(“a”, “b”)

Names of the two systems in the report.

on_error{“record”, “raise”}, default=”record”

What a failing judge does: record it (NaN score) or raise.

n_jobsint, default=1

Threads for judge calls; results stay in dataset order.

metadatamapping or None, default=None

Free-form, JSON-compatible description stored in the report.

Returns:

PreferenceEvaluation

Raises:

TypeError, ValueError

For malformed inputs, a missing answer, or no judge.

Parameters:

• dataset (Sequence[Mapping[str, Any]])

• answers_a (Sequence[str] | Mapping[Any, str])

• answers_b (Sequence[str] | Mapping[Any, str])

• judges (Any)

• names (tuple[str, str])

• on_error (str)

• n_jobs (int)

• metadata (Mapping[str, Any] | None)

Return type:

PreferenceEvaluation

Notes

Each judge is asked twice per item, once in each order (see preference); a judge that changes its mind with the order contributes a tie and lowers position_consistency.