LLMJudge#

class scikitplot.corpus.LLMJudge(llm, *, on_error='raise')[source]#

Judge answers and contexts with a language model.

Parameters:
llmLLM or callable

An LLM from make_llm, or fn(prompt, system) -> str, which is wrapped by make_llm.

on_error{“raise”, “record”}, default=”raise”

What a failed judgement does: raise, or return a JudgeResult with score=NaN and error set. Applies to provider errors, cut-off completions and invalid verdicts; never to invalid arguments, which always raise.

Parameters:

  • llm (LLM | Callable[..., Any])

  • on_error (str)

See also

make_llm

The model behind a judge.

evaluate_answers

Runs judges over a dataset.

Notes

User note. Each method makes one model call (preference: two) and costs tokens; identity records the model and prompt version so scores from different judges are never mixed unknowingly. To judge the same answers with several models (a panel), pass several judges to evaluate_answers, which reports each judge, their mean and their agreement.

Examples

>>> judge = LLMJudge(lambda prompt, system: '{"score": 4, "reason": "direct"}')
>>> judge.answer_relevance("Capital of France?", "Paris.").score
0.75
answer_correctness(question, answer, reference)[source]#

Statement-level F1 of the answer against the reference.

Parameters:
question, answer, referencestr

The question, the answer under evaluation and the reference answer.

Returns:
JudgeResult

score = TP / (TP + (FP + FN) / 2). details: tp, fp, fn. Undefined when all three are zero.

Parameters:

  • question (str)

  • answer (str)

  • reference (str)

Return type:

JudgeResult

answer_relevance(question, answer)[source]#

How directly the answer addresses the question: (rating - 1) / 4.

Parameters:
question, answerstr

The question and the answer under evaluation.

Returns:
JudgeResult

details: rating (1-5).

Parameters:
Return type:

JudgeResult

context_precision(question, contexts, reference=None)[source]#

Average precision of the relevant contexts, in retrieval order.

Parameters:
questionstr

The question the contexts were retrieved for.

contextsiterable

Retrieved contexts, best first (see context_texts).

referencestr or None, default=None

A reference answer, which helps the judge decide usefulness.

Returns:
JudgeResult

score = sum(precision@k for each relevant rank k) / n_relevant; 0.0 when no context is relevant. details: relevant (one boolean per context, in order), n_relevant. Undefined without contexts (no call made).

Parameters:

  • question (str)

  • contexts (Any)

  • reference (str | None)

Return type:

JudgeResult

Notes

The verdict must hold exactly one entry per context index (1-based); a missing, repeated or out-of-range index is an invalid verdict.

context_recall(question, contexts, reference)[source]#

Fraction of the reference’s statements attributable to the contexts.

Parameters:
questionstr

The question the contexts were retrieved for.

contextsiterable

Retrieved contexts (see context_texts).

referencestr

The reference answer.

Returns:
JudgeResult

details: n_statements, n_attributed. Undefined when the reference yields no statement, or without contexts (no call made).

Parameters:

  • question (str)

  • contexts (Any)

  • reference (str)

Return type:

JudgeResult

criteria_score(question, answer, criteria, *, scale=5, reference=None, contexts=None)[source]#

Rate the answer against a caller’s rubric: (rating - 1) / (scale - 1).

Parameters:
question, answerstr

The question and the answer under evaluation.

criteriastr

The rubric, in plain language (“cites a source for every number”).

scaleint, default=5

Highest rating, at least 2.

referencestr or None, default=None

Optional reference answer shown to the judge.

contextsiterable or None, default=None

Optional contexts shown to the judge.

Returns:
JudgeResult

details: rating, scale.

Parameters:

  • question (str)

  • answer (str)

  • criteria (str)

  • scale (int)

  • reference (str | None)

  • contexts (Any)

Return type:

JudgeResult

faithfulness(question, answer, contexts)[source]#

Fraction of the answer’s claims that the contexts support.

Parameters:
question, answerstr

The question and the answer under evaluation.

contextsiterable

The contexts the answer was generated from (see context_texts).

Returns:
JudgeResult

details: n_claims, n_supported. Undefined (NaN) when the answer makes no claim, or without contexts (no call made).

Parameters:
Return type:

JudgeResult

identity()[source]#

Return the judge’s identity: the model’s, the prompt version and hash.

Returns:
dict
Return type:

dict[str, Any]

preference(question, answer_a, answer_b, *, reference=None)[source]#

Which of two answers is better, asked in both orders.

Parameters:
question, answer_a, answer_bstr

The question and the two answers; the score is for answer_a.

referencestr or None, default=None

Optional reference answer shown to the judge.

Returns:
JudgeResult

score for answer A: 1.0 (A wins), 0.0 (B wins) or 0.5 (tie). details: winner ("a", "b" or "tie") and position_consistent: whether both orders named the same winner. Two orders that disagree are a tie.

Parameters:

  • question (str)

  • answer_a (str)

  • answer_b (str)

  • reference (str | None)

Return type:

JudgeResult

Notes

The first call shows A first; the second shows B first. A judge that always picks the first (or second) position therefore yields a tie flagged position_consistent=False, not a win.