evaluate_answers#

scikitplot.corpus.evaluate_answers(system, dataset, *, judges=None, metrics=('faithfulness', 'answer_relevance'), criteria=None, text_metrics=('exact_match', 'token_f1'), on_error='record', n_jobs=1, keep_verdicts=True, metadata=None, clock=None)[source]#

Evaluate a system’s answers with text metrics and LLM judges.

Parameters:

systemcallable or None: system(question) returning the answer (str, a Completion, a mapping with "answer" and optional "contexts", or an object with .answer and optional .contexts). None evaluates each item’s stored "answer".
datasetsequence of mapping: Items with "question" and optional "id", "reference", "contexts", "answer" (see the module notes).
judgesmapping, LLMJudge, LLM, callable or None, default=None: Judges by name ({"gpt": make_llm("openai", ...), "claude": ...}); a single judge is named "judge". None runs the text metrics only.
metricssequence of str, default=(“faithfulness”, “answer_relevance”): Judge metrics: any of faithfulness, answer_relevance, context_precision, context_recall, answer_correctness.
criteriamapping or None, default=None: Rubrics by name, each reported as criteria:<name>.
text_metricssequence of str, default=(“exact_match”, “token_f1”): Deterministic metrics against the references (SQuAD normalisation).
on_error{“record”, “raise”}, default=”record”: What a failing system call or judge does: record it in the item (NaN scores) or raise.
n_jobsint, default=1: Threads for system and judge calls; results stay in dataset order.
keep_verdictsbool, default=True: Keep every JudgeResult (verdicts and completions) in the report.
metadatamapping or None, default=None: Free-form, JSON-compatible description stored in the report.
clockcallable or None, default=None: clock() -> float seconds for system latency.

Returns:

AnswerEvaluation

Raises:

TypeError, ValueError: For a malformed dataset, unknown metric names, invalid options, or an item without an answer when system is None.

Parameters:

system (Callable[[str], Any] | None)
dataset (Sequence[Mapping[str, Any]])
judges (Any)
metrics (Sequence[str])
criteria (Mapping[str, str] | None)
text_metrics (Sequence[str])
on_error (str)
n_jobs (int)
keep_verdicts (bool)
metadata (Mapping[str, Any] | None)
clock (Callable[[], float] | None)

Return type:

AnswerEvaluation