ScoreCalibrator#

class scikitplot.rank_bm25.ScoreCalibrator(method='sigmoid', *, platt_targets=True)[source]#

Monotone map from BM25 scores to probabilities of relevance.

Parameters:
method{“sigmoid”, “isotonic”}, default=”sigmoid”

Platt scaling (two parameters, data-efficient, strictly monotone) or isotonic regression (flexible, needs more labels, can merge ranks).

platt_targetsbool, default=True

For "sigmoid": fit to Platt’s smoothed targets instead of 0/1, which keeps the fit finite on perfectly separable labels.

Attributes:
fittedbool

Whether fit has run.

populationstr or None

The population the labels were drawn from, as declared at fit.

prevalence_float

Share of relevant labels in the population, after any shift_prior.

n_samples_, n_queries_int

Labelled pairs and distinct queries used to fit.

slope_, intercept_float

Sigmoid parameters ("sigmoid" only).

thresholds_, values_numpy.ndarray

Fitted points ("isotonic" only).

Parameters:
  • method (str)

  • platt_targets (bool)

Notes

User. Fit on scores produced exactly as they will be at prediction time – same recipe, same parameters, same analyzer, same candidate selection – and evaluate only on queries not used to fit (require_held_out). A score transformed without fitting is not a probability; this class has no unfitted transform on purpose.

fit(scores, labels, *, query_ids, population)[source]#

Fit the map on labelled (query, document) pairs.

Parameters:
scoresarray-like of shape (n_pairs,)

BM25 scores, finite.

labelsarray-like of shape (n_pairs,)

Binary relevance (0/1 or bool).

query_idsiterable of str

The query each pair belongs to; kept only as SHA-256 digests, for require_held_out.

populationstr

What the pairs were drawn from, e.g. "bm25-top100"; a calibrator is valid for that population only.

Returns:
ScoreCalibrator

self.

Raises:
ValueError

For mismatched lengths, non-binary labels, labels of one class only, a non-finite score, an empty population, a sigmoid slope <= 0, or divergence on separable labels.

TypeError

For non-numeric scores or labels.

Parameters:
Return type:

ScoreCalibrator

classmethod from_dict(state)[source]#

Rebuild a fitted calibrator from to_dict output.

Raises:
ValueError

If the schema is not this one, or a required field is missing.

Parameters:

state (dict[str, Any])

Return type:

ScoreCalibrator

predict_proba(scores)[source]#

Probability of relevance for each score.

Parameters:
scoresarray-like of shape (n,)

BM25 scores produced the way the training scores were.

Returns:
numpy.ndarray of float64, shape (n,)

One probability per score (one column, not scikit-learn’s two).

Raises:
RuntimeError

If not fitted.

ValueError, TypeError

For non-finite, non-numeric or multi-dimensional scores.

Parameters:

scores (Any)

Return type:

ndarray

require_held_out(query_ids)[source]#

Refuse evaluation queries that were used to fit.

Parameters:
query_idsiterable of str

The queries an evaluation will use.

Raises:
ValueError

Naming how many of them were seen during fit. Fit and evaluate on the same queries and calibration error is underestimated.

Parameters:

query_ids (Iterable[Any])

Return type:

None

shift_prior(base_rate, *, population)[source]#

Return a copy valid for a population with another relevant share.

Parameters:
base_ratefloat

Share of relevant pairs in the new population, in (0, 1).

populationstr

Name of the new population.

Returns:
ScoreCalibrator

A new calibrator; self is unchanged.

Parameters:
Return type:

ScoreCalibrator

Notes

User. Adds logit(base_rate) - logit(training prevalence) to every log-odds. Exact when the scores of relevant and of non-relevant pairs are distributed the same way in both populations (only their mix differs) – true for a subsample, not for a different candidate generator. Isotonic outputs of exactly 0 or 1 stay 0 or 1.

to_dict()[source]#

Return a JSON-serialisable state; query ids appear only as digests.

Return type:

dict[str, Any]