MODELCATALOG

Make predict_proba mean what it says.

sklearn.calibration

calibration

Make predict_proba mean what it says.

1 Get data Something to practise on
2 Split & tune An honest test set
3 Prepare Into numbers a model accepts
4 Compose One object, no leakage
5 Model The algorithm itself
6 Unsupervised No labels available
7 Measure Is it any good, and why
8 Utilities Glue and extension

What it is for

A model that outputs 0.9 is not necessarily right 90% of the time. Tree ensembles and SVMs are systematically miscalibrated. If any business rule thresholds on a probability, that number needs to be honest.

Contents

CalibratedClassifierCV bank Wraps a fitted model. method='sigmoid' (Platt) or 'isotonic'
calibration_curve Shows you how badly miscalibrated it currently is
CalibrationDisplay Plots the same

Worked examples 2

Runnable, with the arguments that change the outcome called out. Output shapes are shown as comments where they help.

Check whether 0.9 really means 90%

Before trusting any probability threshold in a business rule, look at the calibration curve.

from sklearn.calibration import calibration_curve
import numpy as np

prob_true, prob_pred = calibration_curve(y_test, p, n_bins=10,
                                         strategy="quantile")

print(f"{'predicted':>10} {'actual':>10}   gap")
for pred, true in zip(prob_pred, prob_true):
    print(f"{pred:10.3f} {true:10.3f}   {true - pred:+.3f}")

#  predicted     actual   gap
#      0.021      0.008   -0.013
#      0.108      0.061   -0.047
#      0.294      0.187   -0.107     <- claims 29%, delivers 19%
#      0.612      0.478   -0.134     <- systematically overconfident
#      0.891      0.812   -0.079
#
# A rule firing at p > 0.6 is catching cases that are really about 48%
# likely. Whether that matters depends on the rule, but you should
# know it before someone else finds out.

Fix it

Wraps the model and recalibrates on held-out folds. Sigmoid for small data, isotonic when you have plenty.

from sklearn.calibration import CalibratedClassifierCV

# cv=5 refits the base model on each fold and calibrates on the rest.
# Use cv="prefit" instead if the model is already fitted and you have
# a separate calibration set you have not otherwise touched.
cal = CalibratedClassifierCV(
    base_model,
    method="isotonic",     # flexible; needs ~1000+ samples
                           # "sigmoid" (Platt) for smaller data
    cv=5,
)
cal.fit(X_train, y_train)

p_cal = cal.predict_proba(X_test)[:, 1]

from sklearn.metrics import brier_score_loss
print("before:", round(brier_score_loss(y_test, p), 5))
print("after: ", round(brier_score_loss(y_test, p_cal), 5))

# Calibration changes the probabilities but not their ORDER, so ROC AUC
# is unchanged. It fixes what the numbers mean, not which rows rank
# highest.