🔬 DeLong's Test

DeLong’s Test#

A statistical test comparing the AUCs of two models.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

DeLong’s test is a non-parametric statistical test for comparing the ROC-AUCs of two correlated models — answering is model A’s AUC significantly better than model B’s? Proposed by DeLong, DeLong & Clarke-Pearson (1988), it matters because two models are usually evaluated on the same test set, so their AUCs are correlated and their difference must be judged with a p-value, not by eye.

Why you need it#

AUCs are random variables — they depend on the sample — so a raw AUC_A - AUC_B gap could be noise. DeLong’s test supplies the standard error of each AUC, the SE of their difference, and a z-statistic and p-value.

How it works#

It treats AUC as the probability that a random positive outranks a random negative, and uses U-statistics to estimate the variances and covariances of those pairwise comparisons, giving

\[z = \frac{\text{AUC}_1 - \text{AUC}_2}{\text{SE}(\text{AUC}_1 - \text{AUC}_2)},\]

which is referred to the standard normal for a p-value.

A worked example#

Model A scores AUC = 0.88 and Model B 0.84. The difference of 0.04 with an SE of 0.015 gives \(z \approx 2.67\) and \(p \approx 0.0076\) — so A is significantly better (p < 0.01).

# delong_roc_test is a small helper (from a gist/package)
from delong import delong_roc_test

# y_true = true labels; y_pred1, y_pred2 = the two models' scores
p_value = delong_roc_test(y_true, y_pred1, y_pred2)
print("p-value:", p_value)

Alternatives#

Other options include bootstrap confidence intervals (resample and recompute the AUC-difference distribution), the older, less accurate Hanley-McNeil approximation, and permutation tests (shuffle labels to build the null).

Theme: Model Evaluation & Uncertainty · All terminology

Hint

Mind map — connected ideas

Multiclass AUROC · Discriminatory Power · Statistical Power · Dataset Shift · Evaluation Set · Model KPIs (Key Performance Indicators)

Hint

More in Model Evaluation & Uncertainty

Average Absolute Error (AAE) · Baseline Heuristics · Bootstrap · Bootstrap Confidence Intervals (CIs) · Coverage · Cramér’s V · KS Statistic (Kolmogorov–Smirnov Statistic) · Likelihood Ratio (LR) · Mann–Whitney U Test (also called the Wilcoxon rank-sum test) · MASE (Mean Absolute Scaled Error) · Mean Absolute Error (MAE) · Mean Absolute Percentage Error (MAPE) · Mean Squared Error (MSE) · Relative accuracy