🧷 Data Leakage

Data Leakage#

When information from outside the training set leaks in, inflating performance.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Data leakage is when information from outside the training data slips into training, giving the model unfair access to future or hidden knowledge. The signature is a model that looks excellent in validation but collapses on real, unseen data.

The four types#

Target leakage: a feature encodes the answer — predicting loan default from debt_collected_after_default, which only exists because of default. Train-test contamination: test information bleeds in via preprocessing — e.g. scaling with a mean and standard deviation computed over the whole dataset instead of the training fold alone. Temporal leakage: using future data to predict the past — forecasting January’s price with March’s trading volume. Group leakage: the same group (patient, user, session) lands in both train and test, so the model just recognises the group.

How to prevent it#

Five guards: fit preprocessing on the training fold only and apply it to the rest; drop features that wouldn’t exist at prediction time; use time-aware splits for temporal data; use group-aware CV (GroupKFold, StratifiedGroupKFold) to keep groups intact; and monitor after deployment — a sharp drop from validation to production is the classic leakage tell.

Why it’s dangerous#

Leakage manufactures a false sense of performance, masking overfitting and poor generalisation, and in regulated domains like finance and healthcare it can turn into a compliance problem when the model fails on the data that matters.

Theme: Validation & Cross-Validation · All terminology

Hint

Mind map — connected ideas

Cross-Validation (CV) · Stratified Group K-Fold · Blocked Splits (Single Holdout) · Sliding Window (Rolling Window) Cross-Validation · Temporal autocorrelation (Serial Correlation) · Data Drift

Hint

More in Validation & Cross-Validation

Blocked Splits (Single Holdout) · Cross-Validation (CV) · Evaluation Set · Expanding Window Cross-Validation · k-fold cross-validation · k-fold Stratified Cross-Validation (Stratified CV) · Multiclass stratified CV · Sliding Window (Rolling Window) Cross-Validation · Stratified Group K-Fold · Stratified Shuffle Split · Time-based splits (a.k.a. Temporal Cross-Validation, Rolling Window Validation)