🧷 Evaluation Set

Evaluation Set#

Held-out data used to measure model performance.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

An evaluation set is data held out from training so a model can be scored on examples it has never seen — the only honest way to estimate how it will generalize. In practice it splits into two roles.

Validation vs test#

The validation set is used repeatedly during development — tuning hyperparameters, early stopping, choosing between models; the test set is touched once, at the very end, for a final unbiased estimate. Any peek at the test set during development contaminates it and inflates the reported score.

The cardinal rule#

Split first, then fit every preprocessing step (scaling, encoding) on the training data only and apply it to the held-out sets. Fitting on all the data before splitting leaks information from the evaluation set into training — the classic data leakage that makes scores look better than reality.

Theme: Validation & Cross-Validation · All terminology

Hint

Mind map — connected ideas

k-fold Stratified Cross-Validation (Stratified CV) · Time-based splits (a.k.a. Temporal Cross-Validation, Rolling Window Validation) · Cross-Validation (CV) · Model Score · Model Stability · Data Drift

Hint

More in Validation & Cross-Validation

Blocked Splits (Single Holdout) · Cross-Validation (CV) · Data Leakage · Expanding Window Cross-Validation · k-fold cross-validation · k-fold Stratified Cross-Validation (Stratified CV) · Multiclass stratified CV · Sliding Window (Rolling Window) Cross-Validation · Stratified Group K-Fold · Stratified Shuffle Split · Time-based splits (a.k.a. Temporal Cross-Validation, Rolling Window Validation)