🏋️ Hyperparameter

Hyperparameter#

A setting fixed before training (e.g. learning rate) rather than learned.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A hyperparameter is a setting chosen before training that controls how a model learns or how it is structured — as opposed to parameters (weights and biases), which the optimiser learns from the data. Hyperparameters are the model’s learning strategy; parameters are its learned knowledge.

The main kinds#

They span four areas. Model structure: number of layers, neurons per layer, a tree’s max depth. Training process: learning rate, batch size, number of epochs, dropout rate. Regularisation: L1/L2 penalty strength, weight decay. Optimisation: SGD momentum, the Adam beta values.

Why they matter#

The same model with different hyperparameters can perform very differently — poor choices cause underfitting, overfitting or painfully slow training, so tuning is critical for accuracy and generalisation.

Tuning, and an example#

Common strategies escalate in sophistication: manual trial, grid search (every combination), random search (sample combinations — often more efficient), Bayesian optimisation (a probabilistic model guides the search), and Hyperband / population-based methods at scale. Training a net on MNIST one might fix learning rate 0.001, batch size 64, dropout 0.5 and 20 epochs; the weights are learned automatically, but performance hinges on those chosen hyperparameters.

Theme: Model Training & Optimization · All terminology

Hint

Mind map — connected ideas

Epochs · Cross-Validation (CV) · Early Stopping · Neural Networks · Model Distillation (Knowledge Distillation) · Quantization

Hint

More in Model Training & Optimization

Active Learning · Binary Cross-Entropy (BCE) · Deep Ensembles · Early Stopping · Ensemble · Epochs · FLOPs · Full Annotation · Label Noise · Log-Odds · Logit Space · Logits · Loss Functions · Model Distillation (Knowledge Distillation)