💡 Decision Trees

Decision Trees#

Models that split data on feature thresholds to reach predictions.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A decision tree is a supervised-learning algorithm that splits data into branches by feature values, forming a tree. Each internal node is a decision (a feature and a threshold) and each leaf a prediction (a class label or a number) — think of it as a flowchart: ask questions, follow branches, reach a prediction. It comes in two flavours: classification trees (discrete labels) and regression trees (continuous values).

How it learns, and splitting#

Training is recursive: start with all data at the root, evaluate candidate splits for each feature, keep the split that best separates the data (minimises impurity), and repeat until a stopping rule (max depth, minimum samples per leaf). Classification trees split by Gini impurity or entropy (information gain); regression trees by MSE reduction. The Gini impurity at a node is

\[G = 1 - \sum_{k} p_k^2,\]

where \(p_k\) is the proportion of class \(k\) at that node.

Strengths and weaknesses#

Trees are easy to interpret and visualise, handle mixed numeric and categorical features, capture nonlinear boundaries and interactions, and need no feature scaling. But a deep tree overfits, is unstable (a small data change reshapes it), splits greedily (it can miss the global optimum), and is weaker alone than an ensemble.

From one tree to many#

Pruning cuts back branches to curb overfitting; random forests bag many trees; and gradient-boosted trees (XGBoost, LightGBM, CatBoost) build trees sequentially to correct earlier errors.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, export_text

X, y = load_iris(return_X_y=True)
tree = DecisionTreeClassifier(max_depth=3).fit(X, y)

print(export_text(tree, feature_names=["sepal_length", "sepal_width",
                                       "petal_length", "petal_width"]))

Theme: AI & ML Concepts · All terminology

Hint

Mind map — connected ideas

Post-hoc Explainability · Deep Ensembles · Uplift Random Forests · SHAP (SHapley Additive exPlanations) · Discriminatory Power · Bayesian Neural Networks (BNNs)

Hint

More in AI & ML Concepts

AI (Artificial Intelligence) · Classification Models · Computer Vision (CV) · Linear Models · LLMs (Large Language Models) · Logistic Regression · Machine Learning (ML) · Medical AI · Natural Language Processing (NLP) · Neural Networks · Regression Models · Support Vector Machines (SVMs) · Target Variable