Decision Trees as Piecewise Models and Their Predictive Structure#

Stage 7 · 🌳 Decision Trees · Lesson 46 of 56 · advanced

◀ Previous · The CART Algorithm · Next · How CART Decision Trees Model Interactions ▶ · ↑ Section

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

Boxes in feature space#

A decision tree has a clean geometric meaning. Every split is a test on one feature against a threshold (“distance ≤ 3?”), so each split is a straight cut parallel to an axis of the feature space. A sequence of such cuts carves the space into rectangular regions — boxes (or, in higher dimensions, hyper-rectangles). Crucially, each leaf of the tree corresponds to exactly one region: the set of points whose feature values follow that root-to-leaf path.

A constant in each box#

Inside each region, the tree makes a single, constant prediction. For a regression tree it is the mean of the training outcomes that fell in that box; for a classification tree, the majority class. So predicting for a new point is just: find which box it lands in, and return that box’s constant. The tree is, in effect, a lookup table over a partition of the feature space.

A step-function model#

This makes a tree a piecewise-constant model. A regression tree’s prediction surface is not a smooth line or plane but a series of flat steps — constant within each box, jumping at the boundaries. Algebraically it is a sum over regions, \(T(x) = \sum_{j} \gamma_j \, \mathbb{I}(x \in R_j)\), where each \(R_j\) is a box and \(\gamma_j\) its constant prediction. This is a sharp contrast to linear regression’s single continuous slope.

Flexible despite steps#

A staircase might seem too crude to model a smooth relationship — but with enough boxes, a piecewise-constant function can approximate any shape, including the non-linear and non-monotone forms that linear or additive models cannot represent. That flexibility is the tree’s power, and also its peril: enough boxes to fit any curve is also enough to fit the noise, which is why the depth controls and pruning of the last lesson matter. The next lesson shows a subtler consequence of this box structure: interactions.

Hint