🌊 Representation Shift

Representation Shift#

A change in learned feature representations between training and serving.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

Representation shift occurs when the internal representation of data — learned embeddings, feature vectors — changes over time between training and deployment, even if the raw input distribution looks similar. It is a special case of distribution shift, but focused on the feature/embedding space rather than the raw input.

Where it appears#

Three places. In neural networks and embeddings, the learned mapping can change (through retraining or new data), so downstream tasks built on the old space fail. In preprocessing pipelines, steps like TF-IDF, PCA or scaling drift as the data changes — TF-IDF weights move as new vocabulary dominates. And in domain shift, inputs that look similar can still drift in embedding space — a face model trained on frontal faces, deployed on side profiles.

Why it matters#

Downstream classifiers and regressors that assume a stable representation degrade; similarity search (nearest-neighbour in embedding space) returns wrong results; and fairness suffers if some groups’ embeddings drift more than others.

Detecting it, with an example#

Detection works on the embeddings themselves: distance metrics (MMD, energy distance, KL), tracking cosine or Euclidean shifts; visualisation with t-SNE or UMAP to watch clusters move; and classifier two-sample tests. The classic example: the word “mask” embedded mostly as cosmetic in 2019 shifts toward face covering in 2020 — breaking any downstream sentiment or topic model that relied on the old representation.

Theme: Distribution Shift & Drift · All terminology

Hint

Mind map — connected ideas

Embedding · Drift Detection · Covariate Drift (a.k.a. Covariate Shift) · Leading Indicators · Classifier Two-Sample Tests (C2STs) · Energy Distance

Hint

More in Distribution Shift & Drift

Cardinality in Categorical Data · Categorical Drift · Categorical Explosions · Classifier Two-Sample Tests (C2STs) · Concept Drift · Covariate Drift (a.k.a. Covariate Shift) · Data Drift · Dataset Shift · Drift Detection · Drift Guardrails · Energy Distance · Jensen–Shannon (JS) Divergence · KS shift (Kolmogorov–Smirnov shift) · Kullback–Leibler (KL) Divergence