🧬 Autoencoder

Autoencoder#

A neural network trained to reconstruct its input through a compressed latent code.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

An autoencoder is a neural network trained to copy its input to its output through a deliberate bottleneck. Because the network must pass everything it knows about an example through a representation that is too small to hold the raw input, it is forced to discover and keep only the structure that actually matters and to discard noise and redundancy. The small middle representation — the latent code — is the part we usually care about.

How it works#

An autoencoder has three pieces:

an encoder \(f_\theta\) that maps an input \(x\) to a low-dimensional code \(z\);
a bottleneck (the latent space) that holds \(z\), whose dimension is much smaller than the input;
a decoder \(g_\phi\) that reconstructs an approximation \(\hat{x}\) of the original from \(z\).

Encoder and decoder are trained together to minimise a reconstruction loss — typically mean squared error for continuous data or cross-entropy for binary or categorical data:

\[z = f_\theta(x), \qquad \hat{x} = g_\phi(z), \qquad \min_{\theta,\phi}\; L\big(x,\; g_\phi(f_\theta(x))\big).\]

Nothing about the target requires labels — the input is the target — so an autoencoder learns in a fully self-supervised way.

Common variants#

Denoising — corrupt the input and ask the network to reconstruct the clean version, which forces robust features.
Sparse — penalise the code so that most latent units are inactive for any given input.
Variational (VAE) — make the latent space probabilistic, which regularises it and turns the decoder into a generator of new samples.
Convolutional — build encoder and decoder from convolutional layers, the natural choice for images.

Where it’s used#

Non-linear dimensionality reduction — a more flexible alternative to PCA.
Denoising of images, audio or text.
Anomaly detection — examples the model reconstructs badly (high error) are flagged as unusual, which is useful for fraud, intrusion and fault detection.
Representation learning — the latent code becomes a feature vector for downstream models.

Worked example#

Feed a 28×28 handwritten-digit image (784 pixels) through an encoder that compresses it to a 32-dimensional code, then a decoder that expands it back to 784 pixels. Trained well, the reconstruction looks almost identical to the original, yet the 32-number code captures the essence of the digit — enough to cluster, search or detect outliers in a fraction of the original space.

Theme: Representations & Embeddings · All terminology

Hint

Mind map — connected ideas

Embedding · Embedding Similarity · Frozen Encoder

Hint

More in Representations & Embeddings

Embedding · Embedding Similarity · Frozen Encoder