💡 Support Vector Machines (SVMs)

Support Vector Machines (SVMs)#

Classifiers that find the maximum-margin boundary, optionally via kernels.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

A support vector machine is a supervised max-margin algorithm for classification (and, as SVR, regression). It finds the optimal separating hyperplane — the decision boundary that maximizes the margin, the distance to the nearest points of each class. Developed from the work of Vapnik and Chervonenkis, with the soft-margin form due to Cortes & Vapnik (1995).

\[\mathbf{w}^\top \mathbf{x} + b = 0 \quad\text{(the separating hyperplane)}\]

Margin and support vectors#

The support vectors are the training points closest to the boundary — and they alone define it (remove any other point and nothing changes). A hard margin separates the classes perfectly; a soft margin tolerates some violations through slack variables \(\xi_i\), with the penalty C trading margin width against misclassification (large C → stricter, narrower margin; small C → wider, more tolerant). The objective minimizes

\[\frac{1}{2}\|\mathbf{w}\|^2 + C \sum_i \xi_i,\]

where the per-point cost is the hinge loss \(\max(0,\, 1 - y_i(\mathbf{w}^\top \mathbf{x}_i + b))\).

The kernel trick#

When data is not linearly separable, a kernel maps it into a higher-dimensional space where it is — without ever computing the coordinates, using only pairwise dot products. Common kernels are linear, polynomial, RBF (the most popular) and sigmoid; the choice trades accuracy against complexity and compute.

When to use it#

SVMs are strong on high-dimensional data (text, images, bioinformatics), resilient to noise, and guard against overfitting, but are expensive on very large datasets and sensitive to kernel choice.

from sklearn.svm import SVC

clf = SVC(kernel="rbf", C=1.0)   # RBF kernel, soft margin
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

Theme: AI & ML Concepts · All terminology

Hint

Mind map — connected ideas

Logistic Regression · Neural Networks · Decision Trees · Linear Models · Multiclass AUROC · Discriminatory Power

Hint

More in AI & ML Concepts

AI (Artificial Intelligence) · Classification Models · Computer Vision (CV) · Decision Trees · Linear Models · LLMs (Large Language Models) · Logistic Regression · Machine Learning (ML) · Medical AI · Natural Language Processing (NLP) · Neural Networks · Regression Models · Target Variable