🧫 Online Experimentation Platforms

Online Experimentation Platforms#

Systems that randomise, run and analyse controlled experiments on live traffic.

Important

✨ AI-generated content. This page was written with the assistance of an AI language model and is provided as a learning aid. Despite careful review, it may still contain mistakes, omissions, or out-of-date information. Whether you are new to the topic, a team lead, or a senior practitioner, treat it as a starting point rather than an authoritative reference: read it critically and independently verify anything you act on (code, commands, figures, and factual claims) against official documentation and primary sources before relying on it.

What it is#

An online experimentation platform is the infrastructure that lets a company run controlled experiments — A/B tests, multivariate tests, bandits — on a live digital product, usually built internally at scale. It gives product managers, data scientists and engineers a way to randomly assign users to variants, log outcomes automatically, analyse them with sound statistics, and enforce guardrails so experiments don’t harm users.

Core components#

Assignment & randomisation — bucket users (by cookie, device or account) into variants, with stable assignment so a user always sees the same one.
Data collection & logging — track clicks, conversions, retention and revenue consistently across pipelines.
Metrics framework — predefined primary KPIs plus guardrail metrics (engagement should rise, latency must not).
Statistical engine — frequentist (p-values, fixed-horizon or sequential), Bayesian (posteriors, credible intervals), and variance reduction (CUPED) or stratified sampling.
Dashboard — automated significance, effect sizes and time series.
Governance / guardrails — flag harmful launches and enforce GDPR/HIPAA compliance.

Modern capabilities#

The leading platforms add sequential testing (safe peeking), Bayesian analysis, multi-armed bandits (adaptive traffic), heterogeneous treatment-effect / CATE analysis by segment, ML-driven personalisation, and scale to hundreds of simultaneous tests.

Who builds them#

Commercial tools include Optimizely, Adobe Target, VWO and feature-flag platforms like Split.io / LaunchDarkly (Google Optimize was retired in favour of GA4). The big platforms run their own: Microsoft’s ExP (Bing, Office, Windows), Airbnb’s ERF, Uber’s and Netflix’s XP infrastructure, LinkedIn’s XLNT, and Meta’s PlanOut.

Hard problems#

Running thousands of concurrent experiments raises real challenges: statistical validity at scale (many simultaneous tests), interference / contamination (users caught in several experiments), metric definition (does the KPI track product health?), sheer scale, and ethics (avoiding harmful or unfair experiments). The payoff is evidence-based decisions instead of intuition, lower launch risk, and continuous optimisation.

Theme: A/B Testing & Experimentation · All terminology

Hint

Mind map — connected ideas

Optimizely · Google Experiments · A/B Testing · Stopping Rules · Sequential Settings · Bandit Algorithms

Hint

More in A/B Testing & Experimentation

A/B Testing · A/B/n Test · Bayesian Sequential Testing · Bayesian Stopping Rules · Conversion Rate Uplift · Fixed-Horizon Testing · Group Sequential Testing · Multivariate Test (MVT) · Optimizely · Risk of Peeking · Sequential Testing (also called sequential analysis) · Stopping Rules · Traditional A/B Test (Fixed-Horizon A/B Test) · Treatment Effect