A/B Testing

A/B testing splits traffic between a control (A) and a variant (B) to measure the causal impact of a change. It removes guesswork from optimization by providing statistical evidence for decisions.

Running reliable A/B tests requires sufficient sample size, a clear primary metric, and proper randomization. Peeking at results before reaching statistical significance is a common mistake that leads to false conclusions.

Beyond simple two-variant tests, teams use multivariate tests (multiple changes at once) and multi-armed bandits (adaptive allocation). The right approach depends on traffic volume and the magnitude of expected effect.