A/B Testing in Web Analytics: Getting Significance Right

The problem with "B is winning"

Run an A/B test for a day, and one variant will be ahead. That is guaranteed — two random samples almost never tie. The question is whether the difference is real or noise, and eyeballing the numbers cannot answer it.

A conversion test is a comparison of two proportions. The tool for that is a two-proportion z-test.

The maths, briefly

With cA conversions from nA visitors in control and cB from nB in variant:

pA = cA / nA
pB = cB / nB
pPooled = (cA + cB) / (nA + nB)

SE = sqrt( pPooled * (1 - pPooled) * (1/nA + 1/nB) )
z  = (pB - pA) / SE

A |z| above 1.96 corresponds to roughly 95% confidence — the conventional threshold for calling a result. Below that, you have not learned anything yet.

The thing to internalise: SE shrinks as sample size grows. Small samples produce large standard errors, which is why early results swing wildly and mean nothing.

In SQL

WITH v AS (
  SELECT variant,
         COUNT(DISTINCT user_id)                         AS visitors,
         COUNT(DISTINCT CASE WHEN converted THEN user_id END) AS conversions
  FROM ab_assignments
  WHERE test_id = ? AND assigned_at >= ?
  GROUP BY variant
)
SELECT variant, visitors, conversions,
       conversions::DOUBLE / NULLIF(visitors, 0) AS rate
FROM v;

Compute z from those four numbers in application code rather than SQL — it is clearer, and easier to test.

Mistake one: peeking

Checking a test repeatedly and stopping when it looks significant inflates the false positive rate badly. Check twenty times and you have roughly a one-in-three chance of a spurious "win" even when both variants are identical.

The fix is to decide the sample size before starting and not look until you reach it. If you must monitor, treat interim results as directional only.

Mistake two: samples too small to matter

To detect a change from 3% to 3.6% conversion — a 20% relative lift, which would be a good result — you need roughly 17,000 visitors per variant at 95% confidence and 80% power.

Most tests on most sites never reach that. This is not a reason to skip testing; it is a reason to test changes big enough to move a number that far, and to be honest when a test is underpowered.

Reporting honestly

A result should carry the sample size, the observed rates, and whether it cleared the threshold. "Variant B: 3.4% vs 3.1%, not yet significant (n=2,100 per variant)" is a useful sentence. "Variant B winning!" is not.

The most valuable thing an analytics tool can do here is refuse to declare a winner that the data does not support.