Experiment Forensics

The usual A/B testing pitfalls (checking too early, tests too small to see anything, winners that don't hold up) are easy to describe. I wanted to measure each one on real experiments, at real sizes.

Two public datasets made that possible. The Upworthy Research Archive holds 32,487 headline tests the site ran between 2013 and 2015: editors wrote several headlines for a story, and readers were randomly shown one. Each row is one headline (an “arm”) with its impressions and clicks. The Hillstrom email test is one experiment with 64,000 customers, randomly sent a men's email, a women's email or nothing, with what each had spent in the year before. Upworthy is big enough to study power, peeking and the winner's curse. Hillstrom has the per-customer history that variance reduction needs.

The archive also comes split into separate releases, so that people can explore on one and confirm on another. That turned out to be the most useful thing about it.

Everything was built on the exploratory release: 4,873 tests. The confirmatory release, 22,743 tests, wasn't opened. When the exploratory analysis was done, I wrote down every definition, exclusion, random seed and threshold in a plan, along with 13 numbers to compare and the result each would need to count as replicating. That went into git. The plan's fingerprint went in as a second commit, and the code refuses to read the confirmation file unless that lock is there and still matches the plan.

Then the confirmatory analysis ran once. It also stopped me later. After the run, I tidied a chart's axes and noted the change in the plan, and the next run refused: the plan had changed after it was locked. So changes like that now go in a separate DEVIATIONS file, and the plan stays as it was when the data was opened.

Before any analysis, there's a known problem. A correction the archive's authors published in 2024 says tests from 25 June 2013 to 10 January 2014, 22% of the archive, weren't properly randomised, and should be treated that way. I dropped every test whose first headline was created in that window: 4,907 of the 22,743 confirmation tests (21.6%), leaving 17,836.

The data backs the correction up. If readers are split evenly, each headline in a test should get about the same number of impressions. Inside the window, 70.1% of tests fail that check at p < 0.001. Outside it, 0.5% do.

Each headline was compared with the one written first in its test. Upworthy had no control headline, and the first one is fixed before any reader sees it, so it can't be chosen for how it did. That gives 67,851 comparisons. The median arm got 3,111 impressions, and the median click rate was 1.21%.

Observed differences are a mix of real effect and luck. Separating the two (the method is in the winner's-curse section below) puts the typical true difference at 0.19 percentage points. The typical test needed a difference of 0.74 points, about a 63% relative lift, to detect it reliably (80% power). Only 0.08% of tests were big enough to detect the typical true effect, and average power against it was 13%.

So most of these tests could only catch a headline that was dramatically better, which most of the time there wasn't.

The tempting habit is to check a test as data comes in and stop when it looks significant. To measure what that costs, I simulated 20,000 A/A tests, where both arms are identical, using click rates and sample sizes drawn from real Upworthy tests, and checked a standard z-test after every batch of readers.

Checked once at the end, 4.9% of A/A tests come out “significant”, as they should. Checked 10 times, 18.7%. Checked 100 times, 30.6%. With nothing to find, almost one test in three finds something.

There are tests built to be checked continuously. I used one, mSPRT, and its false-positive rate stayed under 0.4% however often it was checked. But that safety costs power, and these tests had none to spare: for a headline that's truly 50% better, one check at the end finds it 53% of the time, and mSPRT checked 100 times only 13%.

Of the tests with a headline that beat the first one, 3,820 (21.4% of all tests) won at p < 0.05, with a median lift of 0.79 percentage points. Picking winners that way picks up luck along with real improvement, so a re-run would usually show less.

To estimate how much less, I used empirical Bayes: learn from all 67,851 comparisons what true effects tend to look like, then estimate each winner's true effect given that and its own noise. The median “significant” winner shrinks by about 40%. Fitting the distribution of true effects two different ways gives nearly the same answer: 40.6% with a flexible shape, 39.7% with a normal curve. Only 38.3% of the significant winners have at least a 95% chance of truly beating the first headline.

The extreme cases shrink the most. One test reports 8 clicks from 115 readers against 1 from 111: a 6-point lift, significant at p = 0.02. After shrinkage it's 0.07 points. A typical winner, the same headline with a different image, goes from 0.91 points to 0.54.

While building the analysis, I noticed that some tests had two arms showing exactly the same headline, image and excerpt. That's an accidental A/A test, and in the confirmation set there are 2,011 such pairs. About 5% should differ at p < 0.05. 12.9% do.

So clicks vary between identical arms nearly twice as much as the standard model assumes (the average squared z-score is 1.92, where it should be 1). The data can't say why. It might be something about the arms that the archive doesn't record, or traffic that wasn't spread evenly over time. Either way, the standard errors everyone uses on data like this are probably too small, which makes every result above, if anything, too kind.

Accounting for it in the winner's-curse analysis raises the median shrinkage from about 40% to 88%, and only half the significant winners would still be significant. I report that as a sensitivity check rather than a headline, because I don't know the cause.

CUPED is a standard trick: use what you knew about each customer before the experiment to explain away some of the noise, so the same data gives a tighter estimate. On Hillstrom, the obvious pre-period variable is what each customer spent in the past year.

It made almost no difference. The men's email raised spend by $0.77 per customer, and the confidence interval was $0.57 wide with or without CUPED: a 0.04% variance reduction. Using every pre-period variable, or regression adjustment, didn't help either. On visits, the gain was about 3%.

The reason is in the data. Only 0.9% of customers spent anything in the two weeks, and past-year spending explains 0.02% of the variance in what they spent. CUPED can only remove what the covariate predicts, and here it predicts almost nothing. It's a useful method, but only when the past predicts the outcome, and that's worth checking before counting on it.

The last question is the one every campaign review asks: which customers responded most? I split Hillstrom into 23 subgroups (by recency, past spend, channel, location, new customers and past purchase categories) and asked two separate questions.

Does the email work within each group? For spend, 31 of 46 tests say yes at p < 0.05, and 11 survive the strictest correction (Holm). Does it work differently in one group than in everyone else? That's the question a segment story needs, and it's much harder: across both outcomes, 5 of 80 tests pass at p < 0.05, and 2 survive correction. Both are about visits, and both make sense: the women's email works better for customers who had bought women's merchandise, and worse for those who had bought men's. Every story about spend by segment fails.

All 13 pre-set checks met their criteria on the confirmation set, and most landed close to the exploratory value. Hillstrom is a single experiment with no second release, so its results stay exploratory.