Your A/B Test Is Probably Lying to You

Five years ago, when we first covered controlled experiments, the barrier was tooling and the advice was ‘start.’ The tooling barrier is gone. Optimizely and Visual Website Optimizer have made launching a test a twenty-minute task. Google folded its old Website Optimizer into Analytics as Content Experiments. Testing has spread to the point where reviewing existing experiments is now more common than we propose first ones.

Which surfaces the 2013 problem, visible in a year of those reviews: most of the tests are lying to their owners. Not through tool failure but through method failure, and the lies are flattering enough that nobody audits them. The same three failures account for nearly everything we find, and the fixes are discipline rather than software. This post is the audit, delivered as gently as we can.

Failure one: peeking, the cardinal sin

The pattern. A test launches. The team checks results daily, which is natural. The moment the dashboard shows a decisive-looking lead, someone declares victory and ships, which is fatal.

Early experiment data swings wildly. A few days of visitors is an anecdote, not a sample. A lead that looks commanding on Tuesday routinely evaporates by the following week as the sample grows. Stopping at the first exciting swing does not merely risk error. It systematically selects for it, because you are choosing the moment when noise looks most like signal.

The discipline: decide the test’s duration or sample size before launch, then let it run. The tools’ own calculators help, and modest math beats optimism. Low-traffic pages need weeks, sometimes months, and some pages honestly lack the traffic to test at all, which is a finding worth having.

Everyone peeks. Peeking is human. The discipline is not acting on the peek. The teams that make it routine (test end-dates on the calendar, results reviewed only at close) stop shipping noise.

Also in this family: changing a variant mid-test. New idea means new test. Edits poison the comparison. And running tests through unusual weeks (the email blast, the press hit, the application deadline) whose visitors do not represent your ordinary audience.

Failure two: trivia, the quiet budget leak

Traffic is a testing program’s research budget, and most programs spend it on questions with no dollar range attached. The button-color test is the emblem. Cheap to conceive, easy to run, and almost always inconsequential, because visitors do not abandon donations over hue.

Meanwhile the questions with real stakes go untested. The value proposition: does the page lead with what visitors get, or what the organization is? Form length: every field priced against completions, per the write-path economics our capacity series taught. Proof and reassurance near the moments of commitment. The call to action’s actual promise.

The reframe we push in every program review: rank your test backlog by the revenue or mission value of the behavior being influenced, times the plausible movement. Trivia sinks to the bottom of that list by arithmetic. The uncomfortable, valuable tests rise: the ones touching content and offers, which require stakeholder courage.

If your last five tests could all ship without anyone in leadership caring about the outcome, the program is exercising the tool, not the strategy.

Failure three: tests without hypotheses

The deepest failure is structural. Tests conceived as ‘let’s try things’ produce results that mean nothing beyond themselves.

A proper hypothesis has the shape: we believe this audience hesitates at this point because of this evidence. Variant B addresses it by this mechanism. We expect this metric to move.

The evidence clause is where our research toolkit feeds in. Analytics showing where users stall. Session recordings. Search logs. The usability findings that explain the why behind the where.

Tests built this way pay twice. The result moves a metric, and the result teaches something about your audience that transfers to the next decision.

Which points to the cultural fix that separates mature programs from tool subscriptions: a testing program is a research operation that accumulates customer insight. Keep the archive findable. Hypothesis, variants, verdict, interpretation. Every test. Review it quarterly. Let it compound. We know exactly one organization with a two-year archive of this kind, and their planning meetings are shorter and sharper than anyone else’s we attend, because arguments that recur elsewhere were settled there by test fourteen.

The organizational failure behind the methodological ones

One more pattern from the program reviews, because it explains why the three failures persist despite being widely known. Testing programs get judged by test volume, and volume incentives manufacture exactly the pathologies above.

The dashboard that reports ‘tests run this quarter’ rewards quick trivia over patient substance. The vendor relationship priced per test rewards the same. And the team praised for shipping winners learns to peek, because peeked winners arrive faster.

The fix is changing what leadership sees. Our recommended program dashboard reports decisions changed, projected value of shipped winners with honest ranges, and insights added to the archive, with test count demoted to a footnote.

Two clients have made that reporting switch this year, and the behavioral change followed within a quarter, without a single methodology lecture. Teams optimize what the dashboard honors, in testing as everywhere. If your program’s incentives reward volume, the methodology training is pushing uphill. Fix the dashboard first and the discipline mostly installs itself. Measurement of the measurers, as it were. The same lesson our analytics series keeps finding at every layer: the instrument shapes the behavior it observes.

The honest-program checklist

Compressed for the wall. Hypotheses grounded in evidence. Sample size decided before launch and honored after. One meaningful change per test. Normal-period traffic. Results reviewed at close, not daily. Negative and null results logged as findings, because the element you argued about for three meetings not mattering is information, and the meetings can stop. And the archive maintained.

Programs run this way produce fewer tests than the slot-machine style, and the fewer tests are worth something, which is the entire trade. The tools made experimentation cheap. The discipline is what makes it true. And true is the only kind of testing that compounds into the customer understanding your redesigns, campaigns, and content strategy can actually stand on.

Audit your last quarter’s tests against the checklist this week. If they pass, you are the organization with the archive, and we would like to attend your meetings. If they do not, the fix costs nothing but patience, and the patience pays at the only table that matters: the next decision.