Test, Don't Argue: Getting Started with Website Optimizer

The most productive sentence available in any redesign meeting is ‘let’s test it.’ The headline debate that has consumed three meetings. The form-length standoff between marketing and legal. The eternal argument about the button. Each can be settled in two weeks by the only participants whose opinion converts: your visitors.

What has changed is the price of asking them. Google Website Optimizer, opened to everyone late last year, makes controlled experiments free. With the tool problem solved, the remaining barriers are all discipline. This post covers both.

What the tool actually does

Website Optimizer runs two kinds of experiments. An A/B test splits traffic between whole page versions, your current landing page against a challenger, and measures which sends more visitors to a conversion you define. Usually that is the thank-you page.

A multivariate test varies elements within one page and measures which combination performs. Think three headlines, two images, and two buttons. That sounds more sophisticated and usually is not the right start. Combinations multiply, and each needs traffic to reach a verdict.

Our usual advice for first-year testing programs: A/B tests of meaningfully different versions, on your highest-traffic conversion path. Bold variants, big pages, one question at a time.

Implementation is a matter of tags. A few snippets on the test page and the conversion page, versions defined in the interface, traffic split automatically. Any team that can install analytics can install this. After eighteen months of watching Google give away capabilities that used to cost five figures, we have stopped being surprised. The strategy behind the generosity is simple enough: advertisers who convert better spend more. Take the tool anyway.

Choosing a hypothesis worth the traffic

Testing spends visitor traffic the way research spends grant money: on questions. Most first tests spend it badly, on trivia. Button colors. Font tweaks. The digital equivalent of rearranging the brochure rack. The tests that pay answer questions with real dollar ranges attached.

• The value proposition. Does the page lead with what visitors get, or what the organization is? Test a benefit-first headline against your current one. This is reliably the highest-stakes element on any landing page and the least tested, because it is the one everyone has feelings about.

• Form length. Every field is friction. Some fields are necessary. Opinion currently decides which. Test the seven-field version against a three-field version and price the difference in completions against the value of the extra data. In our experience the short form wins more often than the data-hungry expect, but not always. Audiences researching a major decision will tolerate substantial forms. That is why you test.

• Proof and reassurance. Testimonials, credentials, security language near the form. Elements that cost nothing to add and might move trust-sensitive conversions like donations and applications. Cheap variants, real stakes.

• The call to action’s promise. ‘Submit’ against ‘Get the guide’ against ‘Start your application.’ Not the color, the commitment being described. Language that clarifies what happens next tends to travel well in every published test archive, but your audience holds the deciding vote.

The peeking problem, and other ways tests lie

Now the discipline section, earned by watching enthusiastic teams reach wrong conclusions with real tools. The cardinal sin is peeking: checking results daily and declaring victory the moment the chart looks decisive. Early test data swings wildly. A Tuesday’s worth of visitors is an anecdote, not a sample. Stopping at the first exciting swing turns noise into policy.

Decide the test’s duration or sample size in advance, then let it run. Low-traffic pages need weeks, not days. Everyone peeks. The discipline is not acting on the peek.

Second sin: testing during anomalies. The week of your email blast, the season’s application deadline, a press mention. Each skews who is arriving and what they want. Run tests across normal weeks, or at least across whole cycles, so the verdict describes your ordinary audience.

Third: changing the test mid-flight. New variant idea? New test. Edited copy on a running variant poisons the comparison, and the tool cannot warn you about what it cannot see.

And when a test ends with no significant difference, and many will, that is a finding, not a failure. The element you argued about for three meetings does not matter to visitors, and the meetings can stop. Negative results retire arguments too. Log them.

Sector notes, from tests we’d love to see run

Because our readership skews institutional, the test ideas worth stealing by sector. Higher ed: the request-information form, including length and field order. Does asking for intended major up front help or hurt? Also the visit-scheduling page’s calls to action, and whether prominent outcomes data moves inquiry rates on program pages.

Nonprofits: the donation page above all. Suggested gift amounts and their ordering, monthly-gift prominence, reassurance language near the payment fields. Small percentage moves here are real dollars. December’s giving season is the traffic to test into, not during. Run the tests now and bank the winners by November.

Professional services and B2B: gated versus ungated content. Does requiring registration for the white paper produce more qualified leads or just fewer leads? That one settles a religious war per firm, and the answer differs by firm, which is rather the point of testing.

When results land, report them as decisions, not decimals. ‘The short form won, we project a real lift in completions, it’s live’ travels through leadership meetings in a way conversion-rate tables never will. And archive every test somewhere findable: hypothesis, variants, verdict. Two years of archived tests is an institutional asset that outlasts any staff transition. We have watched exactly one organization maintain such an archive, and their meetings are shorter than yours.

The compounding case

Why this belongs in your quarter’s plans, in math. Every dollar you spend driving traffic buys visits that convert at some rate. Improve the rate twenty percent and you have improved the return on all of it, retroactively and henceforth, without buying another visit.

That is the compounding property that makes testing the rare marketing investment that appreciates. Winning variants keep winning after the test ends, and a program that banks a few real wins a year transforms its economics.

Start modestly and honestly. Your highest-traffic landing page, your longest-running internal argument, one bold A/B test, run to completion. Whichever way it lands, something ends: the argument, or the underperforming page. Both endings pay. And the meeting where you present visitor-decided results instead of opinion is the meeting where your organization’s relationship with its website starts changing. We have watched it happen, and it does not reverse.