Pokémon GO and the Traffic Spike Nobody Budgets For

On July 6, Niantic launched Pokémon GO in Australia, New Zealand, and the United States. By the following week the game was, by every visible measure, the fastest-adopted application anyone in this industry can remember. It topped both app stores within days. It generated street scenes of players clustering at landmarks that made the evening news. By mid-month it had reportedly passed the daily-use figures of applications that took years to build their audiences.

It also spent its first weeks visibly straining. Server errors at peak hours. Login queues. The international rollout deliberately staggered country by country while capacity chased demand, with the Japan launch held back weeks, in a game about Japanese monsters.

The engineering details will surface eventually. They always do, in conference talks and postmortems. But the shape of the event is already clear from the outside, and the shape is the lesson. This is a capacity-planning story of a magnitude the discipline rarely gets to observe in public. It teaches everyone who runs anything on the internet, at every scale below ‘global phenomenon.’ Which is to say: everyone reading this.

What the public record shows, held to what it shows

House rules first. The numbers circulating (multiples past worst-case projections, orders of magnitude in days) are press estimates and reasonable inference, not published engineering data. This post will not launder rumor into citation.

What the outside view establishes without any leaked figures: demand exceeded planned capacity by margins large enough to force weeks of visible degradation and a rebuilt rollout schedule. At a company backed by Google-lineage infrastructure and cloud capacity that can, in principle, scale as fast as anything on earth.

Hold that last clause, because it carries the first lesson. Elastic capacity is not instant capacity. Cloud scaling solves the procurement problem our capacity pieces used to center. Nobody waited for servers to ship. But the bottlenecks above the hardware scale by engineering, not by dial. Database write paths. Session architecture. The login system every user hits at once. A demand curve that goes vertical finds every one of them in sequence.

The AWS-era lesson we filed in 2011 said the constraint moves up the stack when the hardware constraint dissolves. July was that lesson demonstrated at planetary scale, on the news, nightly.

The staggered rollout, which is the teachable move

The decision worth stealing is the one Niantic made under fire. They stopped launching. With demand vertical and systems saturated, the country-by-country rollout became a throttle. Each new market opened only as capacity caught up. Japan, predictably the largest single demand event, was held until the platform could survive it.

Translate that to the scale our readers operate. The launch calendar is a load-management instrument, and you are allowed to use it.

The admissions-decision release that opens by school rather than all at once. The registration window tiered by cohort. The campaign email sent in waves rather than as a single million-recipient detonation timed to a minute, a self-inflicted spike our email coverage has warned about for years. The product announcement that soft-launches before the press release.

Every one of these is the same move. Shaping the demand curve you control, rather than provisioning for the curve’s worst spike. It is unshowy. It requires only a calendar and the willingness to tell stakeholders that simultaneity is a cost rather than a virtue. And it is the cheapest capacity engineering that exists.

The second cheapest is the caching ladder covered here since 2007, which converts the spike’s read traffic into CDN traffic that never touches your systems at all. That remains the reason well-prepared launch days are boring.

The transferable checklist

• Re-derive your worst case, then multiply. July’s core math lesson: the worst-case projection is an estimate built from history, and history does not contain the event that breaks it. Your registration morning, your viral moment, your news-cycle mention will not be fifty times projection. But ten times is well within the observed range of ordinary institutional surprises, and the plan that survives ten times is a different plan than the one that survives two.

• Find your login system’s ceiling. The pattern in every public strain event, this one included: the logged-in path fails first, because it is the path caching cannot absorb. Load-test the login, the form submission, and the personalized page. Know the number before the event does.

• Decide the degradation in advance. The static fallback page. The queue with an honest message. The read-only mode. Graceful degradation is a design decision made calmly, or an improvisation made at 2 a.m., per the incident-response practice. Niantic’s error screens became memes. Yours can be a planned, branded, informative page instead.

• Rehearse the surge. The tabletop exercise our security series prescribes has a capacity edition. Pick the plausible spike scenario, walk the team through the first hour, and find the gaps at conference-room prices.

Two closing notes

Two notes before the month closes. First, the honesty note. The full engineering account of this event is not yet public, and the paragraphs above are outside inference, labeled as such.

Second, the wider observation, because the game itself deserves one line beyond the infrastructure. An augmented-reality layer draped over public space produced, in weeks, foot-traffic effects that businesses and institutions noticed. The sponsored-location experiments are already being reported, and the location-layer story we opened in the check-in era just acquired its largest data point. Whether that story matures or fades with the game’s novelty is a question for a future post.

The capacity lesson does not fade either way. Your worst case is an estimate. Your architecture is a fact. July was the month the whole industry watched the difference. The teams that convert the watching into a load test will be the ones reading the next such story the way our patched clients read Drupalgeddon. With sympathy, from a distance. The drill costs an afternoon. Schedule it while the news footage is still fresh enough to win the meeting.