Lessons from the EC2 Outage: The Cloud Is Someone Else's Computer

The cloud had its first great outage last week, and the casualty list made the point better than any lecture. Reddit, Quora, Foursquare, and hundreds of quieter services spent the better part of April 21 degraded or dark. A failure in Amazon Web Services’ US-East region cascaded through the storage layer beneath them. The cloud is somebody else’s computer, and somebody else’s computer has bad days. Plan for its bad day.

Recovery stretched across days. Amazon’s post-mortem concedes that a small fraction of affected storage volumes were lost outright. Not delayed. Lost.

For an industry that has spent three years, ourselves included, describing the utility-computing trend in mostly admiring terms, the week is a syllabus. The lessons are more precise than the ‘cloud fails too’ headline suggests. Let’s do the precise version, because our own sorting-exercise coverage owes readers the update.

What actually failed, translated

The mechanics, per Amazon’s unusually detailed public post-mortem. A network configuration change during routine maintenance set off a cascade in the Elastic Block Store system, the virtual disks behind most EC2 servers, within one availability zone of the US-East region. The recovery machinery itself then made things worse. Storage volumes frantically re-copying themselves ate the capacity needed to heal. A stampede, which the post-mortem describes with admirable candor.

The crucial vocabulary for what follows. AWS divides each region into multiple availability zones. These are separately powered, separately networked facilities built to fail on their own, precisely so customers can build across them. This outage largely respected those boundaries. It savaged one zone. The region’s others kept working.

Which is the instructive detail the headlines skipped. The sites that went dark were, overwhelmingly, deployed in a single zone. The architecture equivalent of one server in one closet, at cloud prices.

The services built across zones (and a few, famously, across regions) rode it out. Netflix is among the most-cited survivors, whose engineering culture of assuming failure is suddenly enjoying its told-you-so week. The failure was not really ‘the cloud.’ It was single-point-of-failure architecture wearing cloud pricing. The distinction matters, because one is a vendor problem and the other is a decision your organization controls.

The assumptions the week retired

• ‘The cloud is always up.’ Retired, permanently, and good riddance. No infrastructure is always up, and planning that assumes otherwise is not planning. What utility platforms genuinely offer is not immunity but options. Redundancy you can buy with settings rather than capital, at sizes no server closet can match. Options unexercised, as Thursday showed at scale, protect no one.

• ‘Redundancy is included.’ The subtler retirement. Cloud marketing’s honest fine print has always separated the durability of the platform from the uptime of your particular arrangement. The arrangement is the customer’s design. Multi-zone deployment costs more and requires architecture. That is the redundancy line item, and it was always a line item, not a gift.

• ‘Backups are for hardware failures.’ The lost-volumes footnote deserves its own bullet. A fraction of a percent of storage in the affected zone could not be recovered. At AWS scale that is many customers’ data, and ‘the cloud lost it’ consoles no one. Backups to somewhere else remain non-negotiable, cloud or otherwise. Different zone, different region, different provider. The organizations that tested restores this week from copies they controlled had a bad day. The others had a category of day we hope our maintenance-contract readers never meet.

The questions every hosting arrangement owes you

Scaling this to our readership, most of whom run institutional sites on managed hosting and watch the cloud story from a sensible distance: the outage supplies the interview that every arrangement, cloud or not, should be able to pass.

What is the failure domain? When something breaks, what is the blast radius? One server, one rack, one facility, one region? If nobody can answer, the answer is ‘you’ll find out.’

What is our recovery time objective, honestly? Not the aspiration. The tested number, for the realistic scenarios: hardware death, facility outage, data corruption.

Has failover ever been rehearsed? A failover plan that has never run is a document, not a capability. Thursday’s field data on this point is now definitive.

Where do the backups live, and when did we last restore one? The question our resilience-mapping piece made standard. This week upgraded it from prudent to proven.

For most mid-market sites, the answers will be modest. Single-server hosting with good backups and a tested rebuild path is a reasonable choice at many budgets. But reasonable-by-decision and reasonable-by-default are different things. Only one of them survives contact with a Thursday like this one gracefully.

What the survivors did differently, itemized

Because ‘architect across zones’ deserves its practical unpacking, the survivor patterns worth studying, scaled to what they teach rather than what they cost.

The multi-zone deployers ran their applications and databases copied across zones, with failover that had been exercised. The load balancer shifted traffic when a zone degraded, because someone had rehearsed exactly that shift on an ordinary Tuesday.

The design-for-failure school, Netflix being the loudest evangelist, goes further. Assume components fail constantly. Build the application to route around loss as its normal operating mode. And, the memorable part, deliberately break things in production to keep the routing honest.

That philosophy is beyond most institutional budgets, and rightly so. Its transferable core is not. Assume failure, rehearse response. That scales to any size. The single-server site whose team has actually restored from backup, actually rebuilt on fresh hosting, actually timed the process, holds the small version of what Netflix holds at large. Verified knowledge of its own recovery. The industry’s default is hope with paperwork. The outage sorted the two groups with perfect accuracy. It always does.

The sorting-exercise update

Where this leaves our cloud shelf-map: unchanged in direction, usefully chastened in tone. The trend we have tracked since 2008, capacity as utility, infrastructure moving down-market, remains real and remains the future. Nothing about one zone’s bad week repeals the economics.

What the week priced correctly, at last, is the operational maturity the platforms still demand. The tooling. The architecture discipline. The failure-domain literacy that the keynotes skip and the post-mortems teach.

Our advice keeps its shape. Adopt the low-risk edges, design for the middle shelf, and let other people’s production incidents mature the platforms. A sentence we wrote in 2008, confirmed last Thursday.

The cloud is someone else’s computer, professionally run and honestly documented. It is not someone else’s responsibility. That part never outsources. The organizations that internalize it get to enjoy the economics with their eyes open, which was always the only recommended way to enjoy them.