Six Hours Without Facebook: What the Outage Taught Everyone Watching

On October 4, at midday Eastern time, Facebook disappeared from the internet. Not degraded, not slow. Gone, along with Instagram and WhatsApp, for roughly six hours. The outage’s mechanics were visible to every network engineer watching the global routing tables. Its effects reached, by the platforms’ own scale, several billion users and uncounted businesses. In the day’s most instructive detail, it reached Facebook’s own staff, who were reportedly hampered entering their own buildings because the badge systems depended on the same infrastructure that had vanished.

The company’s postmortem and Cloudflare’s outside-in analysis agree on the anatomy. A routine maintenance command went wrong. A configuration change withdrew the BGP routes announcing the networks where Facebook’s own DNS servers live. With the DNS unreachable, every name the company owns stopped resolving everywhere. And the tools needed to fix it remotely depended on the network that was down.

No attack, no data breach. A self-inflicted unreachability so complete it became, for everyone watching, a free masterclass in how the internet actually fits together, delivered at the expense of the one company that could afford the tuition. our job is to make the class stick. The plumbing literacy first, because these acronyms underpin your properties too. Then the audit, which is why this post exists.

The literacy half: an hour on BGP and DNS, worth having

The plumbing, in the plain language we owe its non-engineer readers, because digital leaders who understand these two layers make better resilience decisions forever after. BGP, the Border Gateway Protocol, is how networks announce where they live. The internet is a federation of networks, each continuously telling its neighbors ‘these addresses are reachable through me.’ The global routing that results is the map every packet follows. Withdraw your announcements, which is what Facebook’s errant command did, and you have not turned off your servers. You have erased your roads. Everything still runs, and nothing can be reached.

DNS is how names become addresses. Every visit to any site starts with resolving its name. The resolution follows a chain of servers. Suppose the servers that answer for your name are unreachable, say because the routes to them were withdrawn. Your name stops meaning anything, everywhere, at the speed of cache expiry.

The compounding lesson from October 4: failures at these layers cascade through everything built on the assumption of your existence. Apps polling APIs. Login buttons on other people’s sites. Pixels in a billion pages. The day measurably slowed other sites as embedded Facebook scripts timed out, a borrowed-outage footnote. And, per the badge readers, the physical world’s quiet dependence on the digital one. An hour of any leadership team’s time on this, with your own network people or ours, is among the cheapest resilience investments available. The single-points-of-failure map from 2019 is the worksheet: registrar, DNS, routing, the chain your own name hangs from.

The audit half: what did your operations lose for six hours?

• Run the inventory while the memory is fresh: list, concretely, what your organization could not do during the window. The login flows using Facebook authentication. The pixel-based measurement and ad platforms gone dark. The customer-service channels living in Messenger or WhatsApp. The communities and audiences reachable only through Instagram. The campaigns mid-flight with no landing alternative. For some organizations the honest list is short. For others it is an uncomfortable census of dependencies nobody ever approved as such.

• Grade each entry with the owned-alternative question: for each dependency, what is the owned-channel fallback? The email list where the Instagram audience also lives. The on-site chat beside the Messenger channel. The first-party login beside the social one. And is it real, populated and tested, or notional? The advice unchanged since the Google Reader shutdown, build on what you own, has never had a cleaner demonstration day, and the audit converts the demonstration into your own numbers.

• Fix the asymmetries, not the platforms: the advice is not to leave the platforms. They are where audiences are. It is to end the exclusive dependencies. The community that exists only there. The auth that only they provide. The support channel with no sibling. Each converted from single point to redundant pair.

Two notes on concentration and ownership

Two notes, written with the care the day’s discourse mostly skipped. First, the concentration note. The outage was not evidence that Facebook is fragile. Six hours in seventeen years is, by infrastructure standards, an excellent record. It was evidence of what concentration means when the improbable arrives. A meaningful fraction of the world’s communication, commerce, and identity ran in one failure domain, and the blast radius was the business model working as designed. The efficiency of concentration and the correlation of its failures are the same fact, priced differently on different days, and organizations position for the pricing they can survive.

Second, the owned-audience thesis, updated with its most vivid evidence. Rented infrastructure fails on the landlord’s schedule. It recovers on the landlord’s timeline and communicates on the landlord’s terms. Note that during the outage, the platforms’ own status pages and comms channels were partly down too, and the world got its news from the platforms’ competitors. None of the tenants had any say in any of it.

The email list sent that afternoon arrived. The website stayed up. The first-party channels did what owned things do, which is answer to their owners. Six hours of timestamped evidence. The thesis we have argued since 2013 did not need the help, but the evidence is welcome.

The one-paragraph version for the leadership meeting

On October 4 the internet demonstrated, at global scale and zero cost to you, exactly how the layers under your digital operations fail. It gave every organization a personalized six-hour sample of its own platform exposure.

The response that converts the demonstration into value costs an afternoon. The plumbing hour with your technical people. The dependency inventory while the sample is fresh. The owned-alternative grading. And the short list of asymmetries to fix this quarter, folded into the resilience map the 2019 files already prescribe.

The response that wastes it is the one the news cycle already performed. A day of fascination, a week of memes, and an unchanged architecture. We have been converting other people’s outages into its readers’ preparation since the first capacity postmortems in 2008, and the method has never had a better teaching day. Inventory your exposure while the memory is fresh. The landlord’s schedule is not published, and the next demonstration, per every file in this drawer, is already in the mail.