Heartbleed: The Week the Padlock Broke

The security story of the year arrived April 7 with a marketing budget, which is itself a first. A disclosed vulnerability with a name, a logo, and a dedicated website explaining it. Courtesy of the Finnish firm that found it alongside Google’s researchers.

The substance earns the production values. The flaw, catalogued as CVE-2014-0160, sits in OpenSSL, the open-source encryption library that implements the padlock for most of the secure web. It is as bad as the coverage says. For two years, affected versions have let anyone read chunks of a server’s working memory, with no login and no trace. That memory can contain session cookies, passwords in transit, and in the worst case the server’s private encryption keys themselves. In a security incident, the message you send before you have all the answers matters as much as the patch.

The web’s operations teams have spent the week in triage, ours included. With the first wave now behind us, this post does the two jobs incident coverage should always attempt. The technical sequence, which is widely misordered in the public advice. And the communications lesson, which will outlast the patch.

The sequence, in the order that actually works

The response steps are simple. Their ordering is everything, and the misordering circulating this week, reset your passwords immediately!, manages to make things worse.

The correct sequence for site operators. First, patch the affected OpenSSL versions everywhere they run, because every other step is theater while the leak remains open.

Second, reissue and revoke certificates. If private keys may have leaked, and the flaw’s tracelessness means assuming so for affected servers, new certificates with new keys must replace the old, and the old must be revoked. Patching without rekeying locks the door while the copied key stays in circulation.

Third, and only then, credential resets. Passwords changed before the patch-and-rekey travel through the still-vulnerable channel and may simply hand the new secret to the same leak. Users told to reset everything on day one were told, for affected sites, to do it twice.

The sequencing logic generalizes to every layered incident and is worth keeping as a rule. Close the hole. Replace the compromised secrets that guard the channel. Then rotate the secrets that travel through it.

The inventory problem, which took longer than the fixes

The week’s honest surprise was not the patching, which is routine once targets are known. It was knowing the targets. ‘Which of our systems use OpenSSL?’ turns out to be a question most organizations cannot answer from their own records, because the library lives everywhere and nowhere. In web servers, load balancers, and mail systems. In VPN boxes, monitoring agents, and the embedded guts of vendor devices nobody thinks of as servers.

Our client-facing triage spent its first day on inventory. Version checks across every service that terminates TLS, vendor bulletins for the appliances. Its second day ran the sequence above. That ratio repeated across every competent response we watched.

The lesson slots directly into our dependency coverage, one layer down. Your stack includes libraries you have never consciously chosen, and incident response speed is set by how fast you can enumerate them. The site inventories our governance series prescribes for sites apply with equal force to the software layers beneath them. Organizations that maintain either kind answered this week’s question in hours while their peers grepped in the dark. The homework, assigned again with feeling: know what you run.

The communications playbook, tested in the field

Because users heard about this one from morning television, every organization with logins faced a communications decision this week, and the results were visible in public.

What failed: silence, since users assume the worst and hear from others. And its opposite, the boilerplate blast, ‘we take security seriously,’ that answers no actual question and reads as exactly what it is.

What held: the plain accounting, published promptly, structured as the three things users actually need. Whether you were affected, assessed honestly, including ‘we are verifying’ as an interim truth. What you did: the sequence above, in human language, with timestamps. And what the user should do and when: reset now, or wait for our all-clear, with the reasoning.

The institutions that shipped that structure inside two days, several of the large web platforms modeled it well, converted a frightening headline into a demonstrated competence. The contrast with the trust-eroding alternatives was visible in real time.

Our advice, filed for the inevitable next time. Draft the plain-accounting template now, in calm, with the blanks marked. Incident communications is a writing problem, not just a technical one, and the writing survives contact with crisis far better when most of it predates the crisis.

The questions clients asked, consolidated

The week’s inbox, answered once for everyone.

‘Were we affected?’ It depends entirely on the OpenSSL versions in your stack. The affected range is specific, and plenty of servers ran older, unaffected versions. That is an irony the patch-cadence moralists should sit with quietly. The inventory above is how the question gets answered rather than guessed.

‘Should we tell users even if we were not affected?’ Our advice: yes, briefly. Your users are hearing about this from television, and silence reads as either ignorance or concealment. Two sentences, ‘we have verified our systems were not running affected versions, and no action is needed,’ reads as competence.

‘Do we really have to revoke the old certificates?’ For affected servers, yes. The revocation infrastructure’s well-documented weaknesses, browsers check it unevenly, change the diligence obligation not at all. You control what you issue and retire, not how every client verifies it.

‘Is open source the problem here?’ The question arrives loaded and deserves the stewardship answer from our MySQL-era framework. The flaw was an ordinary human error, of the kind proprietary code hides privately. What open source changed was the speed and breadth of the response. The funding conversation now starting is the ecosystem’s actual lesson. The library was under-funded, not under-principled. The industry that built fortunes on it is about to be asked, correctly, to pay the maintenance it always owed.

What two years of silence should teach

The flaw sat in one of the world’s most depended-upon libraries for two years. It was introduced by an ordinary mistake and missed by everyone, while the project’s maintenance ran, famously as of this week, on a shoestring of volunteers and donations that would embarrass a parish newsletter.

The industry conversation now beginning, how the commons that everything depends on gets funded and reviewed, is overdue by exactly those two years, and we will track where it leads.

For readers, the local translation is the practice we keep prescribing, now with its sharpest example. Dependencies deserve inventories. Incidents deserve rehearsed sequences and pre-drafted words. And the padlock deserves the same governance attention as anything else that carries the institution’s trust, because this month the padlock itself was the weak point. Trust-but-verify has reached the crypto layer. There is, we are fairly confident, nowhere deeper for it to go, which is the only comforting sentence this week’s file supports.