Seven Hours: What Drupalgeddon Taught Everyone About Patch Velocity

The advisory arrived October 15 with the security team’s highest severity marks. SA-CORE-2014-005, a SQL injection flaw in Drupal 7 core’s database layer. Exploitable by anonymous requests. No login required. Capable of complete site takeover.

The community patched with the discipline the platform’s security process has earned praise for in these pages, and for two weeks the story read as a serious advisory handled well.

Then came the October 29 public service announcement, and with it the sentence that moves this event from the security file to the governance file, quoted here in its full weight. Automated attacks began compromising Drupal 7 websites within hours of the advisory, and sites not patched within roughly seven hours of the announcement should proceed under the assumption that they were compromised.

Not ‘check for compromise.’ Assume it. The community has already named the event Drupalgeddon. We spent October 15 inside the patch window and the weeks since inside the cleanup conversations. This report comes in two parts. The triage for the affected, and the systemic lessons for everyone, because the seven-hour clock was never really about Drupal.

The triage, stated without comfort

For sites patched within the window: verified, documented, done. The advisory’s fix was a single-file change deployable in minutes by any prepared operation, which is precisely the point the systemic section will return to.

For sites patched later, the PSA’s guidance is uncomfortable and correct, and softening it would be malpractice. Patching after compromise seals the door behind an attacker who may have installed their own. Backdoors in code or database. New admin accounts. Scheduled tasks. Late patching is the start of response, not the end.

The honest options descend from best evidence. Restore from a backup predating October 15 and replay content changes. Or, where that is impossible, a forensic review by someone who knows Drupal’s persistence tricks, with the assumption working against you.

The diagnostic markers circulating (unexpected users, modified files, unfamiliar menu router entries) help. Their absence proves little. Competent intrusion cleans up after itself.

And the site corollary from every security event. The flagship was patched by lunch. The exposure lives in the long tail. The departmental site, the abandoned campaign install, the forgotten dev copy. That is why the first triage instrument was never a scanner. It was the inventory, for organizations that had one.

The systemic lessons, which were the real payload

• Patch speed is now measured in hours, and ‘we patch monthly’ is a compromise schedule. The seven-hour reality retires a whole class of change-management assumptions. The capability required is not exotic. Monitored advisories. Staging that can validate fast. Deployment that runs in minutes. Someone on call for exactly this. It is the practice our maintenance retainers sell, and after this month, the case argues itself.

• Security SLAs belong in contracts, in hours. Whoever maintains your sites (internal, agency, host) this event wrote a question for every agreement. What is the committed response time for a critical advisory, and was it met on October 15? Our own retainers have carried hour-denominated security clauses for years, written by exactly this category of event. Expect the industry’s paperwork to catch up this quarter.

• Backups are an incident-response asset, tested as one. The restore-from-before option above exists only for teams whose backups go deep enough, restore in practice, and were verified before the emergency. The teams that restored calmly this month had rehearsed. The teams discovering their backups’ true condition mid-incident joined our least enviable category of client story.

• The advisory process worked. The reading of it is the gap. Drupal’s security team disclosed responsibly, patched cleanly, and spoke with rare candor. The PSA’s bluntness is a service, and other ecosystems should study it. The failure surface was downstream, in the thousands of installs with nobody watching the announcements. Platform security is a relay. The last leg is always the site’s.

The communications half, which most postmortems skip

Because affected institutions face audiences as well as attackers, the communications advice. Transferred from the Heartbleed spring and sharpened by this event’s nastier shape.

The assume-compromise standard changes the disclosure math. An organization whose site sat unpatched through the window cannot honestly tell users ‘no evidence of impact’ as if absence of evidence were evidence. The trust-preserving statement is the plain accounting. What happened. What window applied to us. What we did: restored, rebuilt, rotated credentials. And what users should do, with resets sequenced after the cleanup, in the order the spring taught.

For sites holding member or donor data, bring counsel in early. Notification duties vary, and improvising them mid-incident compounds the week.

And for the majority of readers whose sites were patched in time: the two-sentence competence note has value here too, especially for institutions whose communities read the trade coverage. ‘Drupal sites compromised en masse’ headlines are circulating. ‘Ours was patched within the advisory window, here is our normal practice’ is cheap reassurance that doubles as maintenance-program marketing you did not have to invent.

Incidents communicate whether or not you do. The only choice, as the spring taught, is whether the accounting is yours.

What changed permanently

Some events adjust practice. This one adjusts the defaults. In our shop the changes are already made and this post is the notice. Security-advisory response is a named on-call duty with an hour clock, not a task in a queue. Maintenance proposals lead with the seven-hour story because it is the honest new baseline. And the retainer conversation, long treated by this industry as an upsell, has become the difference between clients who experienced October 15 as an unremarkable Wednesday and organizations now doing forensics.

For readers, the transferable summary is short. Know your site. Watch your advisories. Deploy in minutes. Keep restorable history. And write the hours into whatever agreement covers the sites that carry your name.

The next advisory with a clock attached is not hypothetical. It is scheduled by the ecosystem’s ordinary physics, destination unknown, arrival certain. Seven hours was this year’s number. The lesson is to build for it before learning next year’s. The organizations that internalize that sentence will read the next Drupalgeddon the way our patched clients read this one: as news about other people’s week.