The Digg Effect, Memcached, and Surviving Your Best Day

There is a specific phone call every agency dreads and eventually gets. The client’s press hit finally landed: the morning show, the front page of Digg, the wire pickup. The traffic arrived exactly as everyone hoped. The site collapsed under it.

The one day of the year the organization was most visible became the day its website served error pages, and everyone on the call is asking how this happens. So let’s answer properly: how it happens, and the layered, mostly cheap defenses that prevent it.

Why dynamic sites fall over

A database-driven CMS builds each page fresh, on request. Dozens of queries fetch the content, the navigation, the sidebar blocks. PHP assembles the result, and the visitor sees a page. At Tuesday traffic, a few thousand visits spread across a day, this is fine.

But the work per page is real, and it multiplies with visitors. When Digg’s front page or a broadcast mention sends ten thousand people in ten minutes, the math goes vertical. The database that comfortably ran fifty queries a second is asked for five thousand. Connections saturate, queries pile up, and pages that took a quarter-second take thirty. Somewhere in that spiral the whole stack tips over.

Nothing ‘broke,’ exactly. The site did precisely what it was built to do, fresh page every request, at a volume nobody priced.

The insight that reframes everything: during a spike, nearly everyone is anonymous, and nearly everyone is looking at the same few pages. Ten thousand strangers reading one article do not need ten thousand freshly assembled copies. They need one copy, made once, served ten thousand times. That is caching, and it comes in layers.

The caching ladder, bottom to top

• Opcode caching: the free floor. PHP normally recompiles your application’s code on every single request, which is as wasteful as it sounds. An opcode cache like APC keeps the compiled version in memory. Installation is an afternoon, and the payoff is a real speed gain on every page. No production PHP site should run without one. If your host does not offer it, that is information about your host.

• Object caching: sparing the database. Here is where memcached enters. LiveJournal built it to survive its own growth. It is now load-bearing at Facebook, Wikipedia, and most large PHP operations you can name. Memcached is a fast in-memory store where the application parks the answers to expensive questions: this article’s rendered body, that sidebar’s block, the navigation tree. The next request checks the cache before asking the database. During a spike, the cache answers nearly everything while the database sits at a fraction of capacity. Both Drupal and WordPress can integrate it with modest configuration.

• Full-page caching: the spike killer. The top of the ladder. For anonymous visitors, cache the entire finished page and serve it like a static file. The database goes untouched and the application is barely woken. This is the layer that turns ‘ten thousand visitors in ten minutes’ from a crisis into a Tuesday. Drupal ships page caching for anonymous users in core. Switch it on. Startlingly often, we find it off. WordPress gets there via the caching plugins that have matured this year. The cost is a freshness tradeoff, since cached pages update on an interval rather than instantly. For a public site during a traffic event, five-minute-old pages served reliably beat live pages served never.

• Expires headers: the courtesy layer. Tell browsers your logo, stylesheets, and scripts will not change this hour, and repeat visitors stop re-downloading them. Configuration, not code. Small individually, meaningful at volume.

What the ladder looks like installed

Numbers make this real, so a composite from recent engagements. The shapes are typical and the identities are nobody’s. A nonprofit’s Drupal site on a decent single server: the baseline load test sustains roughly forty requests per second before response times go ugly, with the database pegged and every page assembled fresh. The cheapest server is the query you never run.

Afternoon one: APC installed, page caching enabled for anonymous visitors, CSS aggregation on. Same test: several hundred requests per second, database loafing. That is the difference between ‘the radio mention takes us down’ and ‘the radio mention is a good day.’

Afternoon two: memcached wired in for the logged-in sections, expires headers set, and a follow-up test to find the new bottleneck. There is always a new bottleneck. The point of the exercise is choosing which one you can live with.

Total invoice: a small consulting engagement. Total hardware purchased: none. We keep meeting organizations shopping for bigger servers to solve problems that are actually configuration debt. Buy the audit before the iron.

One rung above all this, for the roadmap file: dedicated reverse-proxy caches. Squid deployed in front of the web server is the established pattern at newspaper scale. It takes the full-page principle out of the application entirely, serving cached pages before PHP even wakes. Most institutional sites do not need that layer yet. Knowing it exists is enough for now.

The knobs and the gotchas

Honest complications, because caching has them. Logged-in users bypass full-page caches by design, since their pages truly differ. Sites with heavy authenticated use lean harder on the object layer.

Cache invalidation, meaning making edits appear promptly without giving up caching’s benefits, is a famously hard problem in computer science and a manageable one in practice. Sensible timeouts plus targeted clearing on publish covers institutional sites fine.

And test with the cache on. We have watched teams performance-test their way to false despair against uncached development servers.

Capacity is a marketing line item

The framing we want decision-makers to leave with: if a campaign is designed to attract attention, the capacity to survive that attention is part of the campaign’s budget, as much as the ad buy or the press outreach. The engineering is not exotic. Everything above is standard practice, configurable in days, cheap against a single minute of downtime during your moment.

What is usually missing is the conversation. Marketing plans the spike. Operations discovers it live. So have the conversation this quarter, before your best day schedules itself. What is our realistic peak? Is the ladder in place: opcode, object, page, headers? Have we load-tested against the peak, with caches on?

The organizations that can answer yes get to experience their press hit as a triumph. The others get the phone call. We know which conversation we would rather be in. We have now been in both.

A postscript for the skeptic who says ‘we’re not big enough to worry about this.’ Spikes are not a big-site phenomenon. They are a relative phenomenon, and small sites have the furthest to fall. The regional nonprofit mentioned on a national program takes proportionally more incoming fire than a newspaper does, on a fraction of the infrastructure. The ladder costs the same modest amount either way. Size determines when your spike comes, not whether.