The Machines Are Eating Your Capacity: AI Crawlers as an Ops Problem

The marketing coverage of the AI era has, understandably, followed the visibility story. The citations, the referrals, the zero-click math we published this year. In doing so it has largely missed what the hosting providers, the CDN radar reports, and our own capacity dashboards have documented all year. The infrastructure story.

AI crawler and fetch traffic has become a first-order capacity concern in 2025. The published network analyses agree on the shape. Bot shares of total requests climbing steeply. Double-digit percentages of origin load on some hosts. Fetch patterns that bust caches by design, the answer-time pulls hitting long-tail URLs no human crowd ever warmed. Crawl-delay conventions ignored by a meaningful share of the crawler population. And bandwidth bills inflating to serve machines that may never send a visitor. The crawl-to-value collapse we covered in July, arriving as a line item.

We publish this in November for the same reason our capacity coverage has run in November since the 2008 Beijing pieces. Peak-season planning now requires knowing how much of your headroom the bots are consuming. A load test that ignores a double-digit ambient consumer is a load test of a system that does not exist. The audit, the defense ladder, and the cache architecture. The ops treatment, in the series’ standing format.

The traffic-composition audit, first and separately

The first move, before any policy. Separate the machine traffic into its four classes, because each deserves different policy, and lumping them is how both over-blocking and over-serving happen.

The classes, assembled from the year’s coverage. Training crawlers: the GPTBot tier. Bulk corpus collection, governed by your access policy per the 2023-and-July decisions, throttleable without visibility cost, if permitted at all. Search indexers: the Googlebot tier. The visibility plumbing, served properly, never blocked by accident. The audit’s first job is making sure your defenses can tell them apart. Answer-time fetchers: the citation-era pulls. Content fetched at question time to ground an answer, carrying your AI-surface presence, latency-sensitive and cache-hostile. And agent sessions: the machine customers we covered last month. Visitors that buy, carry revenue, and deserve good service.

The audit mechanics. The log analysis by user-agent and verified IP ranges. The major crawlers publish theirs, and verification matters because impersonation is rampant in the unnamed tier. The CDN’s bot-analytics tier, now standard equipment. And the origin-versus-edge share per class, which is the number the capacity model actually needs.

Many teams that run this census are surprised by their own mix, in one direction or the other, and the surprise is useful. Policy set against an unmeasured population is guesswork, not policy.

The defense ladder, and the cache architecture

• Rung one, the declared tier: robots directives per crawler class (the access policies from July, expressed technically). Crawl-delay where honored. The llms.txt-class conventions where adopted. Cheap, legible, and binding only on the honest, which is why the ladder continues.

• Rung two, the enforcement tier: WAF rules and rate limits by verified class. The throttle for permitted-but-greedy training crawls. The block for the defiant tier. And the commercial crawler-management tooling the CDNs now ship as standard equipment, the managed bot rules, the AI-crawler toggles. Set on purpose, per the no-neutrality finding, because the defaults now matter in both directions.

• Rung three, cache architecture against fetch patterns: answer-time pulls are the new cache-miss generator. The long-tail, freshness-sensitive requests that bypass every human-warmed cache. The responses are the ladder’s classic moves re-aimed. Origin shielding, the CDN’s collapse of duplicate origin fetches, earning its keep against fetch storms. Longer edge TTLs on the stable informational tier the fetchers favor, with purge-on-update doing the freshness work. And static rendering for the citation-bearing content whose machine consumption is now a serving requirement.

• The bill, read quarterly: bandwidth and origin-compute by traffic class joins the standing cost review. Serving the machine economy is now a priced decision, and the price belongs in the same budget line as the policy, as we said in July.

The peak-season math, which is why this runs now

The November point, made with the series’ standing math. Your peak-capacity model (the load tests, the headroom targets, the degradation ladders) was built against human demand curves. The machine traffic changes the model’s floor. The bots do not celebrate holidays. Their load is ambient rather than event-shaped. A system sized for peak human traffic plus zero percent machine overhead is under-sized by exactly your measured machine share.

The adjustments, concretely. The load tests re-baselined with ambient bot load included. The test population plus the measured machine floor, not instead of it. The degradation ladder extended with a machine tier. Under duress, the answer-time fetchers and training crawlers get throttled before any human-facing feature dims. The priority order is obvious once stated and absent from every pre-2025 runbook we have reviewed. And the incident checks updated. The traffic-spike triage now asks ‘human or machine?’ first, because the answer changes everything about the response.

The sites that ran this audit before their peak are entering the season with honest headroom numbers. The ones that have not are, per the rehearsal files’ oldest line, planning with a number that is not real.

The method, unchanged

Our high-traffic coverage opened with an Olympics and is now in its eighteenth year. Its constant has been that capacity is a knowable and unglamorous discipline. The tools are the census, the test, the ladder, and the drill. This year adds a new class of visitor without changing the method. Traffic that isn’t people is still traffic. It gets measured, classified, and priced honestly. Served or declined deliberately, and planned for like every load we have ever modeled.

The loop closing here deserves a sentence. Crawl policy is now three things at once. An infrastructure call, covered here. A content-strategy call, covered in the visibility pieces. And a line item, per July’s economics. The November audit belongs jointly to engineering, marketing, and finance. The same three chairs every mature operational question eventually pulls up.

Capacity planning’s newest crowd doesn’t celebrate holidays, doesn’t read your promos, and doesn’t bounce. It just arrives, at all hours, reading everything. The sites that know exactly how much of their headroom it holds are the ones whose human visitors will never notice it was there. Measure it this month. The season, as ever, is coming either way.