Pay Per Use and the September Deadline: The Agentic Web Gets an Economy

One year to the day after it made blocking AI crawlers the default, Cloudflare raised the stakes again. On July 1 the company announced that Pay Per Crawl is evolving into Pay Per Use. Publishers will be compensated when their content actually appears in AI answers, not just when a bot fetches a page.

Two supporting pieces came with it. First, a formal crawler taxonomy that separates bots into three classes. Search indexing, agents, and training. Second, a deadline. AI companies have until September 15 to segment their crawlers accordingly, or face default blocking across a large share of ad-supported sites.

The rules of the content economy are being written in infrastructure now, on a published schedule. Here is what changed, what September will actually test, and what to do this fall.

Why this design is better

Paying per crawl was always a crude proxy. A bot that fetches a page it never uses pays the same as one that quotes you daily. Paying per use fixes that. Compensation now follows the value delivered, which is how licensing works everywhere else in the content world.

There is a useful side effect too. Measuring which answers used which content requires exactly the visibility infrastructure publishers have been asking for. The payment plumbing and the measurement plumbing are the same pipes.

The taxonomy matters just as much. Until now, robots.txt forced a mostly all-or-nothing choice. The three-class system finally lets a site say what many actually want: yes to search indexing, yes to agents that transact, no to training. Different answers for different machines, expressed as policy.

The honest caveats: nobody knows what a fair price is yet, and the first year of this market will be a price-discovery experiment. Enforcement against crawlers that simply ignore the rules remains an arms race. And a single infrastructure company writing rules for a fifth of the web deserves the same scrutiny as any other powerful landlord.

What September 15 actually tests

The deadline is a test of a simple question: can infrastructure leverage change AI-platform behavior where lawsuits and polite conventions have not?

The compliance list on September 16 will be worth reading closely. It will show which AI companies intend to participate in a consented content economy and which prefer to be defended against. Either outcome is informative. Compliance proves the web’s infrastructure can set enforceable terms. Defiance shows exactly who the next round of tools and regulation should be aimed at.

How we got here, briefly

The path to this moment took about three years, and it is worth thirty seconds of recap.

In 2023, OpenAI published its crawler’s name and made training access a choice any site could express in robots.txt. That was the polite era: a request system that only bound the honest.

In 2025, Cloudflare flipped the default to blocking for a big slice of the web and launched Pay Per Crawl. That was enforcement arriving: requests became rules for a fifth of the internet.

Now payment follows use rather than access, the bots get sorted into classes, and there is a deadline with consequences attached. Request, then rule, then market. That is how economies form, and this one formed fast because the extraction numbers made the old arrangement impossible to defend. AI platforms crawl thousands of pages for every visitor they send back. Search never came close to that ratio.

Your homework for the fall

Start with the one assignment everyone shares. Check what your crawler configuration will say on your behalf when the defaults shift. If your site sits behind Cloudflare or a similar network, some settings may change under you on September 15. An unexamined policy is still a policy. Make yours deliberate.

Then sort by what your content is for.

• Most institutions should stay open, on purpose. If your content exists to be found and trusted, presence in AI answers is the payment. Charging for citations would be charging for advertising. Configure permissive access by class, decide the training question separately, and write down your reasoning.

• Publishers who sell content should run the experiment. Pay Per Use is aimed at your extraction problem. Set prices to learn rather than to win, because early participants will understand this market long before anyone else does.

• Anyone protecting proprietary content should verify the blocks. The taxonomy improves your vocabulary. A quarterly check that the blocks actually hold is still the difference between a policy and a comfort.

From getting crawled to setting terms

Twenty years ago, our very first post argued for making websites visible to machines. Structure your content well, we said, and the crawlers will reward you with an audience.

Two decades later the question has reversed. Machine comprehension is no longer the goal. It is the given. The real questions are what to charge, which machines to welcome, and on what terms.

Here is the satisfying part. The teams that spent years building structured, well-crafted sites did the first job properly. They are exactly the ones now holding content worth setting terms for. Go into September with your settings chosen.