Pay Per Crawl. What does it really mean?

Pay per crawl

The rise of AI has intensified a long-standing tension on the internet. For years, web crawlers, primarily those operated by search engines, have indexed vast swathes of the internet, operating under an implicit agreement that this access would drive traffic back to content creators through organic search results and click-throughs.

However it has become clear that the advent of sophisticated AI models, particularly large language models (LLMs), has fundamentally disrupted this SEO ecosystem.

These AI systems consume immense volumes of data to learn and generate new content, often without providing direct attribution or sending referral traffic to the original sources.

These AI systems consume immense volumes of data to learn and generate new content, often without providing direct attribution or sending referral traffic to the original sources.

The zero-click problem

Zero-click searches have plagued content creators for years as users increasingly find answers directly in search results without visiting the source websites. However, Google’s AI Overviews, Bing’s AI-powered responses, and other AI-driven search features have dramatically exacerbated this issue by synthesising information from multiple sources into comprehensive answers at the top of search results pages.

Unlike traditional featured snippets that at least provided source attribution and some click-through potential, these AI-generated responses often eliminate the need for users to visit original content entirely, creating an even more severe traffic drain for publishers.

This shift represents a fundamental challenge to the traditional SEO model. Content creators who have invested heavily in traditional SEO strategies now find their content being used to train AI models and generate AI responses that compete directly with their own pages for user attention.

The result is a declining organic click-through rate, reduced website traffic, and ultimately, diminished revenue for publishers and content creators.

Pay Per Crawl

This has sparked a global debate, with publishers and creators arguing that AI companies are “stealing” their intellectual property without compensation, effectively devaluing original content.

In response, infrastructure providers are developing tools to reassert control. Cloudflare’s “Pay Per Crawl” is one such initiative, designed to enable content owners to charge AI crawlers for access. This system leverages the HTTP 402 “Payment Required” status code, allowing publishers to set a price for their content.

The core idea is to create a new revenue stream and establish a more equitable exchange, transforming open access into a monetised transaction for AI bots.

Pay Per Crawl represents an intriguing attempt to address the problem. In theory, it’s a way to charge AI bots for scraping your content and claw back some of the value they’ve stolen.

Instead of the binary choice between blocking crawlers or allowing free access, publishers can now set a price for AI systems to access their content, creating a potential revenue stream from what has until now been a one-way extraction of value.

However, while the principle is appealing, Pay Per Crawl only really works if you have significant leverage. The emerging landscape reveals a stark divide between large publishers and smaller content creators, with fundamentally different options available to each when it comes to implementing such pricing models.

Why Pay Per Crawl will benefit major publishers

Large media organisations like The New York Times, The Washington Post, and BuzzFeed are perfectly positioned to take advantage of Cloudflare’s Pay Per Crawl system. These publishers possess several crucial advantages that make pricing their content viable.

Major publishers have established reputations and legal teams capable of pursuing copyright claims and negotiating complex licensing agreements. When The New York Times sued OpenAI and Microsoft in late 2023, it demonstrated the kind of legal firepower that can force AI companies to the negotiating table.

Perhaps most importantly, major publishers have the credibility to set meaningful prices through Pay Per Crawl without fear of invisibility. Their content is so valuable and widely referenced that AI companies cannot afford to lose access entirely, making them likely to pay reasonable per-crawl fees.

These organisations produce (on the whole..!) unique, high-quality journalism and analysis that AI companies genuinely need for training comprehensive models. Their content represents authoritative sources that are difficult to replace with alternatives.

Large publishers generate massive amounts of content daily, providing ongoing value to AI training datasets. This consistent output makes them attractive long-term partners for AI companies seeking reliable content streams.

And why smaller publishers will feel the squeeze

For smaller publishers, independent bloggers, and niche content creators, Cloudflare’s Pay Per Crawl system presents a fundamental problem: setting a price would be basically the same as blocking crawlers entirely. AI companies simply won’t pay. Several factors make this inevitable.

AI companies may need to create budgets for content licensing, but they will naturally prioritise deals with major publishers who offer the most content for their investment. A small blog demanding payment for access is easily ignored when similar content is available elsewhere for free.

The fact is, smaller publishers often cover topics where alternative sources exist. An AI company can find similar content from multiple sources, making any individual small publisher dispensable.

Smaller publishers rely heavily on search engine traffic for discovery. Blocking AI crawlers could eventually mean blocking search engines entirely, as systems become increasingly integrated. This creates a death spiral where reduced visibility leads to reduced traffic and revenue.

This effectively takes smaller publishers back to a binary choice: give content away for free or become invisible.

Evidence of this dynamic is already visible in website analytics, with many reports of significant drops in organic search traffic following the rollout of AI Overviews, with their content being used to generate AI responses that keep users on search result pages rather than driving clicks to their sites.

The evolution of SEO

While the challenges facing content creators in an AI-dominated landscape are significant, it’s important to recognise that SEO isn’t dead – it’s evolving. The fundamental principles of creating valuable, discoverable content remain crucial, but the strategies and metrics are shifting to accommodate new realities.

Traditional metrics like search rankings and organic traffic, while still important, are becoming part of a broader ecosystem that includes AI visibility, brand recognition, and direct audience engagement.

Smart content creators are already adapting by developing AI-resistant content formats such as personalised experiences, and community-driven content that AI cannot easily replicate or replace.

The future of SEO lies in this diversification and innovation.

I predict that content creators who can adapt to this new paradigm will not only survive but thrive in an AI-first world. The key is recognising that the rules have changed.

Get your growth plan

Ready to chat? Whether you’ve got a project in mind or just want to ask a few questions, drop us a message and we’ll get straight back to you.

No sales fluff, no pressure – just honest advice from people who know their stuff.

    Call us maybe? Get in touch with Jelly