The internet is undergoing one of its biggest transformations since the rise of search engines. Increasingly, websites aren't serving just human visitors—they're serving AI agents that search, summarize, shop, and generate answers on users' behalf.

Cloudflare believes the infrastructure supporting that shift needs to evolve just as quickly.

The connectivity cloud provider has announced a broad set of AI-focused capabilities aimed at helping publishers and website owners regain control over how artificial intelligence companies access, use, and monetize online content. The updates include new AI crawler classifications, Answer Engine Optimization (AEO) analytics, smarter AI crawling, and a revamped Pay Per Use model that compensates publishers when their content contributes to AI-generated responses.

Together, the announcements signal Cloudflare's ambition to become foundational infrastructure for what it calls the agentic internet—an emerging web where autonomous AI systems increasingly replace traditional search and browsing.

Cloudflare Wants to Redefine AI Content Access

The company's latest update builds on last year's AI Crawl Control initiative, but significantly expands its scope.

At the heart of the announcement is a new classification system that distinguishes AI bots based on their purpose rather than treating every crawler the same.

Cloudflare argues that many AI companies now separate bots used for search indexing, AI agent interactions, and model training. Others, however, continue using mixed-purpose crawlers that combine all three functions, forcing publishers into an all-or-nothing decision: remain discoverable or risk giving away valuable content without compensation.

To address that imbalance, Cloudflare plans to introduce new default policies beginning September 15, 2026.

For newly created websites—and eventually existing free-tier customers—the default configuration will allow AI search indexing while blocking model training and AI agent usage on advertising-supported pages unless publishers explicitly opt in. Websites will retain full control over those settings through the Cloudflare dashboard.

The proposal also places pressure on AI providers that still rely on mixed-use crawlers by encouraging greater transparency around how automated systems interact with publisher content.

AI Search Is Creating a New SEO Discipline

Perhaps the most notable product announcement is Cloudflare's push into Answer Engine Optimization (AEO).

As users increasingly receive information directly from AI-generated answers rather than traditional search results, businesses are beginning to ask a new question: How often does AI actually cite our content?

Cloudflare's new Attribution Business Insights dashboard aims to answer exactly that.

Instead of focusing solely on web traffic, the platform gives organizations visibility into how AI crawlers consume their content, how frequently AI systems reference their websites, and how much referral traffic individual AI platforms generate in return.

The company believes AEO will become as important in the AI era as traditional SEO became during the rise of Google.

Rather than optimizing pages solely for search rankings, marketers and publishers may increasingly optimize content for AI citations, answer quality, and visibility across large language models.

From Pay Per Crawl to Pay Per Use

Cloudflare is also expanding its vision for AI content monetization.

Last year, the company introduced Pay Per Crawl, allowing publishers to charge AI companies whenever their content was crawled.

Now it's evolving that concept into Pay Per Use, shifting compensation from data collection to actual value creation.

Under the new model, publishers are paid when their content contributes to AI-generated answers rather than simply when a crawler visits a webpage.

The initiative launches through early commercial partnerships with companies including Ceramic.ai and You.com, each implementing different payment models while relying on Cloudflare's underlying infrastructure.

For example, Ceramic.ai will compensate publishers whenever their content appears in AI search results, while You.com plans to support on-demand payments when AI agents access premium content during user interactions.

If broadly adopted, Pay Per Use could offer a more sustainable economic model than blanket licensing agreements, particularly for smaller publishers that lack the scale to negotiate direct deals with major AI vendors.

Making AI Crawling More Efficient

Another challenge Cloudflare is targeting is inefficient AI crawling.

According to the company, more than half of AI crawler traffic repeatedly downloads webpages that haven't changed, consuming bandwidth for publishers while increasing compute costs for AI providers.

Because Cloudflare sits between millions of websites and internet traffic, it can detect when content has actually been updated.

The company is testing signals that notify AI platforms only when pages have changed, reducing unnecessary crawling while ensuring AI systems receive fresher information.

Cloudflare says it is already evaluating these capabilities with several AI partners before making them more broadly available later this year.

Industry Support Signals Growing Momentum

The initiative has attracted support from both publishers and AI companies, highlighting the industry's growing interest in building more transparent relationships around AI-generated content.

Organizations including beehiiv, Ceramic.ai, Condé Nast, and Patreon publicly backed Cloudflare's efforts, particularly around giving creators greater control over AI access while preserving discoverability through search.

That support reflects broader industry tensions.

Publishers increasingly want visibility in AI-powered search experiences but also expect compensation when their journalism, research, or creative work contributes to commercial AI products.

At the same time, AI developers are seeking scalable alternatives to one-off licensing agreements that remain practical across millions of websites.

The Bigger Picture

Cloudflare's announcements represent more than another set of AI features.

They outline a potential framework for how content discovery, attribution, and monetization could function in an internet increasingly shaped by autonomous AI systems rather than traditional search engines.

If AI agents become the primary gateway to online information, today's SEO strategies may gradually evolve into AEO, while publisher revenue models shift from page views toward compensation based on AI-generated value.

Whether Cloudflare's proposed standards become widely adopted remains uncertain. Success will depend on AI companies embracing transparent crawler classifications and publishers seeing measurable returns from new monetization models.

But one thing is becoming increasingly clear: as AI changes how information is discovered, the internet's economic model is changing alongside it. Companies that can balance discoverability, transparency, and fair compensation are likely to play a defining role in what comes next.

Get in touch with our MarTech Experts