As of September 15, 2026, the network infrastructure provider Cloudflare will default to block "mixed" AI crawlers on web pages that use the same bots for search, model training, and agent systems. The company requires a clear division of solutions into search, training, and agent. The measure is related to the fact that publishers and site owners want to maintain visibility in search and AI services, but are not willing to provide their content for free.

Cloudflare warned that the Internet risks shifting to a "zero click" ecosystem, where chat bots collect original content for instant responses while bypassing primary sources entirely. According to the company, more than 50% of traffic from AI crawlers is due to rescanning pages, which rarely benefits, so blocking should also reduce the load on the site infrastructure. The change will affect new customers, existing users' sites and all free plan holders, with hosting owners being able to manually disable the option.

At the same time, Cloudflare is expanding its Pay Per Use commercial scheme for publishers to include not only payment for access to material but also when content is valuable for an AI service. In the first stage, the company works with Ceramic.ai and You.com. Publishers receive payments if their materials appear in a chat bot search.

Earlier, on November 18, 2025, Cloudflare faced massive technical malfunctions that disrupted many sites and applications, including Web3 services.