In context: Cloudflare CEO Matthew Prince recently reiterated his warning that generative AI crawlers and summaries threaten the foundations of the internet’s business model. To protect publishers from a flood of artificial AI traffic that offers virtually no authentic site visits in return, the company is devising methods to combat AI scrapers.
Speaking at an Axios event in Cannes last week, Prince explained that search engines and chatbots using generative AI to summarize web content have significantly reduced the number of human visitors to many websites. Even compared to six months ago, the problem has worsened considerably.
Traditionally, for every six times Google crawled a website, one person might visit and potentially view ads. In contrast, the rate was about 250 to 1 with OpenAI’s crawlers and 6,000 to 1 with Anthropic. Today, Cloudflare’s CEO estimates that Google’s crawl-to-visitor rate has declined to 18 to 1, OpenAI’s has worsened to 1,500 to 1, and Anthropic’s is approximately 60,000 to 1.
This decline is likely because chatbots and search engines now save users the effort of visiting websites. Chatbots can retrieve and summarize information without the user needing to leave the chat interface, and AI overviews from major search engines now offer users answers before they click on search results.
When Google first unveiled AI overviews, it claimed that the technology would boost traffic to the original sources of the summarized content. Similarly, large language models such as ChatGPT have recently started citing sources in their responses to help direct traffic back to content creators.
Must read: The Zero Click Internet
However, Prince claims that users, by and large, aren’t clicking on the footnotes. Instead, many many are accepting the AI’s responses at face value, as trust in the technology has grown over the past six months. Aside from starving websites of traffic and revenue, the trend is potentially dangerous due to AI’s known tendency to generate inaccurate or misleading information.
In response, Cloudflare, which offers cybersecurity solutions for websites, has launched a new tool called the AI Labyrinth. This tool is designed to use generative AI against the crawlers themselves.
Although websites can include instructions to block AI crawlers, many bots either bypass or ignore these directives. When the AI Labyrinth detects such behavior, it leads the bot through a maze of AI-generated links that no human would reasonably follow, causing the bot to waste time and computing resources.
Despite how daunting it might seem to oppose AI giants like Google, Microsoft, and OpenAI, Prince emphasizes that Cloudflare has a strong track record of successfully defending its clients, even against attacks from powerful national governments.