New Cloudflare Bot Empowers Online Publishers to Block AI Crawlers by Default

Software giant Cloudflare has unveiled a tool that keeps bot crawlers from accessing online content without authorization or compensation. According to Reuters, the new Cloudflare AI bot tool will enable websites to receive compensation from AI companies that want to access their content for training their artificial intelligence models.

The Pay Per Crawl Model

With the new tool, new web domains that sign up on Cloudflare will have the option of allowing AI crawlers to access their content. Domain owners who choose to give AI crawlers access will be required to set a price for the access using a ‘pay per crawl’ model.

This model allows website owners to leverage the Cloudflare AI crawlers monetization tool to control how AI firms use their content and compensate them.

“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” Cloudflare CEO and Co-founder Matthew Prince said in a statement.

Cloudflare’s AI bot access charge has been received positively by leading publishers like Associated Press and Condé Nast. Social companies like Pinterest and Reddit have also appreciated the new tools. The internet company said that its goal is to create a sustainable ecosystem where digital content creators and AI firms can thrive.

“The change in traffic patterns has been rapid, and something needed to change. This is just the beginning of a new model for the internet,” Cloudflare’s Chief Strategy Officer Stephanie Cohen said.

Taking Over Online Search

Concerns have been raised over AI crawlers taking over online search. As automated bots, AI crawlers extract huge amounts of data from databases, websites, and other online sources. The data is used to train large language models like OpenAI’s ChatGPT and Google’s Gemini. Previously, online content creators were rewarded by pushing users to the original information sources.

With AI crawlers, this is no longer the case. AI models collect articles, text, and images and generate responses in a way that users don’t have to visit the main source. Cloudflare says that this deprives online publishers of the traffic they need to generate revenue from advertising.

“AI crawlers are typically seen as more invasive and selective when it comes to the data they consume. They have been accused of overwhelming websites and significantly impacting user experience,” Mathew Holman, a Partner at UK Law Firm Cripps said during an interview with CNBC.

Data from Cloudflare shows that Google, for instance, has recorded a drop in the crawls to visitors referred to original sites ratio. This ratio has dropped from 6:1 to 18:1 over the last six months. This means that the search company has been reducing referrals while maintaining its crawling. The decline in referral to original information sources may also be as a result of users getting responses directly from Google’s AI Overview.

Future Impact

Cloudflare works as a content delivery network that enables enterprises to deliver digital applications and content to end users faster. The company does this by catching the data. It also plays an important role in enabling internet users to access digital content seamlessly.

In its 2023 report, the software firm estimated that about 16% of internet traffic globally passes through its content delivery network. Cloudflare’s latest AI traffic regulation tool builds on the one it unveiled last year that enabled publishers to block AI bots with a single click. With the new tool, Cloudflare makes this feature a default on all websites that it serves. If it works well, it could change the way AI bots work in the long run.

“If effective, the development would hinder AI chatbots’ ability to harvest data for training and search purposes. This is likely to lead to a short term impact on AI model training and could, over the long term, affect the viability of models,” Holman added.