Cloudflare announced new tools that allow websites on its network to stop unrestricted AI scraping while also helping content creators identify which content is most scanned by bots, allowing them to eventually charge for access. Cloudflare is also creating a marketplace for sites to negotiate content deals based on more detailed AI audits. These tools will enable content creators to understand how AI models use their content and decide whether to allow or block access.
The rise of generative AI has complicated the value exchange for content creators. Bots no longer fall clearly into “good” or “bad” categories. AI bots, such as those used by large language models (LLMs), do not drive traffic like good bots, but are not malicious like bad bots, which creates a dilemma for site operators.
Cloudflare’s tools allow site operators to block AI bots across their entire site with a single click, or audit which areas are being scraped the most. They can then make decisions about which bots to allow, possibly striking deals with AI model providers. Cloudflare has also created model terms of use that content creators can add to their sites to legally protect against unwanted AI scraping.
Notwithstanding any of the above, blocking bots like Google’s could (and most probably will) reduce or remove sites from search engine results pages (SERPs). Cloudflare CEO Matthew Prince acknowledged that Google’s practices may need to change, but what would motivate Google to modify its practices? Regulation? Perhaps. There are some laws pending. As you can imagine, this is only the beginning.
Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.