Yesterday, a coalition of major publishers (including Reddit, Yahoo, Medium, and Quora) announced their support for Really Simple Licensing (RSL), a new open standard that lets web publishers set machine-readable licensing terms for AI crawlers. The initiative, led by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, aims to solve one of the web’s most pressing problems: how to get AI companies to pay for the content they scrape.
The Promise and the Problem
RSL builds on the familiar robots.txt protocol, but instead of just saying yes or no to crawlers, publishers can now embed licensing terms directly into their files. Want to charge per crawl? Set a subscription fee? Demand payment every time an AI model references your content in a response? RSL supports all these models.
The technical implementation looks straightforward. Publishers add XML-based licensing terms to their robots.txt files. AI crawlers read these terms and, theoretically, comply with them. The RSL Collective, a nonprofit rights organization modeled after music licensing groups like ASCAP, handles the negotiations and royalty distributions.
Major brands have already signed on, including Ziff Davis, O’Reilly Media, wikiHow, Internet Brands, and The Daily Beast. Even Reddit, which already receives an estimated $60 million annually from Google for training data access, has joined the collective.
The Enforcement Challenge
Unfortunately, RSL by itself cannot block bots from visiting websites. The standard relies entirely on AI companies voluntarily respecting the licensing terms they encounter. Given that AI model builders have repeatedly been accused of ignoring robots.txt files, expecting them to suddenly honor payment requirements seems optimistic.
The RSL Collective has partnered with Fastly to provide enforcement capabilities. Fastly acts as “the bouncer at the door,” checking whether AI crawlers have agreed to license content before granting access. However, this only works for publishers using Fastly’s content delivery network. Everyone else can ask for payment but has no mechanism to enforce compliance.
The Cloudflare Comparison
Cloudflare took a different approach. Since July 2025, the company blocks AI crawlers by default for all new domains. More than one million customers have enabled AI crawler blocking since the feature launched. Cloudflare’s Pay Per Crawl system, currently in private beta, uses HTTP 402 “Payment Required” responses to create an actual transaction mechanism. Publishers set prices, AI companies see those prices, and they either pay or walk away.
When Cloudflare blocks a bot, that bot stays blocked (they own the infrastructure). The company handles trillions of requests daily across 20% of the web, giving it both the technical capability and market leverage to enforce its rules.
RSL plus Fastly approximates what Cloudflare offers natively, but with an important caveat: RSL’s system works only if AI companies choose to participate. Cloudflare’s system works because the company controls the pipes.
Will It Work?
The RSL Collective’s success hinges entirely on achieving critical mass. I applaud the effort. The web desperately needs a scalable solution for compensating creators whose work trains AI models. RSL provides the framework for that solution. Will it achieve the scale it needs for success? That’s up to the foundational model builders.
Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.