Shelly Palmer

AI Web-Scraping Gets an “Official” Standard

RSL (Really Simple Licensing) was introduced in September 2025 as a way for publishers to express licensing terms for AI scraping and training. Now, with the 1.0 release finalized, the industry has a stable, “official” version to adopt. It offers a machine readable method for publishers to state how their content can be used and the conditions under which payment or contribution is required.

RSL is “simple” to use. A publisher posts a small file (rsl.txt) that functions similarly to a robots.txt file. Instead of only allowing or blocking access, RSL lets a publisher define licensing terms, usage boundaries, and compensation expectations. The RSL Collective released the specification with support from Cloudflare, Akamai, Fastly, Creative Commons, and a global group of publishers and rights holders.

For content owners, this is progress. It provides a uniform way to declare rights in a world where AI systems rely on scaled access to text, images, audio, and code. It also creates a signaling layer for policymakers who are trying to balance copyright, fair use, innovation, and compensation. RSL puts publisher terms in a consistent format and creates a clear place for the industry to look.

The challenge is compliance. None of the major foundational model developers have agreed to honor RSL 1.0. OpenAI, Google DeepMind, Anthropic, Meta, xAI, Mistral, and Amazon have made no public commitments to read or respect the file. RSL carries no legal force. It only works if AI companies adopt it or if infrastructure providers enforce it. For now, the signal exists, but the receivers remain silent.

Infrastructure support may change the dynamic. Cloudflare, Akamai, and Fastly can enforce RSL rules at the network level if publishers choose to configure their systems that way. Enforcement may also come through new legal frameworks. Courts are deciding whether training on publicly accessible material qualifies as fair use. Legislatures are examining what a licensing economy for AI training might require. RSL provides a structured vocabulary for those debates.

All of this assumes that large language models will continue to need vast amounts of human created data for training. That assumption is under active debate. Researchers like Yann LeCun argue that next generation systems will rely less on text corpora and more on world models and world simulation architectures that learn from perception, interaction, and self-supervised prediction of the physical environment.

For AI companies, RSL creates pressure. Their models depend on large, diverse datasets. They prefer broad access to public content. They do not want to accept a standard that implies payment obligations at scale. Any signal that introduces metering or licensing terms complicates the economics of their training pipelines. The industry needs clarity about how training rights work, and it also needs adequate access to content. Those objectives are difficult to reconcile.

For publishers, RSL offers a constructive tool. It sets expectations and provides a starting point for negotiation (as opposed to litigation). It does not ensure compliance, but it changes the context of the discussion.

Standards shape ecosystems. Robots.txt influenced how search engines behave. RSS established the mechanics for content feeds. Creative Commons licenses reshaped how culture is shared. RSL may do the same for AI training. The industry is not there yet, but the framework now exists.

Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.