AI Scraping

Yesterday, a coalition of major publishers (including Reddit, Yahoo, Medium, and Quora) announced their support for Really Simple Licensing (RSL), a new open standard that lets web publishers set machine-readable licensing terms for AI crawlers. The initiative, led by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, aims to solve one of the web's most pressing problems: how to get AI companies to pay for the content they scrape. Continue Reading →

The rules of AI training have just changed. By default, Cloudflare will now block AI scrapers across the millions of websites it protects (roughly 24% of all sites on the internet). Any AI company seeking to crawl a Cloudflare-hosted site will have to obtain explicit permission from the content owner. This is the first infrastructure-level defense of its kind. Continue Reading →