Illustration created by Midjourney with the prompt “a cemetery filled with famous search engines –ar 3:2 –v 4”
After dinner last night, three generations of my family were sitting in the living room discussing the possibility of finding a movie we could all watch together. While this passionate debate about which movies won an Oscar and which were just fan favorites was heating up and heading nowhere, I asked ChatGPT to “list some great movies that would be appropriate for kids 12-14, parents 32-46, and grandparents.” In just under 30 seconds, it helped us find our movie. Amazing! Except… we didn’t visit any websites to accomplish our search. As someone who writes and publishes 5,000-7,500 original words each week, the ramifications of this new workflow are profound.
Large Language Models Search So You Don’t Have To
Conversational AI applications such as ChatGPT and BARD are built on large language models (LLMs), which are machine learning models that have been trained on massive amounts of text data to generate human-like responses in natural language. They are called “large” because they have a massive number of parameters, often in the range of billions, in many cases from websites found on the public internet. Said differently, LLMs search so you don’t have to.
The Inevitable Decline in Web Traffic
Web traffic will decline as people start using conversational AI applications such as ChatGPT and BARD instead of search engines. The decline in traffic will be accompanied by a decline in website conversions, and — for those who rely on the current economic model — a whole bunch of potentially bad stuff will follow.
From the perspective of search engine optimization (SEO), the decline in website traffic will negatively impact search engine rankings, as search engines like Google use website traffic as a ranking factor. This will make it harder for websites to appear at the top of search engine results pages (SERPs).
From a search engine marketing (SEM) perspective, the decline in website traffic will result in a decrease in the number of impressions and clicks for advertisements, as fewer users will be visiting websites where ads are displayed. This will lead to a decline in the return on advertising spend (ROAS) for advertisers, as they will not be getting the same level of exposure and engagement for their ads.
From a publisher’s perspective, producing high-quality content may cease to be cost-effective, as the content will struggle to generate enough web traffic from search to justify the price of production.
The Hypothetical “ai.txt” File
As anyone who has ever built a website knows, robots.txt is a file that is used to communicate with bots (also known as web crawlers or spiders) to specify which parts of a website should or should not be crawled and indexed by search engines. It is easy to imagine an “ai.txt” file added to the root directory of a website whose job it would be to communicate with bots looking for AI training material.
An “ai.txt” file wouldn’t stop all bots – robots.txt, for example, is only a suggestion; bots are not obligated to follow the instructions, and many don’t – but it would be a good start. Some kind of international regulation would be required to ensure compliance with the rights holder’s wishes.
A Great Use Case for Smart Contracts and Tokens
You may think that smart contracts, crypto, NFTs, and blockchain are over (they’re not, BTW), but tokenizing content that will be used for AI training could be the “killer app” for remunerating rights holders. This would also probably require some kind of government regulation to ensure compliance.
The Paradigm Will Absolutely Shift
Regardless of how this ultimately plays out, one thing is certain: we are on the verge of a significant paradigm shift. The economic models we have come to accept as “the way the web works” are about to be challenged in the ways I’ve outlined, but also in ways that none of us can imagine just yet. The good news is that there is so much money on the line, the “new way the web works” is going to reveal itself sooner rather than later.
Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it.