Who Supervises Self-Building AI?

Anthropic says AI may soon accelerate its own development faster than the company can keep its safeguards current.

In its November 2 SEC filing, Anthropic says internal estimates and expectations for future models suggest it may imminently reach its Responsible Scaling Policy threshold for “dramatically accelerated progress attributable to AI.” So far, it has not designated any current model as exceeding that threshold.

Claude already helps Anthropic run experiments, prepare training data, evaluate models, and write code. Each generation helps build the next. This is recursive self-improvement (RSI), a loop where AI systems help design, train, code, or optimize the next version of themselves, with the potential to lead to an “intelligence explosion,” a hypothetical scenario where AI builds AI at exponentially increasing speed.

The filing gives no date for fully autonomous model development. Anthropic says it does not expect to safely let Claude design and build successive frontier models without meaningful human supervision or understanding. It is considering policy changes to keep humans in control.

I think Anthropic needs a better definition of “meaningful.” A person can approve a decision they barely understand. A review can satisfy a procedure while missing the failure that matters.

As you know, the last machine humans will ever build is a machine that builds other machines. It follows, then, that the last AI model we will ever need to build is an AI model that builds better AI models.

Taken to its logical conclusion: we end up with AI that is so superhuman that we will not understand how it works. A good thought experiment is to imagine that Fable 5.1 has an IQ of 160. We don’t have any trouble working with it, because it’s basically just a narrow-focused genius-level polymath. But what if Fable 10 launches in a year or so with an IQ of 1,600? No one has ever encountered that level of intelligence before. It would identify and solve problems we can’t even identify, would work at speeds we can’t comprehend, and would do so with very little need for our input.

Science fiction, you say? No, just a matter of time, chips, and energy.

Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.