The AI Rules You Can’t Read

The White House does not plan to publicly release its new framework for evaluating advanced AI models, three sources told Axios. That means the government has written the standard by which the most powerful systems in the world will be judged and shared it only with the companies invited into the room.

The framework, outlined in a June executive order, gives the government access to models up to 30 days before release and sets the terms for confidentiality, cybersecurity, insider risk, and intellectual property. It decides which models fall within scope and which “trusted partners” get early access to the most advanced systems, possibly including foreign governments (though the EU declined to comment and the U.K. didn’t respond). Tuesday’s meetings were staff-level and industry-only. Companies that weren’t invited remain in the dark, as does everyone else.

You can argue that some of this needs to be secret. The executive order explicitly classifies the benchmarking process for cyber capabilities, and you don’t publish the tests you use to probe whether a model can write a zero-day, because that is a recipe for bad actors to follow. However, it does not explain keeping the entire framework private, including the parts that decide who is trusted, who gets early access, and what a model has to clear before it ships.

We now have a voluntary framework that only a privileged few can read. What could possibly go wrong?

We spent the summer arguing about open weights and open source and whether Chinese labs would eat America’s lead. I’m not sure the answer to open models is a closed rulebook.

Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.