Which Game Are We Playing?

The race to AGI is speeding up. Governments are doing their best to control and regulate the tech. The anti-AI movement is growing. The pro-AI movement is growing. The calls to slow down or stop AI innovation are growing. We’re in a FOMO-driven race that feels all too familiar. It should. It’s not new. I’ve been doing a lot of reading these past few weeks trying to survey and understand the body of research about this and to explore historical precedents. We’ve been here many times before. Let’s explore.

Everyone Sees the Danger

In May 2023, the chief executives of OpenAI, Anthropic, and Google DeepMind signed a one-sentence public statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Governments have noticed, the EU’s AI Act entered into force in August 2024. The public has noticed too. A Pew Research Center survey conducted this June found that 52% of US adults are more concerned than excited about AI in daily life, and for the first time a majority of adults under 30 (55%) feel the same way.

The calls to slow down are more than three years old. In March 2023, the Future of Life Institute published an open letter asking every AI lab to pause training of systems “more powerful than GPT-4” for at least six months, more than 30,000 people signed it, including Elon Musk and Steve Wozniak. Nobody paused. Last week, Nvidia agreed to backstop as much as $105 billion for a new OpenAI data center campus in Pike County, Ohio, part of a buildout that will deliver roughly eight gigawatts of computing capacity.

The builders say the technology is dangerous, the regulators agree, the public agrees, and the money is accelerating. Everyone keeps going because they are certain that everyone else will keep going. If American labs slow down, Chinese labs speed up. If one lab pauses, its rivals take the market and set the standards.

Four Disciplines, One Trap

This situation has several names because at least four academic disciplines discovered it independently and wrote volumes about it. The modern umbrella term is a multipolar trap, coined in Scott Alexander’s 2014 essay “Meditations on Moloch,” a competition in which every participant is forced to sacrifice a value they all share, because any participant who unilaterally holds back loses to those who don’t.

Economists call the underlying structure a collective action problem. Ecologists know it as the tragedy of the commons, from Garrett Hardin’s 1968 essay in Science. Political scientists call the international version the security dilemma, a term John Herz coined in 1950 and Robert Jervis developed in “Cooperation Under the Security Dilemma” (World Politics, 1978), one nation’s defensive buildup looks like a threat to its rivals, who build in response, leaving everyone less secure.

The AI research community has its own literature on the AI-specific case. The foundational paper is “Racing to the Precipice” by Stuart Armstrong, Nick Bostrom, and Carl Shulman (AI & Society, 2016), which modeled competing AI development teams and reached the uncomfortable conclusion that the more competitors in the race, and the more they know about each other’s capabilities, the less each one invests in safety. Three years later, researchers at OpenAI published “The Role of Cooperation in Responsible AI Development,” an analysis of their own industry’s collective action problem.

Prisoner’s Dilemma or Stag Hunt

Game theorists model this situation two different ways, and the two models call for opposite remedies. The first can be modeled as a classic prisoner’s dilemma where racing is the winning move no matter what your rivals do. If you knew for certain that every competitor had stopped, your best move would still be to accelerate, because now you win everything. If AI development is a prisoner’s dilemma, trust is irrelevant, voluntary restraint is impossible, and the only fix is an outside referee with real penalties that change the payoff.

A second way to model this comes from a story in Rousseau’s Discourse on Inequality (1755). A party of hunters can work together to take a stag and feed everyone well, or each hunter can break off and catch a rabbit alone, a small meal but a sure one.

Every hunter prefers the stag. But if you suspect the hunter next to you is about to bolt for his rabbit, you bolt for yours first, and the stag escapes. Nobody wanted rabbit. They ate rabbit because they couldn’t see each other’s intentions. Game theorists call this the stag hunt or the assurance game, and Brian Skyrms wrote the modern treatment, The Stag Hunt and the Evolution of Social Structure (Cambridge, 2004). In a stag hunt, everyone genuinely prefers mutual restraint and defects only out of fear that the others won’t hold. Fear is fixable. You don’t need to change anyone’s incentives, you need to give everyone proof of what everyone else is doing.

The Nuclear Precedent

We’ve been here before. The US and the Soviet Union both understood that nuclear weapons were dangerous, and both kept building them for decades, each certain the other would never stop. Thomas Schelling’s The Strategy of Conflict (1960) reframed the problem: the obstacle was verification, and the breakthrough was technical. SALT I (1972) explicitly protected each side’s “national technical means of verification,” the diplomatic phrase for spy satellites, and START (1991) added on-site inspections. Once each side could count the other’s silos, “I’ll stop if you stop” became an enforceable deal, and arsenals shrank for thirty years. The hunters could finally see each other, so they could finally hunt the stag.

The Missing Telescope

Within a country, the game is solvable by ordinary law, a government can regulate its own labs the way it regulates its own banks and drug makers, no telescope required. Between countries there is no referee, which is why the international layer binds the domestic one. The standard argument against slowing down in Washington ends the same way, China won’t stop. If being first to AGI is a genuine winner-take-all prize, it is a prisoner’s dilemma, and nothing short of binding, enforceable international rules will change behavior. If the nations are racing mainly out of fear of each other, it is a stag hunt, and the missing piece is the telescope: some way to verify what rivals are actually doing.

There is also a third reading, albeit a cynical one. What looks like a trap is really just the market doing what markets do. The signatories of the extinction statement have been outspending each other ever since they signed it. The verification optimists point out that frontier AI has a physical signature. Training runs require enormous, concentrated clusters of chips that draw gigawatts of power, and chips, like missile silos, are countable and hard to hide.

Something to Contemplate

Which game do you believe we are in? Do you take the extinction statement at face value, or as the safest thing an accelerating company can say? What evidence would convince you that a rival lab, or a rival nation, had actually slowed down? Who would you trust to hold the telescope? Before you answer for the nations, answer for yourself; the same trap operates at every scale. Everyone believes their competitors are speeding up, so everyone speeds up. This may be our fate. But Rousseau’s hunters ate rabbit because they couldn’t see each other. The stag is still out there.

Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.