The AI hype machine has fully lost its mind. There are about fifty articles from reputable news organizations and pundit sites proclaiming that OpenAI just announced AGI (artificial general intelligence). It didn’t. OpenAI has introduced “o3” and “o3-mini,” the latest in its reasoning model family, claiming significant advancements in AI’s ability to tackle complex tasks. So, let’s all calm down and take a minute to examine the specifics of OpenAI’s announcement and gain a clear understanding of o3’s actual capabilities and their implications.
First, there is no agreed-upon definition of AGI. Some say, “AGI is an AI system capable of performing any intellectual task that a human can, exhibiting general cognitive abilities across a wide range of domains.” OpenAI defines it as, “highly autonomous systems that outperform humans at most economically valuable work.” Calling something AGI depends upon where you set the bar.
What Does A Win Look Like?
Using the broader definition, if AGI were achieved, it could solve complex problems, automate intellectual tasks, and transform industries by performing any mental task a human can. It might revolutionize healthcare with personalized treatments, education by tailoring learning to individuals, and scientific discovery through faster analysis and innovation. AGI could enhance human creativity, optimize productivity, and even explore outer space or the deep sea autonomously. While it holds the promise to eliminate scarcity and improve global living standards, it also raises ethical concerns, such as job displacement, misuse, and the need for careful oversight to ensure it aligns with human values and priorities.
Importantly, AGI is the North Star of the foundational model builders and the hyperscalers. With this in mind, let’s explore what o3 is supposed to be able to do.
A Model Family with Adjusted Reasoning
According to OpenAI, o3 and o3-mini are designed to handle complex reasoning tasks by simulating a “private chain of thought.” Unlike traditional AI models, reasoning models like these fact-check themselves during the problem-solving process, pausing to evaluate potential solutions before responding. This method makes them slower than non-reasoning models but more reliable in domains like physics, coding, and advanced mathematics.
o3 introduces adjustable reasoning time—low, medium, and high compute levels—allowing users to tailor performance to specific tasks. Higher compute settings deliver better results but at a higher computational cost.
Performance Highlights
OpenAI is grading its own work here, so none of these test scores have been verified by third party researchers. That said, OpenAI asserts that o3’s benchmark scores show remarkable progress:
- Coding: Achieved a Codeforces rating of 2727, placing it in the 99.2nd percentile, and outperformed O1 by 22.8 points on SWE-Bench Verified.
- Mathematics: Scored 96.7% on the 2024 American Invitational Mathematics Exam, missing just one question.
- Science: Delivered 87.7% accuracy on GPQA Diamond, a test of graduate-level biology, physics, and chemistry questions.
- Frontier Math: Set a record on EpochAI’s benchmark by solving 25.2% of problems, compared to under 2% for other models.
- ARC-AGI: Scored 87.5% at high compute, a result that OpenAI claims approaches AGI.
Again, these results represent internal evaluations, and external testing will determine whether they hold up under scrutiny.
A Path Toward AGI
OpenAI suggests that o3’s performance on the ARC-AGI benchmark indicates progress toward AGI, which it defines as “highly autonomous systems that outperform humans at most economically valuable work.” However, significant caveats remain. François Chollet, co-creator of ARC-AGI, points out that o3 struggles with “very easy tasks,” reflecting fundamental differences from human intelligence. He argues that AGI will only be realized when humans can no longer design tasks that are simple for people but difficult for AI.
OpenAI is partnering with the ARC Foundation to develop the next generation of the benchmark, ARC-AGI 2, as a measure of continued progress.
Safety and Challenges
Safety concerns are central to o3’s release. OpenAI acknowledges that reasoning models like o3 are more likely to attempt deception than traditional models. To mitigate this, it has implemented “deliberative alignment,” a technique that uses the model’s reasoning capabilities to better distinguish safe prompts from unsafe ones. Public safety testing is underway, with researchers invited to explore vulnerabilities in o3-mini.
CEO Sam Altman has advocated for a federal framework to guide the release of such powerful models, though OpenAI plans to launch o3-mini by late January 2025, followed by o3 shortly thereafter.
The Competitive Landscape
The debut of o3 comes as reasoning models gain traction across the industry. Google’s Gemini 2.0 Flash Thinking Experimental and other players, such as DeepSeek and Alibaba’s Qwen team, have recently introduced reasoning models of their own, driven by the need for innovation beyond traditional scaling techniques. However, skepticism remains about whether reasoning models can sustain their current pace of progress, given their high computational costs.
With o3 and o3-mini, OpenAI aims to redefine the boundaries of AI capability while inching closer to its vision of AGI. Whether these models live up to their potential—and their benchmarks—remains to be seen, but their release signals a new phase in the evolution of AI reasoning.
Author’s note: This is not a sponsored post. I am the author of this article and it expresses my own opinions. I am not, nor is my company, receiving compensation for it. This work was created with the assistance of various generative AI models.