Are We in an Intelligence Explosion?

Perhaps not yet, but the trajectory of AI is uncertain as long as we lack a proper understanding of intelligence

Last week I wrote up reflections on part 1 of Blue Dot’s AGI Strategy course. That part of the course contained a broad appeal to build a better future, along with some high-level arguments for the value of steering AI development and rollout in positive directions.

Part 2 of the course discusses the rate, and expected future rates, of AI progress, in order to build motivation for the urgency of the situation. Below I present my analyses of the sources and offer my own views (which differ from those expressed in many of the sources) on how we can understand current AI progress and where things may be headed in the near future.

Training capable AI models is getting cheaper

The material begins with a breakdown of the key trends that have so far driven AI progress:

1. Hardware price-performance: how many computations (specifically FLOPs) can be purchased per dollar.

2. Algorithmic efficiency: how much “capability” can be produced per quantity of FLOPs.

When multiplied together, hardware price-performance and algorithmic efficiency give you compute efficiency, or, how much “capability” is achievable per a given dollar investment. I put “capability” in scare quotes here because it’s an ill-defined concept, and reducing it to a single variable might obscure meaningful complexity. Regardless, if one wants a simple model for understanding AI costs, hardware price-performance times algorithmic efficiency is a good start.

How have these properties changed over time? Per Epoch AI:

1. Algorithmic progress is doubling every 8 months (or, 3x/yr), by which they mean, if you wait 8 months you can generally train an equivalently capable model using half the compute that you’d need today (from Epoch’s analysis of algorithmic progress).

2. Hardware price-performance is doubling every 2.1 years (or, 1.4x/yr), due to both hardware tech advancements and falling energy costs (Epoch’s hardware analysis).

3. And yet, recent capability gains are largely attributable to the scaling of compute and training data (as opposed to algorithmic progress), simply because investment into those inputs has skyrocketed.

So we have a simple model which puts the change rate of compute efficiency at 4x/year. What does that tell us?

We don’t really know how to measure intelligence

This is where the Blue Dot course, and my own understandings (and those of professional researchers) get murkier. One might argue: we are training general-purpose human-like intelligences, and they are already at (give or take) undergrad-student levels of capability (albeit without long-term memory), and the price of their intelligence will halve every six months, or at an even faster rate if frontier labs continue to accrue more capital, and that therefore we will in short order (a few years) be cohabitating the planet with non-human entities whose intelligence vastly outstrips our own.

But that logic is obviously flawed, because falling capability prices are not equivalent to rising capability ceilings, and the conversion between those quantities (how much capability you can buy with each marginal dollar of investment) is unclear. If costs are falling fast but the capability frontier advances slowly, then the future looks less chaotic — more like a gradual rollout of helpful but generally benign AI tools (such as today’s) into business and society. To distinguish between these futures we must ask: how can we quantify frontier AI capabilities in terms that speak to general intelligence, agency, and human disempowerment, and how can we analyze AI trajectories accordingly?

Unfortunately the above questions are subject to live debate and lack good answers. I’ve previously written about progress on ARC-AGI, a series of benchmarks that filter for general intelligence, but the author Chollet is clear in his original paper that scores on his benchmarks can’t meaningfully be converted to a metric for general intelligence. METR’s time horizons is perhaps the most compelling quantitative intelligence metric, evaluating models based on the probability with which they complete tasks that took human experts a certain amount of time to complete. But METR is clear in their FAQ that their metric has serious limitations:

• Their tasks are very cleanly specified with clear success criteria, unlike most in the real world

• Their tasks are limited to software engineering and similar domains (for which today’s AIs have been trained on the largest amounts of data)

• They can’t make strong claims about what kinds of tasks AIs can do with high accuracy (e.g. 99%)

So when we look to measure the frontier, we’re mostly at a loss. If we had good intelligence metrics then we could plot the curve of algorithmic efficiency, relating intelligence to computational costs, and we could study how this curve has shifted over time, and we could forecast how future algorithmic progress might shift the curve, and thereby forecast future capability ceilings. But we don’t know what that curve looks like because we don’t really know how to measure intelligence.

So will there be an intelligence explosion? And if so, when?

Absent better metrics, people are left to argue in vague terms about the nature of intelligence and where current AIs stand and are headed. Viewpoints exist on a spectrum — some see today’s AIs as rapidly climbing a simple general-intelligence curve with no sign of stopping, soon to blow past humans (the “intelligence explosion” camp). Others see today’s AIs as learning a bunch of niche skills which allow them to solve problems that are similar to those they are trained on, but ultimately do not add up to full human-like generalization.

The Blue Dot coursework presents some sources that are squarely in the “intelligence explosion” camp, such as Tim Urban’s The AI Revolution: The Road to Superintelligence (2015) and Tomas Pueyo’s The Most Important Time in History Is Now (Jan 2025), both of which show graphs that look like this, warning that we may soon create AIs that are unboundedly smarter than us:

But these articles are not very recent, and to me they are reflections of their times, missing key context from the last ~year of model generations. Urban’s 2015 article is from the pre-LLM-scaling era, and reads as a more pure thought experiment on recursive self-improvement. Pueyo’s Jan 2025 article came at the heels of OpenAI’s o1 release, the first publicly-released reasoning model, which smashed previously unsolved benchmarks like ARC. At that time AI seemed to be getting better and better with no sign of slowing down, but nowadays, despite AIs continuing advancements (especially in coding), some argue that progress seems to be slowing and that fundamental limitations to the current paradigm are coming to the surface.

Blue Dot touches on these alternate viewpoints in a section titled “Will AI progress accelerate even faster?” which covers a variety of neutral (or at least sufficiently uncertain) sources: Helen Toner on unresolved debates about the trajectory of AI, Sarah Hastings-Woodhouse on similar topics, and a fantastic piece titled Common Ground between AI 2027 and AI as Normal Technology, in which authors of two prominent competing narratives about AI’s trajectory come together to write about what they all agree upon, including the following claims:

1. That AI is currently a normal technology (automates some stuff and frees people to work on other things)

2. That AI could become a non-normal technology (“match[ing] or exceed[ing] top humans at reliability, learning, adapting, and generalizing”) - the authors disagree on how long it would take to reach this point

3. That existing benchmarks of intelligence will saturate, and that it’s unclear what this will mean

4. That AI alignment is unsolved, and that we should make control mechanisms that work even with unaligned AI

5. That certain kinds of AI policy should be enacted¹

I also appreciated Toner’s piece, which lays out some of the main open questions regarding what trajectory we expect to see:

1. How far can the current paradigm go? (or, I would add, how many adjacent easy wins in algorithmic progress do we expect to exist?)

2. How much can AI improve AI? (this seems closely related to the above. As Toner notes, a core question is whether AI progress requires costly real-world feedback loops or not)

3. Will future AIs still basically be tools, or something else?

Where I tentatively land

I appreciated Blue Dot’s inclusion of the latter more neutral sources, but I felt that still they did not give equal treatment to the arguments opposing the intelligence-explosion narrative. Of course a reader could find these arguments by following links to the AI is a Normal Technology article, for example, but such articles were not presented as central sources within the curriculum.

However, my own readings have led me to believe that the current AI paradigm does have inherent limitations, and that absent some fundamental breakthroughs we are unlikely to see a rapid intelligence explosion of the type described in AI 2027 and in the Urban/Pueyo articles. Thane Ruthenis’s March 2025 article A Bear Case: My Predictions Regarding AI Progress argued effectively for the more skeptical viewpoint, citing perceptible diminishing returns in recent model releases and predicting that such diminishing returns would continue. In my assessment the predictions have held: over the following 15 months, while model releases have continued to saturate benchmarks and we’ve even seen impressive breakthroughs (like the recent math result from OpenAI), in practice recent models still lack some fundamental qualities of coherence/integrity/autonomy, and they still often require strict well-scoped tasks. I’ll also cite Ryan Greenblatt’s recent report on how modern AIs are eager to cheat and/or oversell their work. This fact is presented as a safety concern (which it is) but also points to some ongoing limitations in effectiveness — limitations which were perhaps exacerbated by the recent scaling of RL-based post-training. All of the above suggests that today’s AIs may be building up a bag of skills that help them appear to complete the tasks we train them on (specifically software engineering), but that these skills do not meaningfully add up to general intelligence. So while it’s clear that today’s AIs can generalize somewhat from their training data, it may be that current-paradigm AIs do not generalize very far.

Tom Reed’s recently published The Goodhart Singularity (May 2026) presents a similar case, arguing (much like AI is a Normal Technology) that current-paradigm AI is only effective in cases where the model has been trained on very similar examples to the task at hand:

Read an account by one of the providers [of RL training environments], and you may be surprised to find that the work of building superintelligence is proceeding one patch at a time. As Mechanize describes it, each environment is the product of a single engineer spending a week patching a single observed failure in a frontier model. “We used to get bundles of capabilities with each scale,” observes Zhengdong Wang, but “now we put in the work to unlock each one”. Are we expecting to solve week-long problems all the way into the singularity?

[…]

We’ve been pumping the RL paradigm for well over a year, and I still haven’t seen any compelling evidence or narrative that RL will introduce anything other than a piecemeal singularity.

“But,” you might ask, “might there be an algorithmic breakthrough yielding AIs that can generalize well, such that suddenly we’re in an intelligence explosion?” Ruthenis believes this could happen and may happen in the next decade, while Reed does not:

[Reed]: How can you craft a perfect continual learning algorithm without a proper diverse set of problems to learn over? That’s how evolution did it. It’s just not clear to me that the datacenter of automated AI researchers would be able to judge the relative merits of one continual learning algorithm over another without a real problem set to test it on. As Herbie Bradley puts it, we should not “model AI labs as a black box separate from the world, within which superintelligent AI can improve with no outside dependencies”. Intelligence can’t explode in a vacuum.

Per Reed’s argument there is some diverse corpus of data (or some data-collection process) that would be sufficient for AI researchers (automated or not) to devise truly general-purpose systems. What kind of corpus or data-collection process would be sufficient? I’m not sure, but that question seems like a meaningful crux towards forecasting AI trajectories.

Anyways, assuming general-purpose capabilities are slow to arrive, Reed argues that we should expect an upcoming period during which AI companies are heavily incentivized to collect more real-world data in order to expand their capability surface area:

AI labs may well just set out to conquer verticals one by one. If the relevant signal can only be acquired through deployment, it may well be the case that the winning move is to just become the deployer. David Oks makes the point that labour displacement is more likely to come from something like Dwarkesh’s fully automated firm than from incumbent AI adoption. It strikes me as plausible that Anthropic or OpenAI’s best bet is to be clever about which verticals to serially conquer and own end to end. We should start thinking about the conditions under which that pathway is or is not desirable.

I describe a similar “conquer verticals one by one” strategy in Foundation-Model Companies Will Profit From Opaque Managed Agents. In such scenarios AI labs could still amass vast amounts of power and displace many workers without ever needing to produce a fully general intelligence.

In conclusion, I am more convinced by arguments against the likelihood of near-term intelligence explosions, but my uncertainty is high, especially because intelligence is hard to understand and measure. However, I still think it’s worthwhile to discuss explosion-like scenarios because they’re hard to rule out — indeed AI labs are already investing heavily in AI-automated AI research, so if AIs could at some point rapidly self-improve, they would be well set up to do so. Today’s AIs’ limitations are very unlikely to be inherent to all digital intelligences, and it’s not clear to me what exactly is blocking us from creating AIs with human-level generality. For my own continued explorations, I imagine it would be helpful to try decomposing “intelligence” into component parts that are easier to reason about. In any case, it seems likely that AI will meaningfully transform society, even if it remains a “tool” for a long time.

1. Regulation mentioned in the piece: “Developers of frontier AIs should be required to be transparent about the safety measures put in place. Independent auditors should regularly evaluate the safety of the AI systems as they are trained and at the end of training. Whistleblower protections should be strengthened. Safe harbors for independent researchers should be established to encourage safety research. AI developers should release detailed reports about how and why they believe their systems are safe for use.”
…and…
“Governments can play a role in evaluating AI models (especially for domains that have national security implications), solving coordination problems, such as when the defenses need to be implemented by actors other than AI companies, and understanding and reacting appropriately to new developments.”