The Goodhart Singularity

Abstract

Automating AI research will not rapidly produce domain-general superintelligence, because for almost all economically valuable tasks outside coding, the practice data needed to train real capability does not exist and cannot be manufactured through simulation — it can only be generated by deploying models into markets whose preferences are unknowable in advance. Absent that deployment, isolated self-improvement risks becoming a “Goodhart Singularity”: rapid progress on benchmarks that fails to generalize into real-world superintelligence.

The default narrative, and why Reed doubts it

• AI systems are increasingly building their successors, and full automation of AI R&D is expected to drive exponential capability growth — the industry’s working assumption is that domain-general superintelligence follows shortly after.
• Reed notes the field has already narrowed its actual target from general intelligence to “the targeted pursuit of coding and AI research,” even while the rhetoric still points toward general superintelligence.
• His central claim: automating AI research will not by itself rapidly produce domain-general superintelligence.

The five-step argument

1. Most capabilities require practice to master.
2. AI companies lack the necessary data to practice most domains.
3. “Sample efficiency” gains cannot substitute for this, because the relevant data simply does not exist to be used efficiently.
4. This cannot be solved with simulation or synthetic data, because automated researchers cannot evaluate success on tasks they don’t yet understand, and the relevant signal only emerges from market interactions with preferences that are unknowable in advance.
5. Data for superintelligence in non-coding domains therefore requires real deployment throughout the economy.

• Conclusion: the singularity is “bottlenecked on signal” — an isolated system improving against its own benchmarks looks like it’s approaching superintelligence while really optimizing for evaluation performance that doesn’t generalize. Reed names this failure mode the “Goodhart Singularity.”

Domain-specific intelligence requires practice

• Current AI progress looks less like accumulating general intelligence and more like “AIs become good at the things they are given the chance to practice.”
• Capability jaggedness — a model scoring like a Stanford Law graduate on the Bar exam while reliably losing at Tic-Tac-Toe — tracks the distribution of training data, not a general-intelligence phase shift expected from R&D automation.
• Much of current capability progress runs through hand-built RL environments; Reed cites Mechanize’s description of this process as one engineer spending a week patching a single observed model failure, and questions whether this artisanal, one-fix-at-a-time approach can scale “all the way into the singularity.”

Why the needed data doesn’t exist

• AI companies lack the right kind of data to practice most desired capabilities, and won’t acquire it cheaply without deploying models widely across the economy.
• On sample efficiency: DeepSeek reaching GPT-4-level capability at a fraction of the cost isn’t necessarily evidence of an algorithmic breakthrough — Anson Ho and Beren Millidge are cited as attributing intelligence-per-FLOP gains mainly to data quality improvements rather than algorithmic innovation.
• More fundamentally, for most tasks of interest the needed data doesn’t exist in any form companies can feed to their models. Reed’s proposed test: how much coding progress would we have today, at current talent and compute, if the internet had never existed? His answer — a very painful process requiring tens of billions of hours of human labor — describes the situation most economically interesting tasks are already in.
• The core problem is the wrong data type: for most tasks, what exists is “descriptions, commentary, or advice — but no record of the steps involved in performing the task itself.” Training on descriptions of a task is not the same as training on records of performing it.
• Illustrated with LLM performance on the game Dominion: a pipeline that teaches a model to write good-sounding strategy advice is different from one that produces an agent that actually wins, because the objectives diverge. Reed’s summary line: “LLMs relate to most tasks as McKinsey does to running a company.”
• Coding is the exception, because “the tokens left to us by our ancestors are constitutive of the task itself” — the textual record of code is the task, so LLMs excel here in a way they can’t where accurate description doesn’t equal task performance.

Why simulation can’t substitute for deployment

• No ground truth to evaluate against: without access to real-world outcomes, it’s impossible to tell whether a model has actually improved. Reed cites research finding that almost half of SWE-Bench submissions accepted by AI auto-graders would be rejected by actual human maintainers — auto-grading diverges from real merit. An isolated datacenter has no way to check whether one model’s improvements over another actually help it, say, win a presidential election, absent real voters and real outcomes.
• Markets reveal preferences that can’t be simulated in advance: drawing on a Hayekian point, preferences reveal themselves only through behavior, not beforehand — so the signal markets generate cannot be pre-computed or simulated from inside a datacenter.
• Illustrated with the board game Twilight Imperium: elaborate pre-game theorycrafting produces strategies that collapse within minutes of actual play, because real interaction generates outcomes planning alone can’t predict. Deployed AI will similarly hit marketplace realities that diverge from any simulated training environment.
• On continual learning: Reed asks how isolated researchers could design a perfect continual-learning algorithm without a proper, diverse set of real problems to learn over — noting “that’s how evolution did it.”

Implications

• Superintelligence is still possible, and the world likely won’t look normal in fifty years — this argument targets the mechanism and timeline of the standard R&D-automation story, not the eventual outcome.
• Most AI safety concerns don’t rest solely on the speed of R&D automation, but current discourse tends to treat deployment and diffusion as irrelevant to capability progress — mattering only for public awareness and fundraising rather than for building superintelligence itself. Reed argues this gap hasn’t been well explained, and attempts to bridge it.

Policy implications

• Policy focus on internal deployment and R&D automation as the primary path to superintelligence may be misplaced.
• The next phase of capability progress is likely to look like deployment, customer discovery, and real-world data collection, rather than compounding self-improvement inside a private datacenter.

Power dynamics

• Holders of proprietary, deployment-grade data about real economic activity may hold far more leverage than currently recognized, and should use it.
• This especially matters for Europe, which has diminishing prospects for building frontier superintelligence itself but retains rich scientific and industrial data.

Vertical conquest strategy

• Labs may pursue systematic, end-to-end conquest of individual economic verticals rather than waiting for incumbents to adopt their models.
• Citing David Oks, Reed notes labor displacement is more likely to come from fully automated new firms than from incumbents adopting AI — implying labs like Anthropic or OpenAI may find it strategically optimal to pursue vertical conquest directly, a possibility Reed says deserves serious consideration.