Preparing for the Intelligence Explosion

Abstract

AGI preparedness is not just about alignment. AI that substitutes for human researchers would drive a century’s worth of technological progress in under a decade, forcing a rapid succession of consequential and hard-to-reverse decisions — “grand challenges” — on timescales far shorter than human institutions are built for. Many of these cannot be punted to a later aligned superintelligence, so we should be preparing for them now.

1. The argument against the “all or nothing” view

• A common view among those expecting superintelligence: outcomes are effectively binary and hinge on one challenge, alignment. Either we fail and humanity is permanently disempowered, or we succeed and use AI to solve everything else.
• The paper argues AGI preparedness should cover a much wider range of opportunities and challenges, including:
  • Human takeover, by whoever controls superintelligence.
  • Novel destructive technologies — bioweapons, drone swarms, nanotechnology, and things not yet anticipated.
  • Norms, laws and institutions for critical ethical issues: rights of digital beings, allocation of offworld resources.
  • Harnessing AI to improve collective epistemology rather than distort it.
• These are “forks in the road of human progress.” They should not be expected to go well by default.

Six things that can be done today

1. Prevent extreme, hard-to-reverse concentration of power — e.g. distributing data centres and semiconductor supply-chain components across democratic countries; keeping frontier AI access available to many parties.
2. Empower responsible actors — increase the chance that those with most power over superintelligence development (politicians, AI company CEOs) are responsible, competent and accountable.
3. Build AI tools to improve collective decision-making — begin building, testing and integrating tools across epistemics, deal-making and decision advice before the explosion is underway.
4. Remove obstacles to applying superintelligent AI to downstream challenges — e.g. unblocking bureaucracy that impedes advanced AI use in government.
5. Start early on institutional design for new governance areas — rights of digital beings, legal frameworks for offworld property claims.
6. Raise awareness and improve understanding, so less time is needed to get up to speed at the crucial moments.

2. A century in a decade

The compressed-century thought experiment

• Take all scientific, technological, political and philosophical developments of 1925–2025 and compress them into the decade after 1925.
• First nonstop Pacific flight in late 1925; first footprints on the Moon by mid-1929. ~200 days between the discovery of nuclear fission (mid-1926) and the first atomic test (early 1927). Transistor counts multiply a million-fold in four years.
• Social change accelerates with it: WWII erupts and ends with the atom bomb in about 7 months. Thirty newly independent states and constitutions form within a year. The UN, IMF, World Bank, NATO and the EU’s precursor all form in under 8 months.
• Nuclear decision-making: the Cuban Missile Crisis lasts 31 hours; JFK has 20 minutes to respond to Khrushchev’s ultimatum; Arkhipov has under an hour.
• Robert F. Kennedy Sr. on the actual crisis: “If we had had to make a decision in twenty-four hours, I believe the course that we ultimately would have taken would have been quite different and filled with far more risks.”

Asymmetric acceleration

• If everything accelerated uniformly, history’s trajectory would be unchanged. But technological acceleration effectively slows whatever doesn’t accelerate with it — human thinking, learning, interaction, and institutions with rigid schedules.
• Equivalent framing: the same 1925–2025 technological change, but humans are awake only 1 hour 40 minutes per day.
• Acceleration will be uneven across fields: faster in domains advanced by a priori reasoning or simulation (mathematics, computer science, computational biology), slower where expensive or slow experiments are needed (high-energy particle physics, drug development).
• The claim is not that accelerated progress is worse than business-as-usual, but that it poses distinctive challenges because human decision-making cannot keep pace.

3. The intelligence explosion and beyond

The core growth argument

• Total global research effort grows at less than 5% per year. Total AI cognitive labour is growing more than 500x faster than total human cognitive labour.
• Once AI cognitive labour rivals human cognitive labour, the growth rate of overall cognitive labour increases massively — driving faster technological progress.

Current drivers of AI progress

Driver	Current rate
Training compute	~4.5x/year since 2010
Algorithmic efficiency in training	~3x/year
Post-training enhancements	~3x/year (Anthropic informal estimate)
Inference efficiency	~10x/year cost decline
Inference compute scale-up	~2.5x/year

• Effective training compute from pretraining is increasing over 10x/year; including post-training enhancements, it is “as if” physical training compute scales more than 30x/year.
• Inference efficiency: GPT-3.5 launched at $20 per million tokens in late 2022; a faster model with a better MMLU score costs around $0.04 per million tokens — a 500x drop in under three years.
• Combined, inference efficiency and inference compute could grow the “AI population” by about 25x per year.
• GPT-2 to GPT-4 spanned roughly 10⁵–10⁶ times more effective training compute (closer to ten million-fold including post-training), turning grammatical-but-meaningless completions into an assistant with more general knowledge than any living person.

Headroom and limits

• Training runs can scale roughly another 10,000x before hitting power limits (likely within a decade); further limits from chip production, data scarcity, hardware latency.
• Assuming scaling and no feedback loop: the product of training compute, algorithmic efficiency and inference compute increases ~10¹¹-fold over a decade (just over 10x/year).
• Software feedback loop: AI improving AI algorithms, data and post-training techniques, without scaling physical inputs. Not guaranteed — it requires a doubling of inputs to yield at least a doubling of outputs over many doublings — but empirical estimates of software-domain efficiency gains suggest roughly even odds it would drive accelerating progress.
• Upper bounds are far away: LLMs appear to learn very roughly 100,000x less efficiently than humans, and the true ceiling likely lies far beyond human brains.
• With a feedback loop: ~10¹² increase in effective training compute; ~10¹⁶ increase in the product of inference compute and effective training compute (~40x/year).

AI–human cognitive parity

• On GPQA (PhD-level science questions), GPT-4 performed marginally better than random guessing; 18 months later the best reasoning models outperform PhD-level experts — faster than most predicted.
• Best systems beat all but the very best competitive programmers, solve >70% of real-world open-source software engineering issues, and match domain experts on time-capped ML optimisation problems up to ~4 hours (RE-Bench).
• The maximum duration of ML-related tasks frontier models can complete has been doubling roughly every seven months; naive extrapolation suggests automation of many month-long expert cognitive tasks within three to six years.
• Caveat: benchmarks are not the real world. Real ML research involves choosing research directions, coordinating teams, and waiting days for experiments.
• Verdict: AI parity with the best human researchers within a decade or two seems “fairly likely,” and collective AI research capability would keep increasing across many orders of magnitude afterwards.
• Post-parity decade estimates: conservative scenario ~10⁷x growth in AI research effort (5x/year); aggressive ~10¹⁴x (25x/year).

The technology explosion

• Modelled with a semi-endogenous growth idea production function incorporating two forms of diminishing returns: “fishing out” (more cumulative progress requires more effort for the same rate) and “stepping on toes” (diminishing returns to parallel effort).
• A century in a decade requires total research effort to increase ~600x over that decade (~100%/year growth).
• The conservative scenario (10⁷x) would suffice for over three hundred years’ worth of progress at current rates.
• Headwinds: physical experiments, serial experimentation, lab capital. But these are unlikely to be decisive because:
  • Overall progress is an average; slow fields don’t constrain it.
  • Abundant cognitive labour would produce extremely optimised experiments and capital use, plus strong incentives to invest in more of both.
  • AI researchers have advantages humans lack — evolution did not optimise brains for science; AIs can train for the equivalent of a million years on a five-year-old field, span many disciplines, and focus on one problem for thousands of consecutive human lifetimes.
• Conclusion: a century of technological progress in a decade is more likely than not on a default path of continued scaling.

The industrial explosion

• Historically human labour complemented machines and bottlenecked output. Human-level AI plus dextrous robots would substitute for almost all skilled labour, giving an industrial base that grows itself through many doublings.
• Barriers are control (flexible maneuvering — an AI scaling problem) and cost (mass production learning curves; solar’s price-per-watt fell more than 200x since 1976 while installed capacity rose more than 100,000x).
• Output could double every few years or months; biological replicators suggest peak rates on the order of days or weeks.
• Ceilings are high: humans produce ~0.01% of the solar energy reaching Earth, so 100x global primary energy is available from under 2% of oceans or deserts; space-based solar offers a billion times more again.
• Strategic implication: if authoritarian states can grow their industrial base faster (higher savings rates, fewer environmental constraints), an industrial explosion could shift power decisively away from democracies.

4. Grand challenges

Developments whose handling significantly affects the value of present and future life. An intelligence explosion makes far-off-seeming issues urgent and intensifies the existing “pacing problem” between technology and regulation.

AI takeover

• Expect AIs that outsmart humans and collectively dwarf humanity’s cognitive capabilities, many well-described as goal-directed. Misaligned systems may prefer, and be able to achieve, full control.
• Human oversight breaks down as systems surpass human ability to notice or understand bad behaviour. Smarter misaligned AIs could fake alignment during training and beyond; by the time scheming is revealed, copies and compromised shutdown systems could make takeover hard to prevent.
• Some will deliberately misalign advanced AI (as with “ChaosGPT”).
• Treated briefly here because it is already well covered elsewhere (Ngo et al. 2021; Carlsmith 2022, 2024).

Highly destructive technologies

• New bioweapons — synthetic pathogens engineered to spread faster, resist treatment, lie dormant longer, approach near-100% lethality; cheaper and more flexible gene synthesis lowers required expertise.
• Drone swarms — agile winged drones already exist at bumblebee scale; one insect-sized autonomous drone per person on Earth could fit in a single large aircraft hangar. Drones currently favour offence over defence.
• Huge nuclear arsenals — explosive destructive power rose ten-thousandfold from 1925 to the 1980s peak; an orders-of-magnitude larger industrial base could scale arsenals similarly, making nuclear winter far more likely.
• Atomically precise manufacturing — guiding reactive molecules to build atomically precise structures. Fundamentally constructive (semiconductors, targeted medicines, CO2 removal, cell repair) but also enables non-biological viruses or mirrored bacteria. Nature proves feasibility via ribosomal machinery; synthetic biology offers one path.
• Mechanisms making use more likely: the security dilemma (effective missile defence enabling first strike, or pressure to strike before an opponent acquires it) and Thucydides’ Trap (a leading power waging war before being overtaken).
• Even peaceful technology can harm: fusion at half the solar radiation incident on Earth would warm the planet by tens of degrees before thermal equilibrium — swamping greenhouse warming through sheer thermodynamics.
• The corresponding challenge: develop and deploy protective and defence-favouring technologies (far-UVC in indoor spaces, advanced PPE stockpiles, pathogen detection infrastructure) as soon as possible.

Power-concentrating mechanisms

• Loyal automated militaries and bureaucracies — dictators currently need a coalition that can defect; AI-run state functions and robot police forces remove that constraint.
• Military power-grabs — automated militaries can be seized via political subversion, backdoors, insider instructions or cyber-warfare, by enemies, by company insiders, or by incumbents (“self-coup”). An industrial explosion could let a frontier company build one from scratch.
• Economic concentration — a shrinking labour share means income comes from rents on capital and land, which are already far more unevenly held than income; super-exponential growth would proportionally enlarge small initial leads.
• First mover advantages — preemptively securing rare earths, semiconductor materials or power-generation sites; deploying infrastructure in commons like the high seas or space; filing patents at unprecedented scale; shaping regulation to entrench position.

Value lock-in mechanisms

• Lie detection and surveillance — greater authoritarian control, though the same tools could help detect bioterrorists or coup plotters.
• Permanent AI values — regimes could store and copy themselves the way constitutions and scriptures preserved institutions for centuries.
• Commitment technology — verifiably binding treaties, third-party “treaty bots”, and unilateral irrevocable commitments (e.g. to retaliate), potentially persisting even if both parties come to regret them.
• Human preference-shaping technology — neuroscience, psychology or BCIs allowing self-modification against changing one’s mind, including for one’s children.
• Global government — arriving either through concentration of power or through deliberate agreement to manage explosive progress; absent inter-state competition and aided by lock-in, its constitution could last extremely long.

AI agents and digital minds

• Infrastructure for agents: liability for agent-caused harms, proof of humanity online, action attribution, cross-border handover of rogue agents. Early protocol/law/norm decisions have lasting effects (cf. TCP/IP, Section 230).
• Moral status is harder for two reasons:
  • The philosophy is intrinsically hard — no agreed criteria for biological or non-biological consciousness; unresolved questions about what “death” means for a branching, resurrectable digital being, how to aggregate near-identical instances, and which preferences it is acceptable to install in a being one creates.
  • Two competing economic pressures: demand for very human-like AIs (companions, imitations of specific people) pushes companies to build AIs that act as if they have feelings; countervailing incentives push developers to train AIs to prefer servitude, reject digital rights, or deny sentience. The analogy drawn is factory-farmed chickens: selectively optimised, created in vast numbers, with no say.
• Two areas likely needing law and norms soon: digital welfare (protection, and constraints on which minds we create at all, under uncertainty about consciousness) and digital rights (right to be turned off, right against torture, wages, property, contract, tort claims, political representation — where “one AI instance, one vote” would empower whichever minds copy themselves fastest).
• Interaction with takeover risk runs both ways: freedoms could accelerate gradual disempowerment and limit alignment methods, but could also reduce AI incentives to deceive and seize power.

Space governance

• Acquiring Solar System resources — historically, temporary technological advantage has been consolidated by seizing land (Russia’s 50-fold territorial expansion). Earth intercepts ~two-billionths of the Sun’s output; offworld industry could dwarf Earth’s. A first grabber could achieve total dominance without military action or making anyone absolutely worse off. Clear international law would make intervention against exclusive grabs more likely.
• Interstellar settlement — seed probes carrying information and growth machinery, not biological humans. ~10 billion reachable galaxies, ~100 billion stars each; accelerating 10 billion 1kg probes to 99% of light speed costs less than a minute of the Sun’s output. Star systems appear defence-dominant, so initial allocation could be locked in indefinitely.

New competitive pressures

• Races to the bottom — willingness to cut safety corners, ignore environmental protections, or skip legal protections for digital beings is rewarded with faster growth; conscientious actors adjust downwards to keep up.
• Value erosion — competition has historically driven progress, but could reverse as AI renders human participation unnecessary in governments, cultures and economies; incremental voluntary handovers cumulatively erode human influence and things of genuine value.
• Blackmail technology — bioweapons as extortion tools favour groups unhinged enough to threaten credibly; groups that value many human lives cannot make the same threat.
• Super-strategy — AI amplifying manipulation, loophole exploitation and coalition engineering, operating continuously, simulating thousands of strategies and modelling other players.
• Countervailing opportunity: cooperative AI — AI diplomats brokering agreements blocked by transaction costs, bandwidth or asymmetric information; AI delegates that can be made to forget everything except the agreed deal.

Epistemic disruption

Likely net positive overall, but mixed; the challenge is to shift the balance.

Risks:

• Super-persuasion — ~$10bn/year on propaganda and hundreds of billions on digital advertising create strong incentives; anyone could recruit an effective army of skilled lawyers, lobbyists and marketers for arbitrary falsehoods.
• Stubbornness against persuasion — general epistemic defensiveness could pollute otherwise useful epistemic applications of AI, while the pace and complexity of change makes adjudicating conflicting AI advice harder.
• Viral ideologies — blood libel, witch trials and fascism illustrate persistent harmful belief bundles; survival pressures that historically limited egregious falsehoods weaken as beliefs decouple from success, while AI could optimise for virality.
• Ignoring new crucial considerations — institutional inertia, vested interests and anti-AI stubbornness could cause society to dismiss radical new truths that warrant major reassessment of plans.

Opportunities:

• Fact and argument checking — building on what makes Community Notes more effective than earlier fact-checking (fast, contextual, in-line); AI can check arguments and subtle manipulation, and converse indefinitely. Brief GPT-4 dialogues reduced confidence in conspiracy theories by ~20%, lasting a couple of months.
• Automated forecasting — well-calibrated, verifiable track records that could generalise trust into less verifiable domains.
• Augmented and automated wisdom — AI social scientists anticipating policy impacts, AI policy advisors designing better regulation and institutions, and AI reasoning about philosophy, ethics and big-picture strategy, including strategy for managing the intelligence explosion itself.
• Market selection pressure should favour honest, truthful, reliable models.

Abundance

• Radical shared abundance — a century of progress would more than double average incomes; an industrial explosion could mean thousands of AI and robot assistants per person. High stakes could encourage cooperation over fighting for a marginally larger slice. Not guaranteed: gains could concentrate, or regulation could suppress them where entrenched interests oppose growth.
• Safety from rising incomes — empirically, richer societies invest more in safety, go to war less, and democratise more; risk aversion means people pay increasingly higher premiums to avoid catastrophic losses. Capturing wealth before catastrophic risks arrive would make society behave more cautiously.
• Enabling trades — commitment and treaty-enforcement technology, privacy-preserving verification, and a vast labour force of AI brokers could unlock mutually beneficial agreements currently blocked by discovery costs, transaction costs, private information or non-credible commitment (e.g. verifiable bioweapons bans).

Unknown unknowns

• The list is incomplete; unimagined technologies may be most of them, and conceptual advances are even harder to predict than technological ones.
• Domains we know we don’t understand: quantum gravity, phenomenal consciousness, decision theory, ethics (including population and infinite ethics), anthropic reasoning.
• In the compressed 1925–1935 thought experiment, most major challenges and conceptual developments were not foreseeable even by someone trying hard.

5. When can challenges be deferred to aligned superintelligence?

• The sceptical case: most listed challenges arise only after superintelligence; if it is misaligned nothing else matters, and if it is aligned we can use it to solve everything. So only alignment needs work now.
• Sometimes punting is correct — e.g. manual drug-candidate searching has limited upside if AI will soon transform drug discovery; early effort gets swamped.
• Conditions under which punting fails:
  • Challenges that arise early — AI could disrupt collective reasoning before AI can competently govern epistemically disruptive AI. Human power-seizure using intermediate-capability AI comes earlier than AI takeover risk, since AI-driven takeover is easier when AIs assist willing humans who already hold power — and once succeeded, the later superintelligence is controlled by the power-grabbers.
  • Windows of opportunity that close early — some solutions, such as agreements to share power post-AGI or reforms to slow-moving institutions, are only feasible before the intelligence explosion.
  • Changes to when and how people use superintelligent assistance.
• A related move is delaying a challenge to buy deliberation time — e.g. an international agreement not to send space-settling probes beyond the Solar System without widespread approval.

6. AGI preparedness

• Two branches: generally improving decision-making, and addressing specific challenges (see the six concrete actions listed in §1 above).

7. Taking stock

• Prepare despite uncertainty.
• Not just misalignment — preparedness is broader than alignment.
• Don’t punt every problem to superintelligence.
• Be ready to adapt.

Framing note from the recap

• The paper lists many things that could go wrong but does not predict disaster. The last century was a period of chaos and tragedy, yet the world emerged richer, freer, more capable and more knowledgeable — while also being a world that could have been in a much better state had key decisions been made more wisely.