Fun with +12 OOMs of Compute

Abstract

Visualising what 2020’s algorithms could produce with 10³⁵ FLOP makes it seem very likely (the author’s estimate: 90%) that transformative AI would follow within five years, which implies a probability distribution over training compute requirements far more concentrated below 10³⁵ FLOP than Ajeya Cotra’s — and correspondingly shorter timelines.

The hypothetical

• Premise: in 2016 a “Compute Fairy” makes computers 12 orders of magnitude faster (along with memory, bandwidth and other ingredients), and the deep learning boom proceeds unchanged otherwise over the following five years.
• Scale of the resulting runs: instead of AlphaStar and GPT-3 at ~10²³ FLOP, frontier projects train at 10³⁵ FLOP; the general cornucopia of academic and industry models moves from ~10²⁰ to ~10³² FLOP.
• Two questions are posed: (1) what could be built in this world, and (2) what is the probability transformative AI (TAI) appears by end of 2020 in it.
• TAI is used in Open Philanthropy’s sense: AI precipitating a transition comparable to or larger than the agricultural or industrial revolution.

Question one — five systems that could be built

OmegaStar

• AlphaStar scaled up by 7 OOMs to human-brain size (noted aside: AlphaStar has roughly 10% as many parameters as a honeybee has synapses).
• Larger models need less data for equal performance, so matching AlphaStar’s StarCraft level would take at most 10³⁰ FLOP; the budget instead allows training across the ~30,000 games in the Steam library, with equal per-game training time, plausibly inducing transfer learning.
• That consumes ~10³⁴ FLOP, or 10% of budget. Remaining compute goes to next-word prediction over the entire internet, masked-word, masked-image and video-continuation prediction, plus a chatbot interface for billions of users playing “maximize user engagement”.

Amp(GPT-7)

• A transformer with 10¹⁷ parameters and 10¹⁷ data points trained on the world’s video, audio and text — nearly 6 OOMs more parameters and training time than GPT-3, when only +4 OOMs is predicted to reach near-optimal performance on text prediction and the original GPT-3 paper’s task suite.
• The GPT-3→GPT-7 gap is about twice the size of the GPT-1→GPT-3 gap in loss terms.
• Layered on top: a public library of prompt-programmed GPT-7 functions (question decomposition, planning, option ranking, text scoring), plus tool functions such as search and webpage interaction, and functions that spin off and fine-tune new GPT-7 copies.
• These are composed into an amplification scheme — a “chinese-room bureaucracy” of nested calls that decompose tasks, execute sub-tasks and recombine results, fine-tuned so GPT-7 can select and write its own functions. The limiting case is an agent running a continual OODA loop with a text-file mission statement.
• ~18 OOMs of compute remain, enough to run 50,000 such bureaucracies for 100,000,000 function calls each (~3 subjective years) and then artificially evolve them over 50,000 generations, totalling ~10³⁵ FLOP. The same amplification-and-evolution could be applied to OmegaStar.

Crystal Nights

• A recapitulation of evolution in a planet-sized virtual world with detailed physics, seeded with primitive life-forms having varied mental and physical architectures and modular genomes.
• Ajeya Cotra’s estimate for this is 10⁴¹ FLOP against a budget of 10³⁵, so it requires being a million times more compute-efficient than evolution. Proposed shortcuts:
  • Start from “advanced” creatures (large-brained, sexually reproducing land animals) rather than nematode-brain-sized organisms — plausibly one or two OOMs.
  • Grant “magic traits” favouring intelligence and culture: respawning, inheritance of trained neural weights, metabolically cheap large brains, no skull or birth-canal constraint, neurons that do not break or become cancerous.
  • Override individually-optimal but species-suboptimal dynamics (e.g. antler arms races), raise the mutation rate, and intervene when species are stuck in local optima.
  • Optimise directly for intelligence rather than reproductive fitness, e.g. environments littered with monoliths dispensing food for solving reasoning puzzles — analogous to how much faster dog breeding is than wolf evolution.

Skunkworks

• Evolutionary design search plus simulation, following the precedent of NASA’s evolved spacecraft antennas (more efficient and cheaper in human effort than contractor designs) and SpaceX’s Raptor engine simulations.
• With +12 OOMs: simulations 1,000,000x more detailed, 1,000 auto-generated design variants iterated over 1,000 generations, potentially steered by an OmegaStar copy trained on the design “game”.
• Speculated consequences include nanomachine design (by analogy with AlphaFold 2) and easier WMD construction by minor states and non-state actors — a possible “vulnerable world”.

Neuromorph

• The Hodgkin-Huxley neuron model costs 1,200,000 FLOP per neuron-second, so a human brain plus body and virtual environment runs at ~10¹⁷ FLOP/s.
• Since 2020 scanning is inadequate and the model may be insufficiently detailed, the proposal is to scan a brain, reconstruct what the scan shows, fill remaining detail in a random but biologically plausible way, and run it for a simulated year to see whether it learns — as human children with initially random connections do.
• At ~10²⁵ FLOP per run, the budget permits 100,000 generations of 100,000 brains searched over by stochastic gradient descent across simulation parameters, neuron dynamics and initial conditions.

Question two — probability of TAI

• The question operationalises: what is the probability that training a transformative model with 2020 knowledge would take 10³⁵ FLOP or less?
• Author’s stated answer: 90%, with an inside-view answer of 99%. The gap is attributed to epistemic humility, unknown unknowns and deference.
• The only articulated no-TAI possibility is that transformative AI requires a special mental structure findable only by a blind search too massive for Crystal Nights; the author invites readers whose inside-view answer is below 90% to explain the remaining probability mass.
• A poll asks readers for their own inside-view answers.

Why it matters for timelines

• Cotra’s forecasting model is built around a probability distribution over the compute needed to train a transformative model with current knowledge, then layers on falling compute costs, rising willingness to spend, and algorithmic progress.
• Her best-guess distribution places its median at roughly the 10³⁵ mark, corresponding to answering 50% to Question Two.
• The post’s intended function is as an intuition pump for placing the bulk of that distribution below 10³⁵.
• Quantitative implication given: answering 80% rather than 50% shifts the median forecast roughly ten years earlier, from around 2050 to around 2040.
• The author frames the biggest timelines crux as “can we get TAI just by throwing a large but not too large amount of compute at the problem”, and proposes “10³⁵ FLOP or less” as the operationalisation of “large but not too large”.
• A related note: the author argues elsewhere that the relevant target is AI-induced points of no return, likely within a few years of TAI, and gives the same answer for that version of the question.
• Written as part of an investigation into timelines commissioned by the Center on Long-Term Risk; Skunkworks and Neuromorph are credited to Connor Leahy.