The case for reducing existential risks

Abstract

The chance of an existential catastrophe this century is roughly 1–20% — a thousand times or more what the public assumes and far above all natural risks combined — and because the area is extremely neglected, reducing these risks is plausibly the highest-impact thing a person can do with their career, even counting only benefits to the present generation.

Last updated June 2022; first published October 2017.

Framing

• Opens with Einstein’s 1939 letter to Roosevelt anticipating “extremely powerful bombs of a new type.” Within roughly a decade, enough existed for a handful of decision-makers to destroy civilisation.
• Proposed first priority for civilisation: survive. So long as civilisation continues, every other problem remains solvable; extinction forecloses all of them.
• An existential risk (Bostrom) is an event that could cause extinction or permanently and drastically curtail humanity’s potential — distinct from a global catastrophic risk, which allows recovery.
• Public estimates are far too low: in Spencer Greenberg’s survey, over 30% of respondents put the chance of extinction within 50 years below 1 in 10 million.

Natural risks are small and measurable

• A kilometre-wide asteroid impact carries roughly a 1-in-5,000 chance per century — already about 1,000 times the figure many people guess, and higher than the odds of being in a plane crash (~1 in 5 million per flight).
• Other natural risks include supervolcanoes, nearby supernovae or gamma-ray bursts, and natural pandemics.
• Toby Ord’s estimate: all natural risks summed are very unlikely to exceed a 1-in-300 chance of extinction per century.

Progress, and the dangerous epoch it created

• Since the Industrial Revolution, wealth, life expectancy, energy use, democracy, literacy, and individual freedom have risen sharply while poverty has collapsed; violence has arguably declined (Pinker, with countervailing evidence noted).
• The same charts show war-making capacity rising dramatically (Ian Morris’s estimates of global military power).
• Core tension: each new technology usually yields large benefits, but each carries a chance of granting more destructive power than we can wisely handle. The most prosperous period in human history is plausibly also the most dangerous.

Nuclear weapons — a history of near misses

• Cuban Missile Crisis: the US resolved to invade Cuba automatically if a spy plane were downed; one was downed the next day, and JFK reconvened the council anyway. Castro later favoured nuclear retaliation “even if it would’ve led to the complete annihilation of Cuba,” and some Cuban launch commanders had independent authority over tactical weapons.
• A Soviet submarine captain, out of radio contact and taking dummy depth charges for real ones, ordered a nuclear torpedo strike on the American fleet; Vasili Arkhipov’s refusal to approve prevented it.
• JFK later put the odds of nuclear war during the crisis at “between one in three and even.” Many further close calls followed, including after the Cold War.
• Experts today are as concerned about India–Pakistan as North Korea. The structural problem: large arsenals deployable in minutes mean false alarms can escalate rapidly.
• Atmospheric ignition was ruled out; modern models suggest an everyone-killing nuclear winter is unlikely, though model uncertainty is significant and even a mild winter could cause mass starvation and possibly unrecoverable destabilisation.
• Estimated chance of a civilisation-ending nuclear war next century: hard to argue it is under 0.3% — i.e. greater than all natural risks combined.

Climate change

• Most likely outcome is 2–4°C of warming: bad but survivable. Some estimates give a 10% chance of over 6°C and ~1% of 9°C.
• The chance of a massive CO₂-driven climate disaster is perhaps comparable to that of nuclear war, but even 13°C of warming looks unlikely to cause outright extinction directly.
• Nuclear war is therefore rated a larger existential risk, though climate change’s destabilising effects could exacerbate other risks, including nuclear conflict.

Emerging technologies

Engineered pandemics

• The 1918–19 Spanish Flu killed over 3% of the world’s population; rapid global transport would make containment harder today.
• Genetic engineering may soon permit a pathogen as contagious as the Spanish Flu but deadlier and able to spread undetected for years.
• Unlike nuclear weapons — which need huge factories and rare materials, and so are relatively controllable — designer viruses might be produced by a couple of people with biology PhDs. In 2006 The Guardian obtained segments of the smallpox genome by mail order.

Artificial intelligence

• Human dominance over chimps is a matter of intelligence, not strength — the analogy motivating concern about building something more intelligent than ourselves.
• 2017 survey of 350 published AI researchers: median 50% chance of high-level machine intelligence within 45 years, 75% by end of century (estimates varied greatly with question framing).
• Two failure modes, neither involving computers “turning evil”: misuse — one group using a powerful system to gain global control, analogous to a decade-long nuclear monopoly; and unintended consequences from a system that is hard to predict, hard to control, and hard to reverse once deployed. Documented by Bostrom (Superintelligence) and Stuart Russell.
• The same survey put the chance of a “bad outcome” at 10% and an “extremely bad” outcome such as human extinction at 5% — plausibly biased downward, since respondents earn their living from the technology. Combining with a 75% development probability gives ~4% chance of a major AI disaster this century.

Unknown unknowns

• Geoengineering and atomically precise manufacturing seem less imminent.
• More concerning are risks not yet conceived: people in 1900 would not have named nuclear weapons, genetic engineering, or AI. Each new technology is likened to a bet on a roulette number — usually we win, occasionally we lose everything.

Total risk

• Many experts put the total chance of extinction next century between 1% and 20%. Will MacAskill estimates ~1%; Toby Ord’s The Precipice estimates total existential risk at 1 in 6.
• Ord’s (very rough) breakdown:

Existential catastrophe via	Chance within next 100 years
Asteroid or comet impact	~1 in 1,000,000
Supervolcanic eruption	~1 in 10,000
Stellar explosion	~1 in 1,000,000,000
Total natural risk	~1 in 10,000
Nuclear war	~1 in 1,000
Climate change	~1 in 1,000
Other environmental damage	~1 in 1,000
’Naturally’ arising pandemics	~1 in 10,000
Engineered pandemics	~1 in 30
Unaligned artificial intelligence	~1 in 10
Unforeseen anthropogenic risks	~1 in 30
Other anthropogenic risks	~1 in 50
Total anthropogenic risk	~1 in 6
Total existential risk	~1 in 6

• These figures are about a million times higher than common intuition. Selection bias is acknowledged — researchers drawn to the topic likely hold high estimates — but a range including MacAskill’s and Ord’s is treated as plausible.

Scale

• Evaluated with 80,000 Hours’ framework: scale, neglectedness, solvability.
• With ~10 billion people alive mid-century, a 3% extinction risk implies ~300 million expected deaths — probably more than the next century’s deaths from diseases of poverty such as malaria.
• A “medium” catastrophe is more likely still: over 10% chance of an event killing more than 1 billion people, adding at least another 100 million expected deaths plus enormous suffering among survivors.
• This still understates the case, because ending civilisation forfeits the entire future. Carl Sagan’s calculation: at static population and 10-million-year species lifespan, some 500 trillion people are yet to come, making the stakes of extinction a million times those of a war killing hundreds of millions.
• Beyond population, whatever one values could exist in far greater quantity later — a world without want, intellectual and artistic achievement, a more just society, settlement among the galaxy’s ~100 billion planets.

Neglectedness

Cause	Annual targeted spending (highly approximate)
Global R&D	$1.5 trillion
Luxury goods	$1.3 trillion
US social welfare	$900 billion
Climate change	>$300 billion
To the global poor	>$250 billion
Nuclear security	$1–10 billion
Extreme pandemic prevention	$1 billion
AI safety research	$10 million

• Extreme pandemic prevention receives roughly 300 times less than climate change despite comparable risk; AI safety research perhaps 100 times less again.
• Scientific attention shows the same pattern — existential risk research receives less than dung beetle research (Bostrom). Political attention is similarly skewed toward short-term, vote-winning issues.
• Why: no single nation bears responsibility. A country capturing only 5% of the world’s population captures only 5% of the benefit of its spending, so every country underinvests, and defection is individually rational — a prisoner’s dilemma.
• Worse, the main beneficiaries are future generations, who cannot vote, lobby, or pay. Reducing these risks is a trans-generational global public good, which should make it among the most neglected ways to do good. Officials reportedly want to act but are constrained by news and election cycles, and most countries have no agency with these risks in its remit.

Solvability — what can be done

1. Targeted efforts

• Better disease surveillance to spot new pandemics faster.
• Climate mitigation: better solar technology, carbon taxes.
• AI: research on the control problem (Concrete problems in AI safety), with only ~20 people working full-time on such research at the time of writing.
• Nuclear: many experts think deterrence benefits survive far smaller stockpiles, which would cut accident risk and the chance that a war ends civilisation.

2. Broad efforts

• Improve institutional decision-making, since the same decision-makers will handle whichever risks arise, including unforeseen ones.
• Improve recovery capacity: the Global Seed Vault (whose access tunnel flooded from melting permafrost), disaster shelters, and the more neglected and cheaper option of alternative food sources producible without light during a prolonged winter.
• Improve international coordination — better foreign relations, better institutions — since these risks are human-caused and therefore human-preventable, with coordination the binding constraint.
• Broad efforts are more attractive under greater uncertainty; resources should shift from broad to targeted as risks come nearer.
• Education and development help but are rejected as focus areas: far less neglected (US education alone receives ~$800 billion public and ~$1 trillion private) and far more diffuse in effect.

3. Learning more and building capacity

• Global priorities research — a mix of economics and moral philosophy — is called the most neglected and important way to learn more, with only a handful of full-time researchers.
• Individual options: save money, build career capital and transferable skills.
• Community building may beat both, because a community’s capacity can grow faster than individual wealth — a year of one-on-one outreach yielding one comparable person is a ~100% annual return. Communities of safety-minded scientists and of policymakers concerned with future generations are named as gaps.

Cost-effectiveness

• Illustrative figure: $100 billion reducing risk by 1 percentage point would save ~100 million expected present-generation lives, at ~$1,000 per life.
• Greg Lewis’s more detailed model gives $9,200 per life year ($300,000 per life), which the author considers conservative on both risk level and marginal effectiveness; he would not be surprised if the next billion dollars came in under $100 per present life saved.
• Comparison: GiveWell’s Against Malaria Foundation saves a life for ~$7,500 (2017 figures) — so risk reduction is competitive on present-generation benefit alone, before counting future generations, where the effectiveness is orders of magnitude higher.
• Similarly, 10,000 talented people focusing careers here might achieve a 1% risk reduction, i.e. ~1,000 expected present lives saved each.
• Caveat offered: GiveWell’s estimate is far better researched, and these are informed guesses.

Objections the article addresses

• Obligations to friends and family — the argument concerns impartial altruism only; balancing personal life against it is a separate question.
• The risks are lower than argued — considered implausible; hard to be confident total risk is under 1% per century, and even 1% would warrant far more action than currently occurs.
• Little can be done — conceded that solvability is worse than for global health, but scale and neglectedness are held to outweigh it; “hits-based giving” rather than only proven interventions. The author might switch focus if two orders of magnitude more resources were invested. Related sub-objections rejected: that education already handles it, and that economic growth suffices (some growth raises risk, so differential technological development is preferred).
• There are better ways to help the future — value lock-in scenarios (permanent totalitarianism enabled by surveillance or conditioning, a fate possibly worse than extinction), irresponsible AI development under race conditions, the moral status of sentient digital minds, and s-risks (astronomical suffering) may matter as much or more. The practical upshot is more weight on managing transformative technology transitions and more global priorities research.
• Speeding up progress is rejected as a contender: what matters long-run is where we end up, not how fast we arrive; technology also causes many of these risks; and it is far less neglected given >$1 trillion of annual R&D.
• The future will be short or bad — if civilisation is virtually certain to end soon, or if the future is expected to be net negative, the value of risk reduction falls. The author grants a significant chance of early ending but holds the chance of a long future large enough to be worth fighting for, and expects a good future partly because people want and try for one.