Abstract
Bostrom’s Maxipok principle — maximise the probability of an “OK outcome” that avoids existential catastrophe — rests on an implicit assumption the authors call Dichotomy, that future value is very strongly bimodal (either near-best or near-worst); the authors argue Dichotomy is false and that non-extinction events can permanently lock in values or institutions just as much as extinction can, so longtermists should broaden their focus from existential risk reduction to the wider set of “grand challenges” that could shift the future’s expected value by 0.1% or more.
Maxipok and its stakes
- States Bostrom’s Maxipok principle: “When pursuing the impartial good, one should seek to maximise the probability of an ‘OK outcome,’ where an OK outcome is any outcome that avoids existential catastrophe.”
- Existential risk is not only extinction risk — it’s the risk of any “drastic” curtailment of humanity’s potential, a catastrophe that makes the future roughly as bad as extinction.
- Illustrative case: imagine a strong world government that could cut bioweapon extinction risk from 1% to 0%, but that would lock in authoritarian control and undermine political diversity, reducing the value of all future civilisation by, say, 5%. Strict Maxipok would favour the world government even though it makes the world worse overall.
- Two reasons this matters: Maxipok could mean leaving value on the table, and in some cases following it could actively cause harm.
- This post summarises the longer paper “Beyond Existential Risk,” by Will MacAskill and Guive Assadi.
Is future value all-or-nothing?
- Maxipok’s overwhelming focus on existential risk depends on an implicit assumption the authors call Dichotomy: “The distribution of the difference that human-originating intelligent life makes to the value of the universe over possible futures is very strongly bimodal, no matter what actions we take.”
- If futures really do cluster into a very-good cluster (survival and flourishing) and a very-bad cluster (existential catastrophe), the only thing that matters is shifting probability mass from bad to good — making existential risk reduction the overwhelming priority.
Three arguments for Dichotomy, and their problems
- Convergence (“basin of attraction”): perhaps any society competent or good enough to survive is inevitably pulled toward the best possible arrangement, the way objects captured by a planet’s gravity end up in orbit or crash into it. Counter: a future could be controlled by a single immortal dictator (an uploaded human or an AI) who survives indefinitely with no guarantee of ever discovering or caring about what’s morally valuable — nothing about being powerful and long-lived guarantees moral enlightenment.
- Bounded value: maybe value is bounded above at a fairly low level, so avoiding extinction and outright ruin is close to enough to secure most of the possible value. Counter: badness is almost certainly not bounded below (more awful things can always be added), which implies a long negative tail rather than Dichotomy — it doesn’t clearly support the dichotomous view.
- “Extremity of the Best” (“valorium”): perhaps the single most efficient use of resources produces orders of magnitude more value than almost anything else, so civilisation either optimises for it (enormously good future) or doesn’t (comparatively negligible value). Some supporting evidence: Bertrand Russell’s description of love’s intensity (“ecstasy so great that I would often have sacrificed all the rest of life for a few hours of this joy”) and a survey where over 50% of people reported their most intense experience being at least twice as intense as their second-most intense. Counter-evidence: Weber–Fechner psychophysical laws suggest perceived intensity scales logarithmically with stimulus, implying light rather than heavy tails — and it’s unclear how far human experience generalises to cosmic-scale value.
The case against Dichotomy
- Resource division: future resources will likely be split among different groups — nations, ideologies, value systems, or star systems controlled by different factions. If space is defence-dominant (a group holding a star system can defend it, plausible given interstellar distances), the initial allocation of cosmic resources could persist indefinitely — with value scaling continuously with what fraction of resources go to which groups, rather than clustering at the extremes. The same logic applies within a single group’s own resource allocation.
- Uncertainty: we’re deeply uncertain across many competing theories about convergence, bounded value, and heavy tails. Averaging across dichotomous and non-dichotomous distributions under that uncertainty — “like averaging a bimodal distribution with a normal distribution” — produces a non-dichotomous expected distribution, with probability mass spread across the middle range.
Non-existential interventions can persist too
- Even without Dichotomy, Maxipok could be defended by “persistence skepticism” — the view that only existential catastrophes have lasting effects, while everything else (world wars, cultural revolutions, dictatorships) eventually “washes out.”
- The authors reject persistence skepticism: they think it’s “reasonably likely — at least as likely as extinction” that this century will see lock-in of values, institutions, or power distributions, via two mechanisms:
- AGI-enforced institutions: once AGI exists, a superintelligent system tasked with enforcing a constitution or legal structure could maintain it across astronomical timescales — via exact self-copying, distributed backups, and verification of goal stability — and the enforcer itself might then prevent any later modification.
- Space settlement and defence dominance: once space settlement becomes feasible (perhaps soon after AGI), cosmic resources will likely be divided among groups, and defence-dominant star systems mean that initial allocation — and each group’s locked-in values — could persist indefinitely.
- Because early events can shape how later lock-ins go (e.g. a dictator who secures power for 10 years could use it to entrench power for 20 more, then reach AGI within that window), several early choices could have permanent effects: the values programmed into early transformative AI, the design of the first global governing institutions, legal and moral frameworks for digital beings, the principles for allocating extraterrestrial resources, and whether powerful countries remain democratic during the “intelligence explosion.”
So what: beyond existential risk to “grand challenges”
- If Maxipok is false, longtermists should broaden their focus from existential catastrophe alone to “grand challenges” — decisions whose outcomes could alter the expected value of Earth-originating life by at least one part in a thousand (0.1%).
- Concrete new priorities suggested:
- Deliberately postponing or accelerating irreversible decisions to change how they get made — e.g. international agreements on AI development timelines, reauthorisation clauses for major treaties, or preventing premature allocation of space resources.
- Making outcomes as beneficial as possible when lock-ins do occur — e.g. advocating for distributed rather than single-country or single-company control of AI governance, encouraging moral reflection before lock-in, or ensuring humble and morally-motivated actors (rather than ideologues or potential dictators) design locked-in institutions.
- Influencing which values and which groups control resources at critical junctures — e.g. promoting better moral worldviews, working on the rights of digital beings before those rights are locked in, or affecting the balance of power between democratic and authoritarian systems.
- Building better mechanisms for global coordination, moral deliberation, and governance handoffs to transformative AI systems.
- None of this is primarily about reducing the probability of existential catastrophe, but if Maxipok is false these interventions could be just as important as — or more important than — traditional existential risk reduction work.
- Clarifies that existential risk reduction remains valuable in absolute terms; the argument is for going beyond a singular focus on survival to also improving the future conditional on survival — “the stakes remain astronomical.”