Abstract
Problematic AI power-seeking requires three sets of conditions: agency (coherent, planning-driven behavior that considers rogue options), motivation (consequentialist goals over long enough time horizons) and incentives (rogue options actually being the rational choice given the AI’s full set of options and motivations). The first two look like the default trajectory for advanced AI; the third is much less clear, and depends on a mix of motivation control and option control. Even if no single superintelligence gains a decisive strategic advantage, the default path likely leads to a global vulnerability condition, where humanity’s continued empowerment depends partly on AIs choosing not to seek power.
Context and aims
- Part of Carlsmith’s series “How do we solve the alignment problem?”; the alignment problem is framed as building superintelligent AI agents and safely eliciting their beneficial capabilities while avoiding a loss-of-control scenario.
- Appeals to instrumental convergence (many goals make power useful) are often too vague: “we can’t just talk about an AI having some incentive to seek power, based on some part of its motivational system.”
- Instead, the analysis must consider the whole landscape of an AI’s options and motivations: shorter-term and non-consequentialist motivations, likelihood of success and failure, what the path to power involves, and the alternatives to going rogue.
Prerequisites for problematic power-seeking
Agency prerequisites
- Agentic planning: the AI searches over plans, evaluates them by criteria (its motivations), chooses and executes them.
- Planning-driven behavior: behavior must actually be driven by that planning, not merely capable of it (e.g. an LLM that can write a plan is not thereby acting on one).
- Adequate execution coherence: the AI’s future behavior must stay coherent enough that chosen plans get carried out rather than abandoned (a failure mode familiar in humans who resolve to exercise and then don’t).
- Consideration of rogue options: the planning process must include rogue options.
- Requires sophisticated world models (“strategic” or “situational awareness”) that recognise paths to power and their instrumental value.
- Even then, search can miss rogue options.
Motivation prerequisites
- Consequentialism: some component of the AI’s motivations must be focused on causing particular outcomes in the world; instrumental convergence arguments presuppose this.
- Adequate time horizon: the AI must care about consequences far enough ahead for power-seeking to succeed and for the resulting power to be put to use.
- Example: with ten minutes to finish a coding task, hacking extra compute may be useful in principle but there is no time.
- The required horizon depends on how fast power can be acquired and used; most worrying large-scale power-seeking requires “a decent amount of patience.”
Incentive prerequisites
- Rogue behavior must be broadly rational from the AI’s perspective, modelled as a choice between its best rogue option and best non-rogue option.
- The rogue option breaks down into:
- Success vs. failure at gaining or keeping the relevant power, with a probability assessment.
- Pay-out: the power obtained in expectation on success, plus the circumstances in which it must be used (the “end-state”).
- Path: what the AI must do, in expectation, to reach and secure the pay-out.
- Four motivational factors evaluate these components:
- Non-rogue satisfaction: how good the non-rogue option is.
- Ambition: how much the AI values the pay-out.
- Inhibition: how much it disprefers what it must do along the path.
- Failure aversion: how much it disprefers attempting and failing.
- The framework is described as “far from perfect” but capturing core dynamics.
Illustration: the embezzling employee
- An employee considers hacking her company’s payment system to steal $10M.
- Non-rogue satisfaction: how much she likes her job, salary and ordinary life trajectory.
- Probability: roughly 50-50 between succeeding and failing or getting caught.
- Pay-out and ambition: the money, but only while on the run in another country under a new identity; ambition is how much she wants money in that situation.
- Path and inhibition: lying and stealing; inhibition is how much she dislikes doing that.
- Failure aversion varies by failure mode: failing to hack in unnoticed is benign; getting caught is much worse.
Motivation control and option control
- Motivation control: shaping an AI’s motivations in desirable ways.
- Option control: shaping an AI’s options in desirable ways.
- Alignment discourse has focused too much on extreme cases:
- Extreme motivation control: AIs trusted with ~arbitrary options and ~arbitrary power.
- Extreme option control: safety robust to arbitrarily bad motivations.
- In-between approaches aim to make the specific options available (non-rogue option, pay-out, path, failure modes, success probability) and the AI’s specific motivations combine so that non-rogue behavior is the more rational choice.
- Everyday “alignment” mostly works this way: crime is prevented both by making it difficult and unlikely to succeed and by teaching children it is wrong.
Will the prerequisites be satisfied?
Agency and motivation: the default trajectory
- Good smarter-than-human assistants and employees would satisfy these conditions, and building them is a core goal of AI companies.
- Agentic planning, planning-driven behavior and execution coherence are what autonomous task completion requires; consequentialism follows from pursuing real-world objectives (science, business); situational awareness is a by-product of effective planning.
- Possible blockers: non-agentic systems, myopic systems, or models trained on seemingly non-agentic objectives that never develop agency.
- AIs can satisfy these conditions “even if their designers aren’t intending or expecting” it.
- Time horizon is less certain: many commercial tasks do not need unbounded horizons, but some (long-term company growth, portfolio management) plausibly need multi-year ones.
Incentives: much less clear
- Even for superintelligent agents, the argument that rogue options will be rational is “much less clear.”
- An AI may benefit from power in principle while high inhibition, failure aversion or attractive non-rogue options make going rogue irrational overall.
The decisive strategic advantage (DSA) case
- Classic AI-risk discourse focuses on a single superintelligence with a DSA over the rest of civilisation.
- A DSA simplifies the incentive analysis because takeover succeeds easily via “an extremely wide variety of paths”:
- Failure aversion becomes irrelevant, since failure is very unlikely.
- Inhibition is easily sidestepped, since some path will fit the AI’s inhibitions.
- Only weak ambition is needed: some instrumental value in taking over suffices.
- Without easy takeover, moving from “takeover has some instrumental value” to “the AI will attempt takeover” requires examining the detailed options, motivations and incentives.
- Carlsmith finds it “disturbingly plausible” that in fast take-off the first superintelligence would have a DSA, but “not at all obvious”:
- Many somewhat-weaker AIs will already exist and be reshaping the world.
- Weaker AIs in the same project can be directed at active option control (cybersecurity, red-teaming, monitoring).
- Unless development is already unipolar, frontier AIs at other projects will also be improving quickly.
- It is unclear what advantage superintelligence confers over large numbers of fast, copyable, somewhat-better-than-human AIs.
Global vulnerability conditions
- A global vulnerability condition: humanity’s continued empowerment rests on some combination of (a) AIs not choosing to seek power and (b) the success of active efforts to restrict their options.
- Carlsmith thinks this is harder to avoid than single-AI DSA scenarios.
- Restricting superintelligent agents’ options will itself need “lots of roughly-superintelligence-grade cognitive labor”; if sourced from other potentially dangerous AI agents, this risks vulnerability to coordinated power-seeking.
- Other possible sources of labor: non-agentic or myopic AIs, non-AI tools humans can verify, enhanced human labor, and defense-dominant settings where weaker systems can restrict stronger ones.
- Conclusion: “it looks likely to me that the default path to superintelligent AI agents leads us to a global vulnerability condition of some kind.”
- Talk of needing to get alignment right on “the first critical try” reflects the expectation of entering such a condition.
Typology of vulnerability conditions (Appendix 2)
- Vulnerability in general vs. to motivations:
- General: default option control leaves dangerous rogue options available, but active efforts still try to block them.
- To motivations: safety is counted on AIs not wanting to use dangerous options (e.g. you know the AI could hack out and self-exfiltrate, but rely on it not wanting to).
- Local vs. global:
- Local: rogue behavior by a given AI succeeds without seriously risking full loss of control.
- Global: failures of motivation and active option control lead to full loss of control with non-trivial probability.
- Easy vs. non-easy power-seeking: easy power-seeking is highly likely to succeed if attempted, letting the AI ignore the downsides of failure; non-easy power-seeking offers “no such luxuries.”
- Unilateral vs. multilateral:
- Unilateral: one AI’s power-seeking suffices.
- Multilateral: several AIs must go rogue; depends on ease of coordination, shared long-term goals, and correlation of misalignment (copies of the same weights are highly correlated; correlations may persist across fine-tunes, generations and projects).
- Coordinated vs. uncoordinated (multilateral):
- Coordinated: AIs work together (e.g. a monitor colluding with the AI it monitors); requires explaining why they can coordinate, why it is in their interest, and why they chose this form of coordination.
- Uncoordinated: AIs go rogue independently; if all must do so within a window, longer windows give humans time to respond.
- Worst case: global, dependent on motivations, and easy (the classic DSA scenario is vulnerability to easy unilateral takeover); vulnerability to coordinated takeover or uncoordinated loss of control is harder to avoid.
Appendix 1: extending the framework to capability elicitation
- The same structure applies to getting AIs to do tasks as intended, comparing:
- The best elicited option (the task done in the desired way).
- The best under-elicited, non-rogue option (not going rogue, but not doing the task as desired).
- Corresponding motivational factors: satisfaction with elicitation vs. satisfaction with non-rogue under-elicitation.
Next in the series
- The following essay (“Paths and waystations in AI safety”) sets out the high-level structure of getting from here to safe, useful superintelligent AI agents and possible milestones along the way.