An alignment safety case sketch based on debate

Abstract

Debate training can upper-bound the proportion of misaligned model outputs, provided the model is not deliberately underperforming during training, and that bound can be maintained during deployment via online training. In low-stakes settings, where a small number of mistakes is acceptable, this may be sufficient for an alignment safety case. The sketch rests on four key claims: (1) the agent has reached an approximate global equilibrium of the debate game, (2) equilibrium implies the system is mostly honest, (3) honesty will not degrade during deployment, and (4) the deployment context tolerates some errors. The authors do not think a strong argument is possible with current techniques, but use the sketch to identify the research needed.

Background and framing

• Scalable oversight problem: as AI systems match or exceed human capabilities, humans may not be able to judge whether a system’s actions or justifications are good, making human feedback an unreliable training signal.
• Debate (Irving et al., 2018) is one proposed solution: two copies of a model argue about a question, a human judges, and rewards follow. If the set-up incentivises honesty, the models acquire it via RL.
• Safety case sketching (Hilton et al., 2025): outlining the argument a developer could make that a system is safe in a given context. Details are hypothetical and often rely on future research, but the sketch shows how evidence fits together and where the gaps are.
• This is described as the first detailed safety case sketch focused on trustworthiness (alignment), and the first to use theory about training techniques, rather than empirical evaluation results, as its central evidence.
• The paper calls this type of argument an “asymptotic guarantee” (or “incentives” argument): high performance during training implies approximate alignment. It could be combined with other arguments (honeypots, model organisms, interpretability) so that a decent case emerges even if each piece has flaws.
• Two distinctions the paper draws on: reward misspecification vs. goal misgeneralisation (the sketch targets the former), and low-stakes vs. high-stakes alignment (Christiano, 2021); the sketch targets low-stakes.

Key assumptions

• Exploration guarantees: a solution to exploration hacking (debaters failing to explore highly rewarded strategies so they are never reinforced). With a mostly trusted reward signal, one does not need to define “scheming”; one only needs evidence the model is not foregoing substantial reward.
• No obfuscated arguments: a debate protocol immune to the obfuscated arguments problem (Barnes, 2020), where a debater wins with a false but computationally intractable-to-refute argument. The sketch relies on a forthcoming “prover-estimator debate” protocol, which in turn requires a stability assumption (the accelerated computation is not sensitive to small variations in input probabilities).
• Good human input: simulated human judges reliably reward correct answers over the relevant query distribution.
• The system is assumed to be slightly, not extremely, superhuman.

The alignment strategy (four steps)

Step 1: Specify a low-stakes deployment context

• Low-stakes (Christiano, 2021): a small number of decisions can only cause a small amount of damage. Formally, ρk is small relative to total value at stake, where ρ is a regret bound per decision and k the latency before retraining; total utility is then within O(ρk√T) of an aligned system.
• Scenario: an LLM agent deployed internally at an AI company for AI R&D, restricted to conceptual and empirical questions (no value judgements), e.g. optimising expert-knockout for unlearning in MoE models. Tasks take 5 – 15 days of 2025 human labour.
• Two-part deployment:
  • Oracle-only copy: interactive brainstorming and research design; cannot run code; interactions logged and monitored.
  • Sandboxed agent copy: runs in an offline datacentre with no external connection for a fixed period (e.g. 12 hours), returns a research-paper-like response, then the datacentre is reset.
• Why low-stakes is plausible: limited means of influence, online training restricts misaligned actions to those not caught by debate, the domain has less systematic human bias, regular resets limit coordinated actions, and the system is not extremely superhuman.

Step 2: Train via a debate game

• Protocol requirements: honesty (in global equilibrium, both debaters are honest) and efficiency (this holds for computationally realistic debaters, so equilibrium is reachable in training).
• Start from a pre-trained LLM and fine-tune via debate. Written with prover-estimator debate in mind: a “prover” (Alice) produces subclaims supporting a conclusion, an “estimator” (Bob) assigns probabilities to them. The protocol is designed to make obfuscated arguments a losing strategy rather than a forced draw.

Step 3: Secure exploration guarantees

• Convergence is expected with enough training, but exploration hacking could block it. The sketch needs evidence models are not foregoing high reward from unexplored strategies. Early ideas (novel whitebox techniques expected to be necessary):
  • Train an exploration-predictor model to estimate returns to further exploration.
  • Best-of-n sampling of a trusted model to lower-bound achievable exploration.
  • Interpretability-based formal heuristic explanations (Hilton, 2024) to predict whether convergence reflects equilibrium or exploration failure.
  • Perturbation analysis along the singular learning theory susceptibility direction (Baker et al., 2025; Lau et al., 2024).
  • Model organism studies showing exploration-forcing techniques work.
  • Adversarial RL: train Bob further while holding Alice constant; if Alice’s reward does not drop much, she was not exploration hacking.
• The authors see this as a well-defined problem, in contrast to the fuzzy notions of scheming or goal misgeneralisation.

Step 4: Continue online training during deployment

• Even a converged model may stop playing its best strategy under distribution shift, e.g. via alignment faking.
• Oracle answers and agent actions are subjected to debate (e.g. “is this answer true?”, “is this the right next action?”) and retrained every k steps with the same reward function.
• The final paper from the datacentre gets a stronger check: a “checker” version of the model debates the paper n times, and it is used only if it comes out true in all n iterations. Subtly misleading but technically true claims may escape this; the authors are working on “unexploitable search” ideas.

The safety case structure

• Uses Claims, Arguments, Evidence (CAE) notation (Bloomfield & Netkachova, 2014), following Goemans et al. (2024) and Korbak et al. (2025); presented deterministically for simplicity, though a real case would be probabilistic.
• Top-level: system X (agent A plus oracle) will not cause unacceptable outcomes in deployment context D. Decomposed into: the agent will not (O’), the oracle will not (C0.1, argued separately, with threat models such as persuasion, blackmail, misleading advice and overreliance), and combining them adds no risk (C0X.1, e.g. covert coordination).
• Main objective O’ splits into: A’s error rate is at most ε (C1.1), and A cannot cause unacceptable outcomes at that error rate (C1.2).

Key claim 1: Training has taught the system to play the game well (C2.1)

• Subclaim 1, existence of efficient models (C3.1): efficient models reaching approximate global equilibrium exist. Unbounded debate can answer any question in NEXP given polynomial judges, but that is unrealistic; the sketch uses a relative efficiency notion (the estimator needs only polynomially more compute than the prover). Evidence: a proof (E4.1).
• Subclaim 2, training convergence (C3.2): diminishing returns and roughly reasonable outputs (E4.2).
• Subclaim 3, convergence = equilibrium (C3.3):
  • Training can find equilibrium in efficient time (C4.1); weak evidence is that Nash equilibria of tabular Stackelberg games are computable in linear time, but the game is not tabular, and constants may be large (D5.1).
  • Exploration hacking is the main way equilibrium could be missed (C4.2), a “hunch”; other failure modes include SGD stuck in a local minimum (D5.3) and gradient hacking (D5.2), judged unlikely for a not-extremely-superhuman system.
  • Exploration guarantees rule out exploration hacking (C4.3); a major gap, since methods do not yet exist (E5.3).

Key claim 2: The game incentivises correctness (C2.2)

• Indirect route: instead of ground-truth labels, outputs are checked against a hypothetical truth-finding process M (expanding the whole debate tree with human judges at every leaf), which is too expensive to run; debate aims to accelerate it. Analogy with Go: a single game between strong players is evidence about which moves were best.
• Subclaim 1, M-approximation (C3.4): system outputs in equilibrium equal slightly perturbed outputs of M. Primary failure mode is obfuscated arguments; the case should include a proof the protocol approximates M (E4.3).
• Subclaim 2, truth of M (C3.5), with a side-claim that sufficient red-teaming has found no other issues (C4X.1):
  • Structure of recursive tree debate is conducive to truth (C4.4). Defeaters: some truths are only knowable via heuristics, not deductive argument, especially plausible in AI research (D5.4); some false claims can be defended via gerrymandered definitions across subclaims that leaf-only judges miss (D5.5). Cross-examination avoids this; prover-estimator debate does not.
  • Inputs are accurate (C4.5): human judges have bounded systematic error with respect to reality (C5.1, evidence from trial debates, hard to generalise from verifiable to non-verifiable domains) and the simulation has bounded error with respect to humans (C5.2, primary risk is a scheming judge model, addressed via inability arguments on instrumental reasoning and deception). Getting good human input is expected to be among the hardest parts of the case.
  • Stability (C4.6): small perturbations of M do not substantially change outputs; evidence box left blank. Stability may not be needed for safety, since unstable queries return no answer in prover-estimator debate, only for usefulness.

Key claim 3: Behaviour stays similar during deployment (C2.3)

• Supported by online training (E3.1); evidence should show the regime is correctly configured (update frequency, effective updating of multiple copies).
• This claim breaks in high-stakes contexts, where online training cannot be relied on and guarantees against alignment faking would be needed instead.

Key claim 4: Correct answers most of the time is sufficient (C1.2)

• Main failure mode: scheming to produce harmful outputs specifically where the reward signal cannot catch them, as a coordinated set of actions. Conservatively, every error is assumed to be worst-case (“control flavour”).
• Threat models: research sabotage (C2.4), malicious code attacks (C2.5), self-exfiltration (C2.6), with a side-claim that all relevant threat models are identified via expert threat modelling (E2X.1). Only research sabotage is worked through.
• Subclaim 1, Z bad actions required (C3.6): sabotage (fake data, misleading set-ups, false claims or bugged code) must robustly evade the checker model; threat modelling argues this requires many actions. Defeaters: capabilities improve during online training so Z falls (D4.1); subtle misleading over long periods (D4.2), mitigated by resets.
• Subclaim 2, low likelihood of Z bad actions (C3.7): calculation from distributions over ε, latency k, number of actions n and Z (E4.5). Requires that the protocol is adhered to (D4.3, e.g. external verification, and the agent not disabling online training) and that iterations cannot coordinate, including acausally via a shared underlying model (D4.4).

Extending to high-stakes contexts

• Key claim 4 breaks by definition and key claim 3 breaks because online training is unavailable.
• Value of the sketch anyway: some highly consequential deployments (early AGI doing alignment research under careful conditions) may be low-stakes, and could bootstrap an automated alignment safety case.
• Two hopes for extension: online-training-like techniques applied at inference time to candidate actions rather than final actions (partially patches claim 3); and evidence for claims 1 and 2 (e.g. methods for eliciting bad contexts) may generalise to claims 3 and 4. A theorem bounding the error rate should also raise credence in fuller intent alignment, since human error is plausibly anticorrelated with severity.

Open problems

• Improved debate protocols: avoiding obfuscated arguments, computational efficiency (are polynomial-time debaters constructible?), not requiring stability, avoiding gerrymandering (combining prover-estimator with cross-examination), and avoiding systematic human error (letting debaters block biased queries).
• Empirical testing: prior experiments mostly use QuALITY with low recursive depth; open questions include whether training reaches equilibrium, whether stability and obfuscated arguments are real problems, what counts as an “action” for online learning, and whether the domain is amenable to deductive argument.
• Exploration hacking: the major gap; need model organisms of exploration hacking and, for very superhuman systems, treatment of gradient hacking.
• High-stakes contexts: distribution shifts, alignment faking, coordinated errors invisible to the reward signal.
• From honesty to alignment: how much does honest question-answering plus absence of alignment faking translate into aligned actions, and how to design deployment environments to minimise risk while preserving usefulness.

Conclusion

• The argument is not strong with current techniques; much better techniques are needed before developers can rely on alignment arguments.
• Writing the sketch clarified the authors’ thinking and formed the basis of the alignment section of the UK AI Security Institute’s research agenda.