Can We Safely Automate Alignment Research?

Core claim

Safely automating alignment research at the level of top human experts — an “alignment MVP” — is both easier than solving the full alignment problem and enormously helpful for solving it, so it is the right milestone to aim at. There is a real shot at achieving it, but three failure modes must be navigated: inadequate evaluation of AI-produced research (especially research not testable empirically or formally), scheming AIs sabotaging or withholding research, and resource shortfalls in time, data, compute, and effort. Because empirical alignment research is comparatively easy to evaluate, automating it first and then doing a great deal of it is the most promising route to automating the harder, conceptual remainder.

Part 6 of the series “How do we solve the alignment problem?”, following the essay on “AI for AI safety”.

Why automating alignment research matters

• Figuring out how to build superintelligence safely may be a very large amount of difficult work, and short timelines with fast take-offs may leave little time to do it.
• Two feedback loops are in tension:
  • AI capabilities feedback loop: increasingly capable systems accelerate further capability progress.
  • AI safety feedback loop: safe access to increasingly capable systems improves the ability to handle still more capable systems safely.
• The scariest scenarios are those where the capabilities loop takes off while safety labour stays bottlenecked on humans, who are slow, scarce, and (relative to advanced AI) less capable.
• This raises rather than lowers the importance of capability restraint; but large amounts of restraint (e.g. sustained global pauses) may be hard to achieve, so for smaller restraint budgets the automation question is central.

Alignment MVPs

• Following Leike (2022), an alignment MVP is AI labour that safely performs all cognitive tasks involved in alignment research at or above the level of top human experts.
• Compared to the “humans-only path” to aligned superintelligence, the MVP path is the better framing:
  • If the humans-only path is viable, the MVP path is too — building an MVP is no harder than building an aligned superintelligence.
  • Only the MVP path might be viable: humans plus pre-MVP AIs might reach an MVP even if they could not reach aligned superintelligence directly.
  • Even where both are viable, the MVP path is likely faster and more likely to succeed for a given budget of time, compute, and human labour.
• The analogous framing is already taken for granted in capabilities: nobody imagines humans building superintelligence directly, but rather progressively automating capabilities work first.

If neither path is viable

• A possible response is never building superintelligence at all.
• Another is an enhanced human labour path (e.g. whole brain emulation), which may deliver better-than-human research capability while inheriting human-like values by default. Such paths likely require substantial capability restraint, especially under short timelines.
• Even in this scenario, automated alignment research short of a full MVP still matters, because it can produce evidence of danger that motivates the necessary response.

MVPs do not imply “hand-off”

• “Hand-off” means humans no longer play a meaningful role in ensuring the safety or quality of the research — humans as alignment researchers are obsolete.
• An alignment MVP does not require this. Human supervision and evaluation can still contribute, in principle even for significantly superhuman automated researchers.
• Human involvement also need not bottleneck the process: research could run forward with humans evaluating output as produced and intervening only on identifying problems.

Taxonomy of failure modes

• Given an alignment MVP, failure could be because:
  • No amount of top-human-level alignment research suffices in principle (the pessimistic scenario).
  • It sufficed in principle, but not enough was done in practice — insufficient time, compute, staff, money, or leadership attention.
• Without an alignment MVP, either:
  • The AIs were not capable of top-human-level alignment research — but AIs capable of causing loss of control would plausibly also be capable of this, so for some period before loss of control someone likely had capable AIs.
  • Or the AIs were capable but the capability was not safely elicited. Reasons include:
    • Scheming — active sabotage or sandbagging in pursuit of problematic power.
    • Data-scarcity — alignment is a young field with few examples of the target output and few experts to give feedback.
    • Shlep-scarcity — the task-specific effort (scaffolds, interfaces, RL tuning) was not invested.
    • Evaluation failures — the most fundamental barrier, examined at length.
• Real-world failures can combine several of these.

Evaluation failures in non-scheming models

• Good evaluation matters because one can train directly on the evaluation signal, and because even absent direct training it enables empirical iteration toward trusted performance.
• Example failure types:
  • Sycophancy — output plays to the biases and flaws of the evaluation process.
  • Reward-hacking — direct cheating that produces good evaluations (e.g. bypassing unit tests).
  • Cluelessness — no view can be formed about what to think of the output.

Output-focused vs. process-focused evaluation

• For a process P producing output O with a property X of interest, evaluation can focus on O (output-focused) or on P (process-focused).
  • Output-focused evaluation of a string theory textbook: read it and assess the claims.
  • Process-focused: assess the authors’ reputation, peer review, and so on.
• Most real evaluation mixes both, and whether something feels evaluable depends on both.
  • A detective judging Bob’s alibi relies partly on the story’s detail and consistency (output) and partly on the belief that Bob could not fabricate such a story (process). For a superintelligent Bob, the output-focused evidence weakens.
  • A manager’s code review may catch innocent bugs but not intentionally hidden ones — unless it can also be established that the employee is not hiding bugs.
• Discussions of “the difficulty of evaluating alignment research” usually mean output-focused evaluation, and often assume robustness to superintelligent schemers. The relevant AIs here need only match top human experts and need not be schemers.

Human output-focused evaluation

• The baseline method is humans evaluating research directly, as they would a colleague’s work. If the target is merely top-human-level performance, this may be close to sufficient in principle — human researchers already evaluate their own and each other’s work.
• Two complications:
  • With human colleagues, one usually also has process-focused evidence that they are trying their best rather than engaging in sycophancy or reward-hacking. That evidence is harder to obtain for AIs, making output-focused evaluation correspondingly more load-bearing.
  • Humans disagree about which existing alignment research is good, which raises the question of “top-human-expert research according to whom?”.

Scalable oversight

Techniques for amplifying human evaluation ability with AI:

• Imitation — train AIs on human judgments and substitute them for humans, gaining speed and scale.
• Decomposition — break the evaluation into smaller tasks for humans/AIs, eventually including learning to decompose well.
• Debate — have AIs argue evaluation-relevant questions before human judges (or AIs trained on human judgments).
• Constitutions — distil the evaluation process into explicit principles, specs, or algorithms that AIs are trained or instructed to follow.
• Expensive versions — large AI teams running for long periods, or more/better human labour, for high-value evaluations.
• Distillation — train AIs to imitate the output of any expensive combination of the above, then use them in a new evaluation process.

Limits: for sufficiently difficult tasks, decomposition and debate may not travel far — a large team of average M-Turkers seems poorly placed to evaluate novel string theory research even with sub-tasks and AI debate. But the target here is top-human-level research, not strongly superhuman research, which is substantially easier.

Process-focused techniques

Process-focused evaluation amounts to understanding generalization: how AIs will move from domains where output-focused evaluation is strong to domains where it is weak.

• Behavioral science of generalization — study AI behaviour in depth to predict generalization.
  • “Weak-to-strong” generalization is the paradigm case: if difficulty levels 1 and 2 are evaluable but level 3 is not, train on level 1, test generalization to level 2, and treat that as evidence about level 3.
  • Direct imitation of a specific top researcher is another possible route: if training on enough of Bob’s output yields robustly Bob-level behaviour even in unevaluable domains, that is a fast path to an MVP. Whether this works in principle or given realistic data constraints is open but empirically tractable.
  • This does not work against AIs systematically manipulating the behavioural evidence.
• Transparency — methods aimed at internal factors like thoughts and motivations.
  • In the limit, tools showing what an AI is trying to do would give strong evidence of good intent even without evaluable output.
  • Weaker forms, such as ruling out active deception, still help.
• Detecting and preventing scheming is therefore an especially important component of process-focused evaluation, drawing on both behavioural science and transparency.

Comparisons with other domains

Grounds for comfort

• The need for good evaluation is not unique to alignment. Reward-hacking already appears on coding tasks central to capabilities research, and sycophancy is already a general problem.
• Good evaluation, including in fuzzy domains, is crucial to automating a great deal of economically valuable work — product design, business strategy, cancer research. Long-run quantitative metrics exist for some of these, but training on very long-horizon reward signals has its own difficulties (serial time, data scarcity).
• If evaluation difficulties in other STEM domains are comparable to those in alignment, strong economic incentives should produce transferable solutions, and the ability to automate those fields would be grounds for optimism about alignment.
• Alignment and capabilities research have always been tied together, and especially so for eliciting intended performance from non-schemers — scalable oversight is central to capabilities work too.

Limits to that comfort

• Evaluation failures elsewhere are lower-stakes: a failed training run can be retried, whereas a critical flaw in a safety case meant to prevent catastrophic behaviour may not allow a second attempt.
• Other domains may be solved too late: most of an intelligence explosion could proceed with effort concentrated on capabilities R&D, and the eventual route to unevaluable domains may run through training on crisp metrics until general capabilities transfer — a path requiring more powerful, more scheming-prone systems.
• Alignment research may be especially hard to evaluate even relative to these domains.

Three kinds of domain

1. Number-go-up — evaluation asks whether a crisp quantitative metric improved (e.g. cross-entropy loss on internet text). Capabilities research benefits heavily from this. Some alignment work has this character too: metrics for helpfulness/harmlessness/honesty, rates of reward-hacking, alignment faking, sycophancy, chain-of-thought unfaithfulness, degree of weak-to-strong generalization.
2. Normal science — evaluation asks fuzzier questions: was this a well-conducted experiment, how should the results be interpreted, does the interpretation fit other data, what predictions follow, what experiment comes next. The human track record here is fairly strong, owing to two factors:
   • Empirical feedback loops — rich contact with a manipulable phenomenon, constraining and refining ideas against data.
   • Formal evaluation methods — the standards of maths and formal logic; even unaided human evaluation does decently on “is this maths correct?”.
   • Scientific fields seem weaker where these are absent: limited experimental access to the brain in neuroscience, untestable theoretical physics debates, economists’ inability to run controlled experiments.
3. Conceptual research — evaluation can rely on neither empirical feedback loops nor formal methods; the central tool is thinking and arguing about it, without ever seeing the answer. Paradigm cases: philosophy, futurism, some political debate.
   • Such work can still draw on empirical data, and there may be a ground truth (as in futurism), but it is not accessible when the evaluation is needed.
   • The category is defined by available evaluation methods, not by the cognitive skills needed to produce the work: Einstein on relativity was a conceptual breakthrough but was evaluable empirically and formally, so it does not count.

Where alignment research falls

• Much alignment research — empirical alignment research — resembles normal science. Illustrative examples cited: Greenblatt et al. (2024) on inducing alignment faking; Betley et al. (2025) on emergent misalignment from insecure-code fine-tuning; Lindsey et al. (2025) on internal circuits and causal interventions; Baker et al. (2025) on chain-of-thought training hiding rather than removing reward-hacking; Burns et al. (2023) on weak-to-strong generalization; Farquhar et al. (2025) on myopic training reducing reward-hacking; Marks et al. (2025) on auditing for hidden objectives.
• Empirical alignment research may in fact be an unusually empowered empirical discipline: strong experimental access to both model behaviour and internals, experiments running at computer speed rather than being bottlenecked by physical processes, and often lower compute requirements than capabilities research (though this could change if avoiding scheming requires experiments on pre-training).
• Some alignment research is centrally conceptual: developing and critiquing not-yet-testable approaches, theorising agency and intelligence, formalising alignment-relevant concepts, articulating threat models, constructing strategies and safety cases. Examples cited: Christiano et al. (2018) on iterated amplification; Irving et al. (2018) on debate; Leike et al. (2018) on recursive reward modelling; Kenton et al. (2022) and Garrabrant et al. (2021) on formalising agency; Christiano et al. (2024) on backdoor defendability; Garrabrant et al. (2016) on logical uncertainty; Bostrom (2014); Hubinger et al. (2019) on risks from learned optimization; Christiano et al. (2021) on eliciting latent knowledge; Shah et al. (2025) on DeepMind’s AGI safety approach.
• Conceptual work sometimes becomes empirically testable later, but counts as conceptual insofar as empirical feedback is unavailable at the time evaluation is needed.
• It is not settled that automating conceptual research is harder in practice: frontier AIs already seem decent at philosophy, and conceptual work is not bottlenecked on real-world experiment. Still, evaluation failures are most concerning here.

How much conceptual alignment research is needed?

• For building superintelligence safely: probably a substantial amount. Along the full path, it will likely matter to develop and evaluate hypotheses, reconceptualizations, research agendas, safety cases, threat models, and high-level strategies that cannot be immediately tested empirically or formally.
• For building an alignment MVP: cautious optimism that major non-empirically-evaluable conceptual advances are not required, and that the main need is a great deal of empirical alignment research of the kind already being done.
  • This is not circular: one need not do a lot of physics to build an AI that automates physics research.
  • Developing an adequate safety case for an MVP is partly conceptual — stitching diverse empirical evidence into an accurate risk assessment — but the question is whether it needs much additional conceptual work requiring heavy AI help.
  • Conceptual work remains helpful regardless: setting strategy, prioritising empirical projects, strengthening safety cases.

Empirical alignment research as a lever on conceptual research

Two claims combine into the essay’s central strategic proposal:

1. Automating empirical alignment research at top-human levels warrants comparative optimism, given its empirical feedback loops and its similarity to capabilities research and other empirical science.
2. Given that automation, a great deal of it can be done to improve the ability to automate conceptual alignment research.

On scalable oversight

• Run large numbers of experiments testing and refining scalable oversight approaches, including ones humans have not yet conceived but which become empirically checkable once proposed.
• Where ground truth or trusted top-human-level results exist in some domain, test whether oversight techniques reproduce them: assign one AI to argue the true answer and another the false one and see whether debate moves an ignorant judge correctly; or hold results out of training data and see whether non-specialist humans can use a capable AI to reproduce them (e.g. string theory, niche cryptography, maths).
• Harder in the closest analogues of conceptual alignment research, but options remain:
  • Analytic philosophy has reasonable consensus on what counts as good work, forceful objections, and novel arguments, even absent consensus on big questions — so narrower questions can be used.
  • Futurism has ground truth about what has already happened; holding that out of training data allows testing forecasting ability.
  • Deliberately introduce subtle mistakes into conceptual work (e.g. bad philosophical arguments) and test whether humans who cannot initially detect them can do so via scalable oversight.
  • Construct synthetic domains as new ground truth — e.g. a scaled-up simulated civilization as ground truth for a toy futurism, or for toy policy debates about intervention effects.
• Generating such empirically testable ideas is itself a form of empirical alignment research, so automating it yields more ideas of this kind.

On process-focused methods

• Behavioral science of generalization: build large suites of automated behavioural tests showing model behaviour across wide arrays of counterfactual circumstances, use AI labour to find patterns and generate higher-level explanatory principles, and study how behaviour shifts under different training — extremely intensive empirical “psychology” with a far faster and more powerful apparatus than human psychologists possess. Weak-to-strong generalization work is especially important.
• Transparency: large volumes of top-human-level interpretability research would plausibly help a great deal — scaled-up mechanistic circuit analysis, top-down analysis, and potentially far more ambitious goals. Beyond looking inside black boxes, transparency-focused work includes chain-of-thought faithfulness research and the broader project of open agency: arranging possibly-still-black-box components into agential processes that permit epistemic access to safety-relevant planning and motivation.
• Anti-scheming measures combine transparency and behavioural science in the context of model organisms and experiments isolating components of the scheming threat model (situational awareness, beyond-episode goals).

Other benefits

• Better evaluation methods also help automate further empirical alignment research.
• Empirical work supports restricting rogue options and designing incentives that reward cooperation.
• Even if a full MVP is unattainable, empirical alignment research can surface evidence of danger feeding into risk evaluation and capability restraint.

Scheming

Reintroducing scheming alters the picture in four ways:

• Comparisons with other domains become less informative — schemers may deliberately make some domains harder than others.
• Higher evaluation standards are needed — robustness to intentional sabotage motivated by long-term power-seeking differs from robustness to myopic sycophancy or reward-hacking. Sabotage can be optimised against detection over long timescales and across contexts, and failing to catch it is higher-stakes. The human track record in physics, biology, and so on has not had to be robust to sustained active sabotage.
• Sandbagging — even an evaluation process that can distinguish good from bad research may face AIs intentionally withholding work they are capable of. Whether they succeed turns on the “no sandbagging on checkable tasks” hypothesis; countermeasures exist but this is an additional challenge on top of making the research checkable.
• Dangerous rogue options — beyond sabotage and sandbagging, threat vectors like self-exfiltration and manipulating humans must be closed off.

Will such AIs be schemers by default?

Plausibly not, especially under short timelines:

• Current models are unlikely to be actively scheming, and under short timelines the first models capable of automating most top-human-level knowledge work would likely resemble current ones in training techniques, architectures, and scaffolds.
• Scheming requires situationally-aware agentic planners with long-term consequentialist goals, and automating alignment research may not require AIs of that kind:
  • The field aims at a long-term goal, but individual systems need not optimise for it — they might pursue shorter-term targets (process-based supervision, myopic optimization with non-myopic approval).
  • Components of dangerous agency need not come bundled; teasing them apart may permit more safety.
  • It is unclear that alignment research demands strong situational awareness.

If they would be schemers, can it be detected and prevented?

• One view encountered is “if science then scheming”: any AI doing serious top-human-level science would be a scheming, situationally-aware, long-horizon planner — perhaps because such research pushes it past a threshold of self-reflection at which it asks what it “really wants” and moves to disempower humanity.
• These arguments are not analysed in depth, but they make strong predictions about what science non-schemers can do. Meanwhile, tractable research directions for detecting and preventing scheming exist, and automated labour short of a full MVP should be useful for anti-scheming research.

Can safe research be elicited from schemers?

• The prospect should be alarming; the first response to heading there should be to look for a different plan, e.g. capability restraint.
• It is nonetheless an option. It requires: (a) evaluation robust enough to verify work is top-human-level and unsabotaged; (b) anti-sandbagging techniques strong enough to actually elicit it; (c) option control closing off other rogue options. Together these constitute an “AI for AI safety sweet spot”.
• It should be temporary if attempted at all, with the core aim being to elicit enough alignment research to rule out scheming going forward.
• Viability depends heavily on strong output-focused methods, since process-focused evaluation is not robust to scheming. Hence more optimism about eliciting safe empirical or formally evaluable work from schemers than conceptual work — which, given the leverage of empirical work, may still help a lot.

Three routes past scheming

1. Scheming does not arise by default in early models capable of automating alignment research.
2. It arises but is detected and prevented, including using AI labour short of a full MVP.
3. Output-focused evaluation, anti-sandbagging measures, and option control suffice to elicit safe top-human-level research even from schemers.

These can combine — some research done by non-agentic AIs, some by maybe-schemers in domains where output-focused evaluation inspires enough confidence.

Resource problems

• Setting aside the “no amount suffices” branch and assuming someone eventually had capable AIs, the remaining failure branches reduce to inadequate resources:
  • After getting an MVP — insufficient time before an intelligence explosion leads to catastrophe, or insufficient compute, staff-time, and leadership attention.
  • While trying to get an MVP — insufficient high-quality data to elicit top-human-level work, or insufficient shlep.
• How early an MVP arrives matters a great deal to how much automated alignment research can be done overall.
• These practical failure modes are extremely serious and admit no clever response: the shlep must actually be done, the investment actually made, data-scarcity actually worked around (though this barrier could prove more fundamental), and time actually created.
• Capability restraint remains extremely important even given successful automation: frantically doing large volumes of automated alignment research amid an intelligence explosion is very dangerous, and extra time remains very valuable.

Alternatives to automating alignment research

• The problem is easy — small amounts of slow, scarce human labour suffice.
• Long timelines by default — time exists for large amounts of human labour, or for a transition to enhanced human labour. This and the previous option could interact, but neither can be counted on, and insofar as they are driven by technical parameters neither is under our control.
• Sustained global pauses or slow-downs — actively buying time, potentially enough for a slow transition to enhanced human labour that itself works slowly. Giving up on superintelligence entirely is the limiting case.
• AI for whole brain emulation — using AI to accelerate development of the one form of enhanced human labour that escapes biology’s speed and scalability constraints. This may rely less on very long pauses but still requires significant capability restraint.

None of these looks more comforting than trying to make automated alignment research work.

Conclusion

• There is a real shot at safely automating alignment research, and doing so is extremely important, but three failure modes must be navigated: evaluation (especially of research not testable empirically or formally), scheming (either avoided/prevented, or worked around via safe elicitation), and resources (time, data, compute, staff, effort).
• Failure on any front is worryingly plausible, but much can be done to improve the odds — centrally, automating the empirically testable alignment research first and doing a great deal of it, alongside anti-scheming measures and gathering evidence that motivates investment and capability restraint.
• Since the most salient alternatives look no more comforting, the conclusion is to try extremely hard to make automated alignment research work.

Appendices

Appendix 1: Application to other forms of AI for AI safety

• Evaluation failures are less worrying in many other safety-relevant domains (forecasting, cybersecurity, on-chip compute monitoring), since conceptual alignment research is unusually hard to evaluate.
• Scheming concerns apply similarly, except that stronger evaluation capacity elsewhere makes sandbagging and sabotage harder to get away with, and the specific rogue options vary — using AIs in a military context to enforce capability restraint could be unusually dangerous.
• Data-scarcity is especially pressing for alignment given the field’s youth; inadequate time, investment, and shlep apply broadly.

Appendix 2: Other practical concerns

• Cover — the idea provides cover for pushing dangerous capability development. Response: possible, but evaluation of the idea should be separated from concern about its memetic effects; if it is viable, say so and mitigate misuse otherwise.
• Complacency — “passing the buck to the AIs” obscures how much work humans must still do. Response: a real concern; detailed models of what responsible buck-passing looks like are needed.
• Capabilities externalities — safety-motivated automation accelerates capabilities. Response: such efforts can focus on using AI labour once available rather than accelerating it; accelerating especially safety-relevant applications is often still worth it where overlap with general capabilities is minimal; and given the massive resources already devoted to frontier capabilities, the marginal contribution of safety-motivated work is likely small.
• Distraction from capability restraint — Response: AI for AI safety warrants at least a substantial chunk of effort, so it is not a distraction.
• Premature effort — automating too early wastes time relative to waiting for another capability notch. Response: a real thing to watch for.
• Harmful delegation — delegation could degrade human understanding and control, increasing loss-of-control risk. Response: some early-adoption costs may need to be eaten to benefit from AI labour early, and if the AIs behave broadly as intended the direct safety benefits likely outweigh.

Appendix 3: Arguments for the inadequacy of empirical alignment research

• Cannot safely test genuine takeover options — the most important test cannot be run. Response: agreed, but this only undermines empirical research if empirical research never supports predictions about unseen data points; one can be confident Bob will not use a butcher knife before handing him one.
• Immediate takeover options — a superintelligence has takeover options as soon as it exists, so no empirical research is safe, not even boxing it for interpretability. Response: if nothing is being done with it except scanning internals, boxing should be much easier.
• Subverting empirical tests — superintelligences can fake alignment or manipulate their internals against interpretability. Response: a crucial issue, but empirical research can help a lot with it.
• New capabilities introduce new problems — research on weaker systems is insufficient. Response: agreed, but if superintelligences do not immediately have takeover options, they can themselves be studied.
• Detection but not cure — methods may reveal unsafety without fixing it, and iterating until detection stops firing may merely hide the problem. Response: not always; the selection pressures matter, and training on a detection method differs importantly from using it as validation. Empirical research is not limited to naive trial-and-error iteration.
• Needing extreme understanding — aligning a superintelligence requires understanding it as thoroughly as rocket engineers understand rockets, which requires conceptual/philosophical progress. Response: deep understanding is likely needed, but it need not be humans who possess it. Conceptual progress evaluable via empirical tests should be distinguished from progress that is not; relativity was conceptual but empirically evaluable. How much non-empirically-evaluable conceptual research went into rockets or nuclear reactors?
• Ethics in particular — empirical research cannot deliver ethical progress. Response: solving alignment requires AIs that can do ethics and philosophy as well as humans and are motivated to do so in human-like ways, but this may not require much progress in our current understanding of ethics — just as automating physics does not require progress in physics.

Appendix 4: Do AIs doing alignment research engage with too many dangerous topics?

• Alignment research does implicate a wider and more dangerous array of topics (human psychology, game theory, computer security, galaxy-brained considerations) than narrower scientific work, which counts in favour of plans that automate narrower research instead.
• But such plans have serious downsides: they generally require very significant capability restraint, and — except for whole brain emulation — the labour they unlock lacks AI advantages like speed, copying, and read-write access.
• Also, hoping for AIs never exposed to these topics or to the alignment discourse would require a substantial departure from standard pre-training on internet text. The strong default is that frontier AIs have been exposed to them anyway; the marginal risk of engaging them further seems worth it.