An Alignment Safety Case (short presentation)

Introduction

1. Research by UK AISI. That’s what your tax money is used for, Brits.
2. Today we train AIs by having people rate its answers (good answer - thumbs up, bad answer - thumbs down), i.e. Reinforcement Learning from Human Feedback.
3. That only works if people can tell good answers from the bad ones. If an AI becomes smarter than humans, humans lose the ability to properly grade it (and the AI can just learn to sound convincing) - scalable oversight problem.
4. So how can we make sure that a superintelligent AI is trustworthy? By having it play a game with another copy of the exact same superintelligent AI.

The idea itself

1. Most of us probably can’t check a maths proof but we can watch the experts argue about it and see which one of them runs out of arguments first.
2. So how about we do the same with superintelligent AIs?
3. Safety case - a written argument that a system is safe enough for a specific use. Let’s have one superintelligent AI argue for the case, and the other against it - a “debate game”.
4. Whichever side runs out of arguments first - loses. A human is a judge (and decides, who won).
5. The whole experiment should be run in low stakes - i.e. a situation in which a single decision cannot cause irreversible damage (so if an AI starts misbehaving, we have time to shut the whole thing down). Probably some isolated environment, in this case.

Important elements

1. Both AIs need to really play the game well. They can’t hold back winning arguments. Authors want “exploration guarantees”. An AI cannot act like a student who gives the teacher a wrong answer so that they don’t move on to harder material.
2. The debate needs to be somehow immune to obfuscated arguments. Basically, neither side should be able to lie its way to the victory by presenting a false but impossible to refute argument.
3. Human judges must reliably reward correct answers.
4. AIs probably cannot be vastly superhuman - just slightly superintelligent.

The gaps

1. It’s not possible to do with the techniques we currently have.
2. No one really knows how to prove that the AI isn’t sandbagging.
3. The debate format that is immune to lies is also a WIP.
4. It really needs to be low-stakes. In high-stakes, a single mistake can lead to a catastrophe.

Conclusion

1. There’s a list (above) of research problems to solve. Apparently they’re in UK AISI’s official research agenda now.
2. Playing games is cool and all of you should sign up for playing mafia tonight.