Abstract
Superintelligence will be more powerful in both upside and downside than any technology humanity has previously faced, and the possibility of existential risk means we cannot simply be reactive. Navigating it will require coordination among leading development efforts, an IAEA-style international authority overseeing efforts above a capability or resource threshold, and the technical capability to make a superintelligence safe.
Framing
- The authors consider it conceivable that within the next ten years, AI systems will exceed expert skill level in most domains and carry out as much productive activity as one of today’s largest corporations.
- Nuclear energy is offered as the standard historical analogue for a technology carrying existential risk; synthetic biology as another.
- Risks from today’s AI must be mitigated too, but superintelligence “will require special treatment and coordination.”
Three starting proposals
1. Coordination among leading development efforts
- The goal is to ensure development occurs in a manner that maintains safety and smooths integration with society.
- Implementation could take multiple forms:
- Major governments setting up a project that many current efforts join
- A collective agreement — backed by a new organisation of the kind proposed below — that the rate of growth in frontier AI capability is limited to a certain rate per year
- Individual companies should separately be held to an extremely high standard of acting responsibly.
2. An “IAEA for superintelligence”
- Any effort above a certain capability or resource (e.g. compute) threshold would be subject to an international authority able to inspect systems, require audits, test compliance with safety standards, and place restrictions on degrees of deployment and levels of security.
- Tracking compute and energy usage could go a long way and gives some hope the idea is actually implementable.
- Suggested sequencing: companies voluntarily begin implementing elements of what such an agency might one day require; then individual countries implement it.
- Scope constraint: such an agency should focus on reducing existential risk and not on issues that should be left to individual countries — such as defining what an AI should be allowed to say.
3. Technical capability to make superintelligence safe
- Framed as an open research question that OpenAI and others are investing heavily in.
What’s explicitly not in scope
- Companies and open-source projects should be able to develop models below a significant capability threshold without the described regulation, including burdensome mechanisms like licences or audits.
- The rationale: today’s systems create tremendous value, and while they carry risks, those risks feel commensurate with other Internet technologies for which society’s likely approaches seem appropriate.
- Applying similar standards to technology far below the bar would water down the focus on the systems of genuine concern.
Public input
- Governance of the most powerful systems, and decisions about their deployment, “must have strong public oversight.”
- People around the world should democratically decide on the bounds and defaults for AI systems. The authors state they don’t yet know how to design such a mechanism but plan to experiment with its development.
- Within those wide bounds, individual users should retain a lot of control over how the AI they use behaves.
Why build it at all
Two stated reasons:
- It will lead to a much better world than currently imaginable — with early examples cited in education, creative work and personal productivity — and the world faces problems requiring much more help to solve.
- Stopping it would be unintuitively risky and difficult. Because the upsides are tremendous, the cost to build decreases each year, the number of actors is rapidly increasing, and it is inherently part of the technological path we are on, stopping it “would require something like a global surveillance regime, and even that isn’t guaranteed to work.”
Acknowledgments: Miles Brundage, Jade Leung, Anna Makanju, John Schulman, Helen Toner, Wojciech Zaremba.