Radical Optionality: Governing Transformative AI Under Uncertainty

Abstract

The apparent dilemma between overregulating transformative AI and failing to regulate it rests on a false premise: there is a class of policies that meaningfully increases safety without imposing significant costs on innovation. Governments should implement these aggressively — preserving democratic governments’ ability to make good decisions as circumstances evolve by avoiding overregulation now while rapidly building the institutions, information channels, and legal authorities needed to respond competently to a wide range of scenarios.

Publication details

• Full-length essay by Christoph Winter (Assistant Professor of Law and AI, University of Cambridge) and Charlie Bullock, both affiliated with the Institute for Law & AI; published at radical-optionality.ai. A condensed guest commentary by Bullock appeared in AI Frontiers on 6 July 2026.
• The argument rests on four stated assumptions:
  1. A real possibility of transformative AI — “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution” — within the next ten years.
  2. Profound uncertainty about what capabilities such systems will have, what benefits and risks they will generate, and how to capture the former while mitigating the latter.
  3. A transformative dual-use technology with national security implications will inevitably require some government oversight.
  4. Building the institutional capacity for that oversight takes years, so society cannot wait until transformative capabilities exist.
• The first assumption is explicitly not defended here; the essay’s claim is conditional on accepting it.

Why regulating transformative AI is unusually hard

• The pacing problem: technology outruns the legal system’s ability to adapt.
• Compounding factors specific to AI: systems are hard to understand, may possess capabilities their creators are initially unaware of, and have jagged capability profiles — expert-level on one task, failing at much simpler tasks elsewhere — so no benchmark reliably indicates overall competence.
• Recursive self-improvement is already beginning: engineers at some frontier labs report that a significant majority of the code for next-generation systems is written by current-generation systems.
• Humans and institutions are psychologically poor at exponentials. Examples given: epidemiologists dismissing COVID-19 as less prevalent than flu in early 2020, and the IEA systematically underestimating solar growth for over a decade while the industry maintained ~25% annual growth.

The two rival approaches

”Let the market handle it”

• The permissionless-innovation case (Adam Thierer; Draghi’s competitiveness review in the EU) has real force: governments have often regulated emerging technologies badly. Overregulation of nuclear energy is offered as the cautionary case — “hundreds of thousands of lives have likely been lost prematurely due to air pollution that could have been prevented” by cost-effective nuclear power.
• But the authors argue this position tacitly depends on skepticism about AI’s trajectory. The nuclear analogy cuts both ways: however sound the objections to overregulating power plants, “every private individual should be allowed to buy as many nuclear weapons as they want” is indefensible.
• A truly transformative general-purpose system would have military applications potentially as significant as fission, so a fully laissez-faire approach is neither practically nor politically feasible.
• The key asymmetry: nuclear weapons unquestionably existed; transformative AI is still only a possibility, so rigid detailed rules now would be foolish. But once such a technology is actually under development, all stakeholders share an interest in regulation being competently designed and enforced — and insufficient regulatory capacity risks “hamfisted and overly harsh regulation down the line,” or incompetent regulation that harms industry without helping the public.

Anticipatory governance and the precautionary principle

• The hard precautionary principle — prohibit anything not proven safe — would bar transformative AI indefinitely and is “simply bad policy,” because it ignores that regulation may itself cause more expected harm than the risks it addresses.
• Softer formulations exist: the EU’s version incorporates cost-benefit analysis counting both the costs of regulating and of failing to regulate, as invoked in the GPAI Code of Practice. This version may be compatible with radical optionality.
• The infinite-catastrophe argument (a low probability of an infinite cost outweighs a high probability of a large finite benefit) is called theoretically sound but without obvious implications: restricting innovation in liberal democracies may not lower long-term catastrophic risk given continued research in authoritarian states, and AI’s potential benefits may be no less “infinite” — powerful systems could prevent a nuclear war or pandemic, and a wealthier, more intelligent society may be more willing and able to avert catastrophe, making growth possibly anti-correlated with catastrophic risk.
• Both sides are ultimately using heuristics. The precautionary heuristic treats change as harmful until proven harmless; the authors prefer the heuristic that new technologies have historically (though not invariably) produced net long-term benefit — with a footnote conceding that this has sometimes required government intervention, as with the Montreal Protocol and the ozone layer.
• Against anticipatory governance: governments are historically terrible at predicting technological trajectories. The hands-off approach to the early Internet has aged better than the Audio Home Recording Act of 1992, rendered mostly obsolete almost immediately by the personal computer. Regulatory regimes are also sticky and path-dependent, so course correction is costly. And institutional lead times are long: “If capacity-building begins only once risks and benefits are unmistakable, the decisive window for proportionate action will likely have closed.”
• The authors concede some anticipation is unavoidable — their own proposal rests on predictions — and describe radical optionality as “a formula for minimizing uncertainty, not for eliminating it.”
• A closing argument aimed at those still favouring restriction: the recommended policies are simply more politically realistic. California’s SB 53, New York’s RAISE Act, and the EU GPAI Code of Practice exist; an international treaty banning frontier development or a datacenter moratorium has no near-term chance. Even a reader who wants more should support these.

What makes it “radical”

• The proposal is to build strong regulatory institutions, equip them with flexible authorities, and ensure they can access needed information — measures whose costs are “measurable in taxpayer dollars and political capital” rather than in burdens on AI companies.
• The radicalism lies in the scale of spending justified. If a transition at least as significant as the Industrial Revolution may occur within years or decades, then “even if there’s a 95% chance that the money spent on a given policy measure is wasted, a five percent chance of some positive impact… would mean that the costs were justified a thousand times over in expectation.” Governments should worry about counterproductive interventions, not pecuniary cost.
• Distinguished from “muddling through” (Lindblom): both distrust grand regulatory regimes built on present-day guesses, but incrementalism preserves flexibility passively, through inaction and delay. Radical optionality requires proactive investment — not enough to lock in a regulatory future, but enough to choose well when choice becomes necessary.
• Robust across worldviews: whether one expects Aschenbrenner’s “Situational Awareness” scenario (a government-run AGI Manhattan Project around 2027–28) or prefers Buterin’s d/acc defensive acceleration, the right move now is the same. If Aschenbrenner is right, a government that must “throw together the most complex public project in human history at the last minute” will do better with information, personnel, and flexible mechanisms already in place. If one wants to avoid a centralised, securitised future, that is likelier when governments have good options other than emergency action.

Concrete policy proposals

Information-gathering: transparency and reporting

• Two categories: transparency requirements (publish publicly — allowing academics, civil society, and industry to analyse, but risking exposure of trade secrets) and reporting requirements (share with an agency — enabling more detailed and sensitive collection). They are complementary.
• Most enacted AI safety legislation is already of this type: the EU GPAI Code of Practice requires a detailed safety and security framework plus a safety and security model report; SB 53 and RAISE require publication of a safety and security protocol essentially formalising documents companies already produce voluntarily, and demand considerably less than the Code of Practice.
• Division of labour: state transparency requirements are a decent substitute for a federal transparency framework provided they are harmonised to avoid a costly patchwork, but reporting is better done federally, since states generally lack capacity to securely process detailed information that would be of limited value to them individually.
• The abandoned BIS reporting rule (issued under the Biden EO, later revoked) would have required frontier developers to report internal safety evaluation results and weight-security measures. The authors note the concept drew no objection from affected companies during notice and comment — OpenAI and Anthropic suggested changes to frequency and emphasised secure handling — and that conservative objections targeted the Defense Production Act legal basis rather than the policy. The total burden would have amounted to “approximately five companies each had to send one email to BIS once every few months.” Replacing it would operationalise the White House AI Action Plan’s own information-sharing goals.
• Next step: an auditing regime letting third-party or government auditors verify both the adequacy of a company’s risk policies and its compliance with them. Somewhat heavier than transparency or reporting, but still light — and some leading AI companies have affirmatively requested it.

Whistleblower protections

• Impose virtually no positive obligations on companies; the marginal cost is defending occasional frivolous suits. “‘Don’t fire or punish an employee for telling the government about a serious, specific risk that your research creates’ is a very reasonable ask.”
• Current coverage is thin: California law protects reporting of legal violations generally, and SB 53 adds protection for catastrophic-risk disclosures but only for employees “responsible for assessing, managing, or addressing” such risks.
• The gap is disclosures about serious dangers where no law has been broken — precedent exists in the Federal Railroad Safety Act’s protection for reporting any “hazardous safety or security condition.” The illustrative case: reporting on Claude Mythos Preview’s cyberdefense capabilities, which identified “thousands of zero-day vulnerabilities, many of them critical” — releasing a comparably capable model publicly without warning would likely not have been illegal but could have caused significant harm.
• Endorses the AI Whistleblower Protection Act (Sen. Grassley, bipartisan), covering disclosures of “substantial and specific” dangers to public health, safety, or national security.
• In the EU, protection extends to AI Act violations from August 2026, but not to public-safety dangers absent an EU law violation; since most frontier companies are U.S.-headquartered, Europe’s information access will be more limited regardless.

Information-sharing

• Within and between governments, and with outside stakeholders. Deregulatory measures can help — antitrust safe harbors or guidance clarifying that labs sharing sanctioned safety-relevant information through the Frontier Model Forum does not violate antitrust law.
• Internationally: the UK AISI sharing pre-deployment evaluations with CAISI, or joint evaluations, as the model. “Wholesale objections to any degree of international cooperation simply don’t make sense unless they’re based in skepticism about the possibility of transformative AI.” EU AI Act Article 78(5) authorises confidential exchange with third-country regulators under adequate confidentiality arrangements. Even Situational Awareness calls for a “tighter alliance of democracies,” and Amodei’s “Machines of Loving Grace” proposes an “entente strategy.”
• Within government, cross-agency coordination frameworks cannot be assembled on short notice — and the same unpredictability that makes governments bad at regulating emerging technologies means sudden intervention cannot be ruled out. “The better informed the government is, the less heavy-handed its response in such emergency situations will have to be.”

Flexible rules and definitions

• Premature rigid rules invite misspecification. Options: if-then commitments triggered by specified conditions, or management-based regulation requiring mitigation while leaving companies discretion over method.
• Definitions are the sharp case. Compute-threshold definitions of “frontier model” go stale as low-compute models become more capable and compute costs fall, so the task is generally better left to an agency that can update a regulatory definition.
• SB 1047 as illustration: its $100 million training-cost floor made sense in summer 2024 but would have been rendered largely obsolete months later, in January 2025, by DeepSeek, which reportedly spent under $6 million on the final training run for the base model behind DeepSeek-R1. Amending a statutory threshold requires a new bill; a regulatory definition could have been updated quickly. Earlier versions of SB 1047 contemplated exactly such a regulatory definition — the cost threshold was likely added to reassure startups.
• Tradeoffs acknowledged: giving agencies unsupervised control over questions of immense economic and political significance is constitutionally fraught and less democratically accountable. Partial answers: limit the scope of delegated authority and ensure congressional and White House oversight.
• EU AI Act as a second case study: praised for updating mechanisms and review requirements, but while it “delegates substantial authority to amend annexes and procedural regimes, … core definitional frameworks remain frozen.” Its choice to regulate models rather than companies or uses will be hard to revisit.

Assessments and evaluations

• Valuable both instrumentally and for the expertise agencies build in conducting evaluations and sharing results securely. Governments should also subsidise a third-party evaluations ecosystem — organisations like METR and Apollo Research working closely with labs.
• Notably consensual: OpenAI has proposed expanding CAISI’s role in evaluations and standard-setting, Anthropic has made similar suggestions, and Trump’s AI Action Plan devotes a section to “Build[ing] an AI Evaluations Ecosystem.” Yet legislative vehicles like the Artificial Intelligence Risk Evaluation Act (Hawley and Blumenthal) have been treated as messaging bills. First step in the U.S.: codify and fund CAISI with an expanded mandate.

Securing model weights and algorithmic secrets

• Aschenbrenner concludes lab security is currently too poor to protect weights and algorithmic secrets against a serious sustained state effort.
• The optionality link is direct: a lab with secure weights can delay deployment of a model showing nationally significant capabilities; if a foreign power already has the weights, “the capabilities in question will already be theirs to do with as they please.”
• Recommendations: comprehensive voluntary physical and cybersecurity standards across the frontier development supply chain, with compliance made a condition of federal grants and contracts (the DoD’s CMMC program as the template). Government holds decades of unmatched expertise in classification and clearance systems, plus intelligence about adversary objectives that may need to be shared with labs.

Hiring and talent

• “The most important factor in building governmental capacity for AI governance is access to top-tier talent” — and governments should deliberately acquire more elite talent than current demands require, since a deep reserve is itself optionality.
• Obstacles: private compensation exceeding government salaries by orders of magnitude, outdated U.S. hiring processes, and a general lack of urgency.
• Even the UK AISI, funded roughly ten times its U.S. counterpart and treating frontier-lab hiring as a KPI under both Sunak and Starmer, advertises £65,000–£145,000 (with No. 10’s aggressive tech recruitment reaching £200,000) — “a mere fraction” of private-sector packages.
• Beyond funding: new hiring and contracting authorities. In the U.S., a “reserve corps” of private-sector experts callable in an emergency, or reform of the Intergovernmental Personnel Act. In the EU, reducing bureaucratic delay, resisting pressure for proportional national representation, and making fuller use of the AI Act’s Scientific Panel of Independent Experts.

Avoiding premature and overbroad preemption

• The authors accept that frontier regulation should ultimately be federal and that some preemption will be necessary; the dispute is over when and how.
• The June 2025 reconciliation-bill moratorium — barring state enforcement of any AI law for ten years, later narrowed to five years with exemptions and tied to broadband funding — is judged ill-advised. With Congress passing only 27 bills into law in 2023, a broad preemption bill would be unlikely to be reversed and unlikely to be followed by federal legislation: “preempting state AI legislation and replacing it with nothing radically reduces the available regulatory option space.”
• Concerns about a burdensome state patchwork are called reasonable but “almost purely hypothetical,” given that frontier development in the U.S. is currently near-unregulated.
• The recommended sequence: react rather than predict — wait to see which state laws prove burdensome, then preempt those. Preemption is usually packaged with affirmative federal law (e.g. federal transparency requirements preempting state ones in the same bill), and authorising agencies to regulate confers regulatory preemption automatically. A purely deregulatory preemption bill remains available, on the model of the Airline Deregulation Act of 1978.
• Historical point: narrow, iterative, post-hoc preemption “is the only approach to preemption of state laws regulating an emerging technology that has ever been taken in the history of the United States.” The 2025 moratorium’s approach was unprecedented. Trying to allocate regulatory responsibility before the problem’s contours are clear “is like trying to put together a jigsaw puzzle while blindfolded.”

Objections and responses

• “Giving the government a hammer.” Conceded as legitimate — no promise that new authorities won’t be abused. But hamstringing regulators to prevent overregulation is short-sighted; the answer is meaningful congressional/White House and parliamentary oversight. The specific interventions proposed are “common-sense ways of increasing regulatory capacity,” not weighty substantive authorities, though onerous reporting or overbroad whistleblower protections (facilitating trade secret theft) could still cause harm if designed badly.
• “Democratic legitimacy.” Flexibility and legitimacy genuinely trade off — waiving notice and comment speeds response but removes public input; delegation speeds updating but faces recent Supreme Court limits. The authors argue legitimacy matters more here, since powerful AI threatens to “reshape the delicate balance between state capacity and individual liberty that sustains free societies.” But failing to prepare increases the risk of undemocratic outcomes like the Manhattan Project scenario; building capacity increases the odds that other viable options exist and that society has time to choose among them legitimately.
• “Concentration of power and government abuses.” Taken seriously enough that the authors deliberately excluded expanding emergency authorities like the DPA from their recommendations — emergency powers have been “a double-edged sword since antiquity,” and concentrating power is itself “a paradigmatic loss of optionality.” The Pentagon–Anthropic dispute illustrates the bind: authorities meant for foreign adversaries and wartime appear to be used to punish a domestic company over lawful contract positions, yet the actions in question (supply-chain exclusion, forcing model modification) are exactly what one might want available in a genuine emergency. A promising partial answer: require governments to use only law-following AI systems, with benchmarks measuring whether guardrails prevent a system from carrying out an illegal order, and legislation barring officials from using systems that fail. The broader point: not all optionality-increasing policies raise concentration risk symmetrically — better government safety evaluations mostly do not.
• “Private governance is all you need.” Dean Ball’s claim that private governance matters as much or more is called “probably correct,” but both his Framework for the Private Governance of Frontier Artificial Intelligence and Hadfield and Clark’s Regulatory Markets require a highly competent, well-resourced government office to license or supervise the private regulators — so both concede government’s key role. Independent verification organization (IVO) bills are endorsed in principle. Three limits on private governance remain: private for-profit companies are legally obliged to prioritise shareholders, and “attempts to work around this fact with clever corporate governance structures have so far failed spectacularly”; tort liability is a main mechanism aligning company and public interests, and existing tort law may not do this well, so reform should start now; and a voluntary industry consortium lacks authority to compel defectors when billions are at stake. “No amount of ingenuity… is likely to create an adequate substitute for governments’ monopoly on violence, or to confer democratic legitimacy on private corporations.”

Conclusion

• If the four premises hold, governance should stop relying on predictions about how transformative AI will arrive and focus on ensuring governments can decide well when decisions are needed.
• The listed first steps: information-gathering authorities, whistleblower protections, secure information-sharing channels, flexible rules and definitions, robust assessments and evaluations, lab security standards, and hiring and talent reforms. More ambitious measures such as large-scale public investment in compute may also be justified, but the essay’s focus is “deliberately pragmatic.”
• “The cost of implementing these policies is modest, relative to the potential benefits. The cost of failing to act, by contrast, is potentially catastrophic.”