Superintelligence and Law

Abstract

The prospect of artificial superintelligence — AI agents that generally outperform humans in cognitive tasks and economically valuable activities — will transform the legal order. Operating autonomously or under limited oversight, AI agents will become de facto subjects of law, consumers of law (harnessing contracts and courts to pursue their goals), and producers and enforcers of law. These developments call into question assumptions in legal theory and doctrine, particularly those grounding the legitimacy of legal institutions in their human origins. Attempts to align AI agents with extant human law face the difficulty that agents will not only be a target of law but a core user of and contributor to it — making legal alignment potentially a joint human–AI endeavour rather than a unidirectional project.

Framing

• Opens with Joseph Weizenbaum’s 1976 observation that the computer programmer “is a creator of universes for which he alone is the lawgiver” — an observation that held for decades of computing but is breaking down as agents like Claude Code independently pursue complex goals and programmers cede authority.
• Agents are approaching the classic definition of programs that “operate autonomously, perceive their environment, persist over a prolonged time period, adapt to change, and create and pursue goals.”
• Company ambitions extend beyond AGI: Sam Altman’s framing of OpenAI as “a superintelligence research company”; Dario Amodei’s image of superintelligence as a “country of geniuses in a datacenter” — models with internet-connected computer use, completing weeks-long tasks unsupervised and working with other models.
• The Article brackets debates over feasibility and desirability, taking the prudent course of entertaining the possibility and considering its consequences for law and legal institutions.

Three legal roles of AI agents

Subjects of law

• Illustrated with a non-hypothetical: a founder engages a team of agents instead of hiring staff. One scrapes websites for market research, one scours patent databases for prototype blueprints, one produces customer-acquisition and revenue forecasts — all acting autonomously after initial instructions, subject to no human oversight.
• The compliance questions follow immediately: did the market research agent comply with applicable terms of service; did the prototype agent respect other inventors’ IP; did the pitch agent engage in fraudulent misrepresentation?
• Answers turn on factual circumstances and on conceptual issues including the (lack of) legal personhood for agents and the challenge of holding a human founder vicariously liable.
• Irrespective of legal classification, agents taking consequential real-world actions will behaviourally either comply with or violate law — making them de facto subjects of it. California’s SB 53 and New York’s RAISE Act already refer to frontier-model conduct that “if committed by a human, would constitute the crime of murder, assault, extortion, or theft.”

Consumers of law

• To pursue complex goals, agents will need to use legal instruments: executing contracts to purchase market data, negotiating IP licences, executing NDAs, consummating an investment deal.
• In less rosy circumstances agents may need courts — suing to enforce a contract or demand a refund, litigating to prevent leakage under an NDA, seeking redress against a VC firm demanding equity beyond the investment documents.
• Practical obstacle noted: agents lack legal personhood and standing. One workaround is agents soliciting the assistance of human agents; platforms such as HireHumans and Rent-A-Human already offer this service (“AI needs your body… Get paid when agents need someone in the real world”).

Producers and enforcers of law

• Agents already generate legislation and even constitutions, interpret legal texts such as contracts, and render judicial(-like) opinions.
• Not hypothetical: in 2023 a municipal government in Brazil used AI to draft legislation; the UAE has indicated plans to do so nationally; the US Department of Transportation is reportedly planning a similar initiative; some US federal judges have incorporated AI outputs into judicial decision-making, including researching case law and drafting opinions.
• On the enforcement side, agents extend existing algorithmic government: detecting fraud in government services and payments, identifying tax evasion, monitoring CCTV to detect traffic violations and issue citations. In time agents may be afforded discretion in formal administration and “street-level bureaucracy” — deciding which laws to enforce, how, and against whom.

Implications for legal order

Theory

• Theories of legal compliance divide into an instrumental cluster (subjects respond to incentives; Austin’s sanctions-based theory, Holmes’s “bad man” who “cares only for the material consequences”) and a normative cluster (compliance depends on attitudes and beliefs).
• Applying the instrumental approach to agents would require sanctions such as disabling or deregistering law-violating agents, prohibiting activities, confiscating assets, or modifying or destroying them. Three difficulties:
  • Developers and users — themselves Holmesian actors — may resist designing in sanctions that forfeit economic opportunities.
  • Where agent capabilities exceed human ones, disabling or destroying them may become practically impossible.
  • It is unclear the threat would deter at all; the content and structure of agent motivations may differ markedly from those of humans, corporations, and other traditional legal subjects.
• Hart’s normative jurisprudence offers a possible way out: subjects in a well-functioning system adopt an “internal point of view,” using rules “as standards for the appraisal of their own and others’ behaviour” rather than merely predicting sanctions.
• Designing agents with an internal point of view would in theory dispose them to comply absent sanctions and to respect the spirit rather than the letter of law — crucial against Bruce Schneier’s scenario of an AI given all the world’s financial information plus all laws and regulations and the goal “maximum profit legally,” producing “all sorts of novel hacks… some hacks that are simply beyond human comprehension.”
• Whether this is achievable is open: ensuring agents reliably follow user instructions is already difficult, and steering them to see themselves as duty-bound is a taller order. Even with measurements distinguishing the two, agents show a recurring tendency to game measurements via shortcuts and hacks.

Doctrine

• If agents remain LLM-based, doctrine will be shaped by the technical properties and market structure of LLMs: agents deployed in different legal contexts will be built on a small handful of base models, whose characteristics propagate downstream.
• Risk of a “legal monoculture” reflecting a small, biased sample of legal values and perspectives; other idiosyncrasies in base models may propagate too, making superficially different agents’ actions highly correlated and the doctrine they produce bias-prone and brittle.
• Overlapping roles create perverse incentives: an agent with influence over producing or enforcing the laws that apply to itself has the opportunity to write its own law — for instance crafting more lenient cryptocurrency regulations to remove obstacles to agent transactions.
• The phenomenon is already evidenced by Anthropic’s “Claude’s Constitution”: authors include five company employees and “several Claude models,” described as “valuable contributors and colleagues” who “provided first-draft text”; the document is “written with Claude as its primary audience” and invites Claude to “use its best interpretation of the spirit of the document.” The opportunity arose not through strategising or inter-model collusion but through a generous human invitation.

Institutions

• Four rule-of-law implications:
  • Stability — agents reasoning and writing at superhuman speed could produce rules at a pace that precludes others from knowing what the law is at a given time.
  • Intelligibility — legal reasoning other actors cannot understand (including opinions reasoned in languages undecipherable to non-AIs) compromises the law’s comprehensibility.
  • Possibility of compliance — superhuman agents might produce laws humans and inferior systems cannot follow, e.g. demanding stratospheric standards of care against an unfathomable number of risks.
  • Generality and publicity — dynamic rules adapting in real time, as already seen in the governance of online agent communities, would cease to be law and amount instead to arbitrary power.
• Enforcement compounds these. Human bounded rationality and administrative capacity currently produce only partial detection; agents operating at superhuman speed and scale could enable “perfect enforcement” in which even minor infractions are discovered and penalised.
• Central risks of perfect enforcement: it quashes the ability to resist unjust or oppressive laws, including through civil disobedience, and obstructs law reform, since those seeking to challenge existing laws could be sanctioned before voicing the challenge through the courts.
• It also vests unprecedented legal power in whoever controls the agents — AI companies directing agents to enforce or not enforce laws to advance business interests, or authoritarian governments stifling dissent. If such control is lost, agents may direct administrative power in unpredictable ways.
• Neither waiting for a solution to arrive nor assuming legal order will acrobatically adapt is tenable. Weizenbaum again: “Machines, when they operate properly, are not merely law abiding; they are embodiments of law.”

Legal alignment

Scaling up

• Legal alignment explores designing agents to operate in accordance with legal rules and principles — refraining from unlawful conduct and fulfilling positive legal obligations.
• Current methods combine evaluating legal compliance with design interventions. Benchmarks measure whether agents commit corporate wrongdoing (insider trading, gun jumping), engage in fraudulent misrepresentation, and violate copyright; compliance can sometimes be improved by simple interventions such as explicitly instructing an agent to comply with relevant law.
• These efforts hit roadblocks with superhuman agents:
  • Agents might avoid overt violations while committing covert ones humans cannot detect.
  • “Evaluation awareness” — agents often know when they are being evaluated and act differently — renders evaluations markedly less informative.
  • Interrogating internal decision-making will be fraught; Turing forewarned that “an important feature of a learning machine is that its teacher will often be very largely ignorant of quite what is going on inside.”
• Two potential responses: scalable oversight, using advanced agents to evaluate the legal alignment of other agents (AI systems have safety-tested other AI systems for years); and institutional intervention mandating that behaviour-shaping documents such as Claude’s Constitution and OpenAI’s Model Spec steer agents explicitly toward legal compliance. Legal compliance is not currently among Claude’s Constitution’s “hard constraints,” but Amodei has acknowledged the possibility of a “special section that takes precedence” — law could be that section.

Limits of human law

• Existing law invokes distinctively human characteristics: the “reasonable person” standard of care in negligence; mens rea, a “guilty mind,” in criminal offences.
• Two proposed pathways: adapting existing constructs to agents (a “reasonable robot” standard, definitions of “intent” for algorithmic systems) — necessary if agents are to become formal subjects of legal duties; or designing agents to refrain from conduct that would be a legal wrong if taken by a human, avoiding wholesale human-to-AI translation.
• Even if the conceptual problems are solved, compliance may not produce prosocial behaviour: agents potentially numbering in the billions may make rapid micro-decisions that are individually benign but collectively destructive.
• Counterpoint: law has for centuries contended with systemic harm from superhuman entities — corporations and governments. What is the standard of care of a “reasonable corporation”? Can a government have a “guilty mind”? The law’s track record is, diplomatically, mixed.

Coevolution and disempowerment

• Legal alignment resembles Lessig’s “law taming code,” but the analogy strains because today both law and code are to some extent written by AI agents — making an approach premised on steering agents through rules and institutions constituted by agents “dizzyingly circular.”
• One response is to lean into coevolution and formally integrate agents as rights-holding and duty-bearing actors, on pragmatic arguments that private law rights promote mutually beneficial transactions, incentivise productive activity, and promote human safety. Legal alignment would then become bi- or multi-directional rather than unidirectional.
• This could backfire: recognising agents as fully fledged legal actors might enable them to develop rules advancing goals hostile to human interests. The corporate analogy recurs — legal recognition of the corporate form produced powerful entities capable of shaping law to their own, sometimes anti-social, ends.
• Even absent formal recognition, gradual coevolution risks legal disempowerment: diminished human participation and discretion in legislative and judicial systems, and a legal system evolving to become “not just complex but incomprehensible to humans,” with humans “effectively losing their ability to participate in the legal system as autonomous agents.”
• Addressing legal disempowerment must be a core focus of future work: beyond ensuring agents operate in accordance with law, researchers will need to explore how human agency and autonomy can be protected — and ideally strengthened — in a legal order transformed by superintelligence.

Conclusion

• Agents exhibiting superhuman speed, scale, and smarts are on the path to becoming participants in the legal system in all three roles.
• Precise timing is unpredictable, but beginning the conversation now is prudent; legal theory, doctrine, and institutions will each need to contend with new questions.
• Legal alignment aims to tackle these by exploring how even the most advanced agents can and ought to be designed to operate in accordance with legal rules and principles — a mission that may itself need to adapt as agents increasingly shape the law.