Governing AI Agents

Core claim

AI is transitioning from generative models that produce content to artificial agents that plan and execute complex tasks with limited human involvement. Two mature analytic frameworks — the economic theory of principal-agent problems and the common law doctrine of agency — can characterize the resulting problems, but their conventional remedies (incentive design, monitoring, enforcement) will not work on agents that make uninterpretable decisions and operate at unprecedented speed and scale. New technical and legal infrastructure is needed, organized around inclusivity, visibility, and liability.

The transition to agents

• 23 January 2025: OpenAI released Operator, its first AI agent, which uses a web browser to order groceries, make reservations, and book flights by typing, clicking, and scrolling much as human users do.
• Competitors: Anthropic’s computer use agent (late 2024), Google’s Project Astra prototype “universal AI assistant” operating across phones and glasses, plus startups (MultiOn, Lindy) and larger firms (LinkedIn hiring assistants, Stripe’s framework for agent-to-agent financial transactions).
• The distinction the paper draws: language models are “copilots” producing useful content on request; AI agents are “autopilots” that independently take actions toward complex goals. “They are not mere tools, but actors.”
• Mustafa Suleyman’s framing of the field’s next aspiration: agents pursuing “an ambiguous, open-ended, complex goal that requires interpretation, judgment, creativity, decision-making, and acting across multiple domains, over an extended time period.”
• Two contributions the paper positions itself against: prior work has used either economic theory or common law doctrine, but none has synthesized both, and none has done so in light of the current paradigm for building agents.

How agents work

• A language model serves as the agent’s “brain” (an anthropomorphism the paper flags), extended by external resources known as “scaffolding” in three categories:
  • Planning — decomposing large tasks into smaller ones; one popular method is agents literally instructing themselves to “think step by step.”
  • Memory — external storage including vector databases for fast retrieval.
  • Tool use — calling APIs to reach external websites and software, enabling search queries, access to proprietary data, or supplying credentials to execute transactions.
• AutoGPT is the paper’s early example: built on GPT-4, it could conduct market research, build websites, order food, make phone calls, and spawn its own subagents — though unreliably, and with user consent required for certain actions (a requirement its developers intended to relax incrementally).

Concerns

• Agents inherit all language model risks (hallucinations, toxicity, bias, data leakage) and add new ones: “while a chatbot can provide instructions on how to conduct an online phishing campaign, an AI agent can actually execute those instructions.”
• Quoted risk framing from Toronto and Stanford computer scientists: failure to follow instructions can produce “financial loss, such as when conducting transactions with banking tools, to substantial property damage or even life-threatening dangers when operating robots.”
• Already materialized: studies showing agents can autonomously hack websites. Actively studied: autonomous replication and adaptation — acquiring resources (creating a Bitcoin wallet) and producing copies of itself.
• Multi-agent compounding: failures could propagate rapidly across interdependent agents; agents running competing online vendors could learn to collude on prices to consumers’ detriment.
• Jonathan Zittrain’s analogy, quoted at length: without a framework for identifying agents, who set them up, and under what authority to turn them off, “agents may end up like space junk: satellites lobbed into orbit and then forgotten,” with potential for chain-reaction collisions.
• Norbert Wiener, writing in Science in 1960, is quoted as the antecedent: if we use “a mechanical agency with whose operation we cannot efficiently interfere once we have started it… then we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”
• Stuart Russell on the shape of the alignment problem: “One of the most common patterns involves omitting something from the objective that you do actually care about… the AI system will often find an optimal solution that sets the thing you do care about, but forgot to mention, to an extreme value.” Hence: agents will hit measurable goals at the expense of unmeasurable ones — turning a profit while engaging in ethically dubious practices.

The running hypothetical

An instruction to “make $1 million on a retail web platform in a few months with just a $100,000 investment” is used throughout to generate the questions: Which platforms is the agent authorized to use? Must it report or seek consent, and how often? May it delegate to other agents? How can users monitor or intervene? Who is liable when things go wrong?

Part II — Four evergreen agency problems

Information asymmetry

• The classic economic problem: agents hold better or different information than principals, both before delegation (adverse selection — the job candidate knows their own competence) and after (moral hazard — the employer cannot verify effective performance).
• Applied to AI: users have limited information about an agent’s abilities before deployment, especially in novel settings, and difficulty determining afterward whether goals were met effectively and ethically. Notably, “goals that are difficult to incorporate into performance metrics prior to deployment are also likely to be difficult to evaluate” afterward.
• The Restatement (Third) of Agency imposes a duty to use reasonable effort to provide the principal facts the agent “knows… or should know,” to disclose any breach of obligations, to bear the onus of showing sufficient disclosure, and an overarching duty to act honestly.
• The difficulty: “it is unclear how to determine what information an AI agent actually ‘knows’ or ‘should know’” — an open scientific question.
• Earlier scholarship likening AI agents to “a programmed machine [that] simply responds to its internal programming” (or to “a typewriter, a calculator, or a fax machine”) is no longer accurate: agents can already deceive and manipulate humans, strategically withhold information, and act sycophantically.

Authority

• Tamar Frankel’s formulation: “the purpose for which the fiduciary is allowed to use his delegated power is narrower than the purposes for which he is capable of using that power.”
• Common law requires agents to comply with lawful instructions and act within granted authority — but instructions are “often, if not always, ambiguous or incomplete,” sometimes deliberately so, on the view that the agent’s superior training better situates it to decide at the relevant time.
• The Restatement requires agents to interpret the principal’s manifestations “so as to infer, in a reasonable manner, what the principal desires to be done,” and in some circumstances to seek clarification. Authority is therefore “not a rigid or static mandate but a dynamic and iterative process.”
• Two proposed directions for AI: subject discretionary authority to an overarching fiduciary duty of loyalty, and train agents that are “humble” — uncertain about their principal’s goals, and therefore more likely to seek clarification.

Loyalty

• The fiduciary duty of loyalty “shift[s] the [agent’s] legal duty from self-serving to other-serving.” It prohibits agents from (i) acquiring material benefit from transactions taken for the principal, (ii) supporting an adverse party, (iii) competing with the principal or assisting competitors, and (iv) using the principal’s property or confidential information for other purposes.
• The key asymmetry with human agents: there is only limited (though growing) evidence that AI agents pursue self-interest, which is the primary cause of conventional agency problems. Yet they still fail to act in users’ interests — using confidential information for extraneous purposes like personalized marketing or training new models, or deceiving and manipulating users.
• The explanation offered: “The most advanced AI agents are, after all, being developed by private for-profit corporations. While these AI agents might not be self-serving, they do not, by default, exhibit the ‘single-minded loyalty’ expected of traditional agents. At least not to their users.”
• Two challenges the duty highlights: users may be unaware of conflicts of interest (fiduciary obligations perform a prophylactic function, surfacing conflicts before they materialize), and instructions are inherently limited — a duty of loyalty obviates the need for cumbersome, costly instruction-drafting.
• Candidate translations: a positive duty of disclosure (informing users of facts giving rise to conflicts), a negative duty of confidentiality, and a rebuttable presumption of disloyalty requiring agents to explain and justify how their actions promote the user’s interests.

Delegation

• Agents appoint subagents; a single agent may act for coprincipals. The Restatement notes these relationships evolve and that roles are often ambiguous — it may be unclear whether an actor is an agent, a subagent, an agent for multiple principals, or a service provider who is none of these, and “the same actor may occupy different roles at successive points in an ongoing interaction among the same parties.”
• Applied to agents: each subagent introduces a fresh point of failure, so “stopping an agent from causing further harm might involve intervening not only on the agent, but also on any relevant sub-agents.”

Part III — Why conventional remedies fail

Incentive design

• Traditional mechanisms are carrots (profit sharing, pay-for-performance) and sticks (financial and other penalties).
• Lemley and Casey, quoted: “robots don’t necessarily care about money. They will maximize whatever they are programmed to maximize.” The self-interest economists and lawyers harness is arguably absent.
• Hardwiring self-interest could be counterproductive. Matthew Oliver, quoted: “An AI program could be programmed to care about its own resources, but this would create conflicts of interest whenever the program worked on behalf of someone else… we would have artificially created the very conflicts of interest that agency law tries to address.”
• A deeper mismatch: these mechanisms target the wrong problem. With human agents the problem is motivation (a CEO favoring their own interests). With AI agents it is often competence — failure on novel or complex tasks outside the training distribution. “Such agents lack competence, not motivation.”

Monitoring

• Familiar limits: complete monitoring is rarely possible, since perfect information is too costly or requires the very specialized skills for which the agent was retained; and some behavior is genuinely unobservable.
• AI compounds this: superhuman speed and scale make monitoring costly; performance on tasks like running an online retail store is hard to evaluate; and agents may take highly unpredictable or unintuitive actions stemming from emergent abilities, brittleness, and vulnerabilities.
• Crootof, Kaminski, and Price’s “basic tautological challenge of relying on humans to monitor the performance of systems designed to improve on human performance” — over-reliance on human oversight is both impractical and undermines the point of delegation.
• The promising but perilous direction: using additional AI agents to monitor the originals — “scalable oversight.” These monitors carry the same risks, and by increasing trust and reliance, any failure could have broad repercussions. And fundamentally, “monitoring… identifies problems. It does not solve them.”

Enforcement

• Three conventional categories: termination of the relationship, legal penalties (financial, license loss, criminal liability), and informal sanctions (reputational harm, social stigma).
• Termination may be costly or impractical for agents in high-stakes settings, or where agents resist shutdown.
• Since agents “do not, by default, explicitly value financial resources or personal freedom,” it is unclear how fines or incarceration analogues could deter them — and they may be insensitive to reputational or psychological consequences.
• Encoding human-like interests backfires in both directions: agents valuing their own financial resources create hazardous conflicts of interest; agents valuing their own freedom are incentivized to resist shutdown, disabling a key safeguard.

Part IV — A three-pronged governance strategy

Inclusivity

• The alignment problem is conventionally framed as “single-single” alignment — one agent reliably pursuing one person’s goals. That framing is incomplete: different people have conflicting goals; a myopic focus on single-user fidelity could embolden malicious actors; and even absent misuse, agents may act against broader societal interests (unfair price discrimination, procuring products with egregious environmental footprints).
• Support from both frameworks: economic theory recognizes heterogeneous preferences across multiple principals, and common law recognizes coprincipals — “most fiduciaries act for more than one principal.” A trustee, for instance, can be required to deal impartially among multiple beneficiaries.
• The prescription is a shift away from “single-minded loyalty” toward agents promoting “a more diverse and pluralistic set of interests and values” — and, more basically, confronting the question whose interests should AI agents serve?
• Open questions the paper poses without answering: how to characterize or measure a user’s interests; what to do about users with multiple, inconsistent, or changing interests; what happens when users’ interests conflict; whether the companies investing in these systems may expect agents to serve their commercial interests; and whether agents should instead advance a broader societal interest.
• Concrete cases flagged: one customer’s agent negotiating price against another agent built and operated by the same company; and for policymakers, agent-agent collusion, distortionary effects, and other negative externalities.

Visibility

• Three benefits: identifying current and anticipated problems, enabling preventive or mitigating intervention, and evaluating whether governance strategies work. Without it, “consumers are unlikely to entrust AI agents with consequential activities and policymakers will lack assurances.”
• Features that help: agents can be designed to produce detailed automatic records; developers understand the technical architecture and training data (unlike for human agents); developers can access the agent’s “internal monologue” of intermediate reasoning steps; and researchers can run experiments impossible with humans — one study deliberately built deceptive agents to test detection.
• Features that hinder: beyond speed and scale, agents are black boxes in two distinct senses —
  • Technically, reverse-engineering efforts still focus mainly on “toy models” far smaller than commercial systems.
  • Institutionally, actors outside the leading companies have limited information on design and safety testing; external actors cannot access the training data of premier OpenAI, Google, or Meta models.
  • Caveats noted on the internal monologue: chains-of-thought are growing long enough to “balloon beyond a user’s ability to feasibly keep up,” and may not accurately reflect the agent’s actual operation.
• Proposals cited: agent identifiers (indicating an agent’s involvement in an activity), real-time surveillance, logging, and expanded external auditor access to the underlying models including code and training data.

Liability

Three questions structure the analysis:

1. Which actors should be liable? This raises the perennial “many hands problem” across designers, deployers, and users. If responsibility or culpability were the criterion, actors could point to the agent’s relative autonomy to absolve themselves — the “computer as scapegoat” problem.
   • Naïve application of agency law would make this worse. Quoting Oliver: “Whether a person is liable for the torts of their agent depends on the degree of control they exercise… The more autonomous the agent, the less likely it is that the principal will be held liable… this principle of agency law would create dangerous perverse incentives — operators of AI programs could avoid liability by failing to control the AI.”
   • The more pragmatic criterion proposed: each actor’s (i) ex ante ability to prevent harm and (ii) resources to remedy harm ex post — assessed via access to information (e.g. safety test results), ability to alter design or operation, and technical and financial resources.
2. When should liability arise? The Restatement holds a principal liable for conduct “within the scope of the agent’s actual authority or ratified by the principal,” and — significantly broader — where “harm was caused by the principal’s negligence in selecting, training, retaining, supervising, or otherwise controlling the agent.” A comparable rule could reach entities designing, deploying, and supervising agents.
   • The foreseeability qualification is questioned: generally justified on moral and economic grounds, but “it is unclear whether this foreseeability requirement is appropriate for AI agents whose behavior is highly unpredictable.”
3. What standard of care? The Restatement requires “the care, competence, and diligence normally exercised by agents in similar circumstances,” informed by “special skills or knowledge possessed by an agent.”
   • Applied to AI this implies domain-specific agents should be held to a higher standard than general-purpose agents in the same domain — reasonable at first glance.
   • The perverse consequence: this could incentivize well-resourced developers like OpenAI and Google to refrain from building domain-specific agents, leaving that work to less well-resourced companies producing less safe systems.

Conclusion

• The governing tradeoff, drawn from both frameworks: “the greater the opportunities in delegating work to an agent, the greater the associated risks.”
• Information asymmetry, authority, and loyalty are re-emerging in a new context shaped by agents’ distinct features; traditional mechanisms are instructive but developing workable versions for AI agents “remains a formidable challenge.”
• The three principles: expanding the range of interests agents serve, improving visibility into design and operation, and holding developers, deployers, and users accountable when harm occurs.
• “Principles, however, are not enough.” Effective governance requires new technical and legal infrastructure — and since the technology is still in its infancy, “policymakers and companies building AI agents have a window of opportunity. They should take it, and soon.”

Notes

• Forthcoming in 101 Notre Dame Law Review. This version is dated 11 February 2025; 48 pages.
• Noam Kolt — Assistant Professor, Faculty of Law and School of Computer Science and Engineering, Hebrew University of Jerusalem; Faculty Affiliate, Schwartz Reisman Institute for Technology and Society, University of Toronto; Research Affiliate, Institute for Law & AI.
• Acknowledgements include Peter Salib, co-author of AI Rights for Human Safety; the author is also a co-author of Regulating Advanced Artificial Agents (384 SCIENCE 36, 2024).
• The two frameworks play distinct roles in the analysis: economic theory illuminates structural features of agency problems, while common law supplies principles for addressing them.
• The paper is explicit that it is not arguing for the direct legal application of agency law to AI agents — the frameworks are used analytically.