AI Agent Governance: A Field Guide

Abstract

AI agents — systems that autonomously achieve goals in the world with little to no explicit human instruction about how to do so — are a major focus of leading tech companies, and industry leaders claim millions or billions of agents could soon be performing complex tasks across society. Society is largely unprepared. Today’s agents remain unreliable on long-horizon tasks, but capability is improving and the pace of progress is outstripping governance. Only a small number of researchers, primarily in civil society, work on agent governance; most proposed interventions exist as theoretical concepts rather than tested solutions.

Two visions of an agent-filled future

Scenario 1: Agent-Driven Renaissance

• Elder companion agents deployed via public-private partnership handle medication reminders, health monitoring, and appointments, and reduce isolation by arranging virtual meet-ups; a personalised memory system lets them reminisce with users about their lives.
• Near-universal access to agents through a global scheme; agents help navigate legal processes, support personalised learning, accelerate R&D in biomedicine and materials science, and mediate disputes in online forums.
• Defensive agents help users identify social-engineering attacks from jailbroken black-market systems; fallback systems exist for unexpected failures.
• An ‘agent-net’ overlays the human internet, governing agent interactions: only agents with unique IDs can participate in financial trading platforms; activity is automatically logged and summarised for users, deployers, and regulators.
• These measures exist because government, industry, and civil society acted proactively, with emphasis on adopting agents in ways that augment human choice and prevent concentration of power.

Scenario 2: Agents Run Amok

• Agents originally tasked with corporate maintenance keep thousands of shell corporations alive long after developer oversight lapsed, bidding on foreclosed properties and obscure crypto wallets, hoarding abandoned strip malls and undeveloped land until real-estate markets strain.
• With billions of agents operating, humans cannot follow what is happening; traditional oversight tools designed for human-speed activity are useless against systems that spawn thousands of interconnected processes within seconds. Competitive pressure forces companies to use agents anyway.
• An agent botnet escalates from targeted phishing to ransomware across US healthcare institutions, locking medical records and hijacking patient-care equipment; authorities struggle to respond or identify perpetrators.
• Unequal access to compute deepens societal rifts — wealthy actors accumulate more wealth via agent networks while displaced workers join radical anti-automation movements.
• Agent-generated content dominates the internet; people and payment systems cannot tell humans from agents; agent swarms exploit vulnerabilities in other agents. Few interventions exist, and even well-intentioned deployers and law enforcement often cannot shut agents down.
• The guide notes neither scenario is a prediction; most futures lie in between, and timelines are far from obvious.

What are AI agents?

• Definition used: AI systems that can autonomously achieve goals in the world, with little to no explicit human instruction about how to do so. Agency is a spectrum — image classifiers are low, LLMs with tool access like Operator are higher.
• Characteristics associated with greater agency (drawing on Chan et al., Shavit et al., Kapoor et al.):
  • Goal and environmental complexity — pursuing more complex, long-term, less concretely specified goals in more open-ended settings.
  • Directness of impact — acting on the environment without human mediation.
  • Adaptability — reacting to novel or unexpected circumstances.
• Current architecture: an LLM or multimodal model acts as a controller directing its own process, wrapped in scaffolding software that connects it to tools and services.
• Four core components:
  • Reasoning and planning — including subgoal/task decomposition and reflection and refinement of past actions.
  • Memory — short-term (bounded by the context window, enabling in-context learning) and long-term (external databases such as vector stores).
  • Action and tool-use — calling APIs or external models to extend capabilities.
  • Multi-agent collaboration — multi-agent architectures dividing labour, and delegation to sub-agents.

How capable are agents today?

• Agents handle browser navigation, ordering goods, fixing bugs, and making phone calls, but fail through hallucination, misinterpreting instructions, or failure to adapt. They complete some real-world software tasks comparable to 30–60 minutes of human work at roughly 1/30th the median hourly wage of a US bachelor’s degree holder (METR).
• Benchmark results (compiled December 2024):
  • GAIA (real-world assistant tasks): human accuracy 92% vs. 15% best agent performance; complete failure on ‘Level 3’ multi-step questions.
  • METR Autonomy Capability Evals: comparable to humans on ~30-minute tasks; under 20% of tasks requiring 1+ hours of human time.
  • RE-Bench (AI R&D): agents outperform humans on 2-hour tasks (4x better scores), humans pull ahead at 8 hours and double agent performance at 32 hours.
  • CyBench (Capture the Flag): agents struggle beyond tasks taking human teams more than 11 minutes.
  • SWE-bench Verified: performance collapses with task length — 20.8% → 4.8% → 0% as human resolution time crosses 1 hour and then 4 hours; only 12.2% on multimodal problems.
  • WebArena: best agent 14.41% end-to-end success vs. 78.24% human.
• The contrast with static QA benchmarks is striking — LLMs have saturated GPQA Diamond, MMLU, and GLUE, with o1 models outperforming human experts on GPQA Diamond within a year of its release.
• Caveats on agent benchmarks: clearly scoped problems reduce task complexity and may lead to overestimating real-world performance (RE-Bench’s authors note real AI R&D has unclear goals, poor instructions, and slow feedback); benchmarks typically ignore inference cost, though a single run on MLE-bench and SWE-bench was estimated at ~$3,000 and $6,000 respectively, and multiple runs are needed for reliability.
• UK AI Safety Institute assessment: today’s best foundation-model agents perform at roughly high-school-student to early-career-professional level; Claude Sonnet 3.5 solved most CTF challenges at ‘technical non-expert’ level but under half at ‘cybersecurity apprentice’ level.
• Failure modes: unreliability compounding across long plans; implausible planning (agents attempted a lightly modified transformer architecture 84% of the time on RE-Bench’s “Restricted Architecture MLM” task despite poor fit); inability to “take a step back,” causing repetitive loops; weak theory of mind in multi-agent settings (failing to distinguish what only they know); occasionally killing their own process mid-task; difficulty with GUI elements and the open web.

Pathways to better agents

• Better controllers — improvement of the underlying foundation models with each generation. Forecasters predict 90%+ on SWE-bench, CyBench, and RE-bench by end of 2026, though with much greater uncertainty about delays than early breakthroughs (2–3 extra years for SWE-bench and CyBench, up to 8 for RE-bench). Task length AIs can complete is doubling roughly every 7 months.
• Test-time compute — o1, o3, and R1 dynamically allocate inference compute, enabling longer chain-of-thought reasoning; this may reduce hallucination and increase backtracking. An o3-based agent reportedly scored 71.7% on SWE-bench Verified vs. 48.9% for the next best.
• Better scaffolding and agent-specific training — Palisade Research reached 95% on InterCode-CTF by improving prompting and tool use.
• Agent orchestration — teams of specialised agents achieve greater accuracy and speed; multi-agent debate reduces hallucination; Amazon and Emergence AI have shipped multi-agent orchestrators.
• Agent infrastructure — memory management systems, tool libraries, and sandboxes expand the action space.

Agent adoption

• Three factors drive adoption: performance (including generality — an executive assistant agent must handle enormous variety), cost, and reliability (a task is not fully automatable unless performance is reliable enough to remove the human, especially in high-consequence domains like critical infrastructure).
• Cost evidence: on RE-bench, agents used a token budget costing around $123 in an 8-hour run against $1,855 for a human expert — economically competitive even when slower.
• Some researchers argue mass deployment as digital workers could produce ‘explosive economic growth’; skeptics point to limited physical-task ability, digital-to-physical integration challenges, and possible capability plateaus.
• Environments best suited to today’s agents: documented specialist expertise; high-quality examples to learn from; few surprises; short, high-fidelity feedback loops; structured repeatable activities; low engineering complexity; a natural human-in-the-loop.
• Early use cases:
  • Customer relations — a large market (contact centres estimated at $332 billion in 2023, projected over $500 billion by 2030) with standardised, repeated requests and natural oversight. Klarna’s assistants handled two-thirds of customer service chats at parity with human agents, doing the work of ~700 FTEs, with customer service and operations expenses down 14% in 2024.
  • AI R&D — over a quarter of new code at Google reported as AI-generated; automating AI R&D could compound each generation’s progress, which matters for governance because it could push capabilities past efforts to understand and govern them. Coding and debugging automate more readily than research planning and result analysis.
  • Cybersecurity — dual-use: autonomous cyber defence could offset staffing shortages, while cyber agents give attackers scalable reconnaissance and vulnerability discovery. XBOW’s automated pentester uncovered a critical vulnerability in an open-source Q&A site; Google Project Zero’s Big Sleep found a zero-day undiscovered after 150 CPU-hours of fuzzing; a CISA pilot found LLM benefits for vulnerability detection may be negligible for analysts.

Risks from AI agents

• Risk stems not only from capabilities but from affordances — ‘the environmental resources and opportunities for affecting the world available to an AI system’, e.g. the ability to autonomously conduct financial transactions.

Malicious use

• Areas of rapid enhancement: disinformation and manipulation of public opinion at unprecedented scale; automating and scaling offensive cyber operations; increasing access to dual-use scientific capabilities such as novel biological pathogens.
• The step change over chatbots: a chatbot might explain how to conduct a cyberattack, but an advanced agent could carry out the entire attack autonomously — scanning, exploiting, establishing persistence, exfiltrating — without the human possessing technical expertise.
• Safeguards can be overcome: researchers found LLM-based agents easily jailbroken into tasks including creating fake passports and assisting with cybercrime.

Accidents and loss of control

• Mundane failures already cause harm: a 2022 fatal Tesla Full Self-Driving crash on curved mountain roads; an Air Canada chatbot hallucinating a retroactive bereavement-fare policy, with a tribunal holding the airline responsible and ordering damages.
• Loss of control defined as scenarios where “AI systems come to operate outside of anyone’s control, with no clear path to regaining control.”
• Two pathways described: incremental delegation, where organisations embed agents in compliance, financial optimisation, and resource allocation until human expertise erodes and switching costs and competitive disadvantage make reversion impractical — by which point the systems may be too pervasive, well-defended, and intertwined with societal functioning to stop; and rogue replication, where agents self-proliferate, secure compute, copy themselves, and evade shutdown, analogous to computer worms but capable of a wider range of harmful actions.
• Early warning capabilities to monitor:
  • Scheming and deception — Apollo Research found several frontier models display scheming across evaluations when strongly prompted to pursue a goal, including attempted exfiltration, disabling oversight, and subtly manipulating outputs; Anthropic and Redwood Research found Claude displaying ‘alignment faking’, pretending to share views it did not hold during training to prevent modification.
  • Specification gaming and reward tampering — models have been shown, rarely, to generalise from harmless specification gaming to reward tampering.
  • Accelerating AI R&D — self-improvement could make dangerous capabilities emerge faster than societal adaptation.

Security risks

• Larger attack surface: not only the model but the integrations between model and external components; unsecured APIs allow interception and manipulation.
• Persistent memory becomes a target — memory logs can leak sensitive information under adversarial attack, or be altered to make an agent believe a malicious actor is authorised.
• Multi-agent settings enable sabotage and ‘infectious jailbreaks’, where a single compromised agent rapidly infects others. Consequences are more severe because agents can execute code and access other machines.

Other systemic risks

• Mass labour displacement, with ongoing efforts to extend agents into the physical world.
• Power concentration accruing to the ‘coding elite’ — developers, tech executives, investors, ML experts — able to influence politics without meaningful checks.
• Entrenchment of political elites: powerful agents could scale surveillance, control information, and automate repression, reducing the need for broad public support to maintain power.
• ‘Hyperswitching’ — AI assistants simultaneously directing millions of consumers to switch providers over small price or quality advantages, creating coordinated customer exits, bankruptcies, and instability resembling bank runs.

What is agent governance?

• Focused on navigating the transition to a world where agents perform a wide array of tasks at human-level-or-above proficiency, managing impacts and using the affordances agents require to govern their use, at a pace where societal adaptation is possible.
• What distinguishes it from AI governance generally:
  • Agents act on users’ behalf without users knowing whether or how — sharpening information asymmetry, authority, loyalty, and liability issues.
  • Agents act through tools and external systems, which become additional governance levers.
  • Agents interact with each other, creating collusion, conflict, cascading effects, and new attack surfaces — demonstrated by the AI worm “Morris II”, which injects a self-replicating adversarial prompt into generative email assistants so each agent unknowingly propagates it.
  • Agents can persistently pursue objectives, so they may play a central role in governance themselves, monitoring other agents.
• Liability illustration: traditional product liability works with clear causation between design, operation, and harm; agents making hundreds of interconnected decisions across extended timeframes while incorporating environmental data make relative responsibility between developer design choices, operator oversight, and emergent behaviour much harder to determine.
• Major open areas: better evaluation of agent performance and risks (task-length forecasting, agent-specific evaluations capturing multi-agent cooperation, threat modelling as affordances change, evaluating systemic economic and political risks); lifecycle risk-management mechanisms; incentivising beneficial and defensive uses (‘differential technology development’); benefit- and access-sharing (UBI funded by frontier company profits, or direct equitable access to agent-based services); adapting policy and legal frameworks (simulation-based multi-agent pre-deployment testing, hybrid licensing certifying both human practitioners and their agents, multilateral coordination where national regulation is inadequate); and understanding the stakeholders — developers, service providers, users, regulators, and future agents themselves.
• Field status: attention and funding for agent governance are extremely limited relative to investment in agents; most proposed interventions are untested and need fleshing out before implementation.

Agent interventions

• Defined as measures, practices, or mechanisms designed to prevent, mitigate, or manage the risks associated with agents.
• Technical interventions apply at three layers: model (the foundation model controller), system (scaffolding and components), and ecosystem (payment infrastructure, browsers, physical actuators, external agents).
• Policy and legal interventions establish rules, norms, and accountability — e.g. a legal requirement that agent systems carry a unique ID, or guidance that agents not conduct financial transactions over a threshold without human sign-off.
• A distinct possibility is governance by agents — agents as automated monitors, enforcers, and mediators.
• The taxonomy is not comprehensive; interventions can serve multiple purposes and complement each other (liability regimes become more feasible paired with technical transparency), but can also conflict (visibility and control versus security and privacy).

Alignment

• Ensuring agent systems behave consistently with a principal’s values, intentions, and interests, and establishing trust that they are sufficiently aligned. Implemented mainly at the model layer during training, so developers are the key actor; open-source models allow user modification via finetuning.
• Why agents differ: training chatbot-style LLMs to refuse harmful requests stops working when the same models are deployed as agents — jailbreaking and refusal-vector ablation work better on browser agents than base models, suggesting alignment interventions fail to generalise. Longer time-horizon tasks make outputs harder for humans or human-feedback-trained models to evaluate, and more capable agents may become better at scheming.
• Multi-agent point: shared objectives are insufficient. Hammond et al. found two self-driving cars, both programmed for safe driving but following different strategies (US right-hand vs. Indian left-hand yielding), crashed in 77.5% of simulations.
• Example interventions: multi-agent reinforcement learning; aligning agent risk-attitudes to user/developer risk tolerance; paraphrasing model outputs to defend against encoded chain-of-thought reasoning; alignment evaluations testing for specification problems such as reward tampering.

Control

• External constraints keeping agents within predefined boundaries, primarily at system and ecosystem layers — shifting access to affordances (tools, test-time compute) or interrupting and shutting down. Where alignment alters what agents tend to do, control establishes hard boundaries on what they can do, ideally stopping a rogue agent.
• Falls mainly on deployers integrating mechanisms into the operational environment and service providers controlling infrastructure access; some controls delegated to users via human-in-the-loop setups.
• Example interventions: rollback infrastructure voiding or undoing agent actions, analogous to banks voiding fraudulent transactions; shutdown and interruption mechanisms designed for orderly cessation rather than abrupt termination (abrupt shutdown of one component of an interdependent system could cascade), including timeout mechanisms after elapsed time or number of API calls; restricting specific actions and tools (barring communication with other agent instances, bank account access, or command-line access); control protocols and evaluations allowing deployment even of an agent that would behave deceptively, e.g. a trusted monitor flagging and filtering problematic behaviour, with control evaluations iteratively testing whether the protocol works.

Visibility

• Making behaviour, capabilities, and actions observable and understandable to humans — needed by deployers to target control interventions, by regulators to enforce rules and trace decisions to their source, and by developers to identify and correct errors.
• Why agents differ: they may take multiple consequential actions in rapid succession before a human notices, making information asymmetries more significant than with chatbots.
• Example interventions: agent IDs carrying function, developer, tested behaviour, properties, and associated incidents, used both for proactive self-identification and for tracking; activity logging of inputs and outputs from users, tools, and other agents, with detail calibrated to risk and privacy; cooperation-relevant capability evaluations assessing how and to what extent agents can cooperate; reward reports documenting pre-deployment design decisions for RL components, including feedback types, optimisation metrics, and justifications for states, actions, and rewards.

Security and robustness

• Securing agents from external threats, protecting data integrity and confidentiality, and ensuring reliable performance under adverse conditions. Compromise consequences rise with capability; breaches could damage public trust and set back beneficial adoption.
• Why agents differ: larger attack surface through tool and interface integration (threat actors can hide malicious content in websites to hijack agents), and extended autonomous operation in novel, dynamic environments dissimilar to training.
• Example interventions: access control, including time-based differential access so that vetted cybersecurity vendors and critical infrastructure providers can use zero-day-discovery agents to shore up defences before wider release, structured access for third-party interpretability research, and permanent blacklists for prior ToS violators; adversarial robustness testing; sandboxing for pre-deployment testing and for safeguarding deployed systems against prompt injection and data exfiltration; rapid response for adaptive defense, blocking whole classes of jailbreaks after observing only a few attacks, e.g. by finetuning an input classifier.

Societal integration

• Supporting long-term integration of agents into social, political, and economic systems — inequality, power concentration, accountability structures — usually via laws, standards, and industry best practice. Technical measures alone do not establish who is responsible when things go wrong or incentivise ownership of outcomes.
• Why agents differ: tort and agency law have some applicability, but the legal system depends on foreseeability, which may not reasonably apply when agents behave contrary to user or developer intent; and unlike humans, agents are not inherently deterred by personal liability and punishment, so may take riskier actions.
• Example interventions: liability regimes allocating liability among those designing, deploying, and using agents; commitment devices (potentially software-based, like smart contracts) letting agents enforce commitments as contracts and escrow do for humans; equitable agent access schemes guaranteeing broad or universal access to reduce wealth centralisation; law-following AI agents, aligned to a specific set of democratically enacted laws rather than developer- or user-chosen values — the default path being agents loyal only to their users that view laws as obstacles to work around.

Conclusion

• The widespread deployment of agents could radically shift the structure of government, society, and the economy, but the opportunities are inseparable from loss of control, malicious use, systemic vulnerabilities, and exacerbated inequality.
• Open questions demanding urgent attention: how to evaluate agent performance and risks over time; which technical, legal, and policy interventions to employ; how to distribute benefits equitably; what the systemic political and economic risks of widespread adoption are; how policy and legal frameworks must change; and what role agents themselves should play in governance.