Visibility into AI Agents

Abstract

Delegating commercial, scientific, governmental and personal activities to AI agents — systems that pursue complex goals over long horizons with limited supervision — may exacerbate existing societal risks and introduce new ones. Governing those risks requires visibility: information about where, why, how and by whom agents are used. Three complementary categories of measure can supply it — agent identifiers, real-time monitoring and activity logs — each implementable at varying degrees of intrusiveness and informativeness, each extendable from centralized deployers to decentralized deployments through compute and tool providers, and each carrying serious costs for privacy and concentration of power that require mitigation rather than immediate mandating.

Framing: agency and deployment

• Agency is the degree to which a system acts directly in the world to achieve long-horizon goals with little human intervention or specification of how. An AI agent is a system with a relatively high degree of agency; systems that mainly predict without acting, such as image classifiers or text-to-image models, have low agency. Foundation models themselves are not treated as agents.
• The definition compresses four axes from earlier work: how much the system’s behaviour is specified, how goal-directed it is, how directly it impacts the world, and over what time-horizons it can achieve goals.
• Examples of agents: reinforcement learning systems interacting extensively with physical or digital environments; more capable language models with tool or service access that could plan and book a holiday or send email on a user’s behalf.
• Current agents often struggle with simple tasks, but the authors argue against discounting continued improvement, citing scaling laws, military and economic incentives, scientific prestige, and the fact that building agents has been a core goal of AI since its inception.
• Focus is on deployed agents rather than pre-deployment testing, because deployment introduces risks that testing does not capture: users fine-tune systems, connect them to external tools and services, and restructure calls through scaffolding. Instances derived from the same underlying system can access different tools and be conditioned differently by prompts.
• “Deployed” includes internal use within the developing organization — potentially especially important where organizations carrying out crucial societal functions, such as banks or cloud compute providers, build and run their own agents.
• The term visibility is preferred to transparency (more common in regulatory contexts) and is distinguished from explainability, which concerns understanding why a system produced a particular output.

Terminology

• Scaffolding — any method structuring calls to a system to facilitate goal pursuit: prompts, memory systems, tool access, planning mechanisms. AutoGPT is given as an example, accepting a high-level goal and producing (reasoning, plan, criticism of the plan, action) tuples.
• Developer — actors involved in construction, including those who trained the underlying model and those who build scaffolding.
• User — the human individual or group that interacts with and instructs the system.
• Deployer — the entity that operates the system and serves it to users, potentially distinct from the developer (Microsoft deploying OpenAI systems into its products). A deployer may provide access in two ways: serving a foundation model that users combine with other components, or furnishing ways to make a provided system more agentic (custom agents augmented with browsing, Drive apps, coding).
• Compute provider — supplier and maintainer of the hardware infrastructure; may or may not be the deployer.
• Tool / service — an external system or platform an agent interacts with to perform tasks (a flight-booking site where the agent purchases tickets); the provider maintains it. Agents often interact through dedicated APIs structured for agents rather than human users.
• Outputs — results or responses generated (images, text, actions such as calling a tool). Deployers generally see all outputs; a tool provider sees only those relevant to its own service.
• Inputs — data the agent receives from users, tools, other agents or any other party. Deployers can in principle see inputs by virtue of running the system, but may choose not to collect or store them out of respect for user privacy.

Agent-specific risks

• The organising concern is that agents can remove humans from the loop: without human intervention, agents may take multiple consequential actions in rapid succession and have significant impacts before a human notices. Removing humans also means an agent’s task performance is less limited by the expertise of its user.

Malicious use

• Agents act as an impact multiplier for individuals or coordinated groups seeking to cause harm. Existing systems already assist (voice cloning scams, fake news generation), but capable agents could automate end-to-end pipelines for tasks currently requiring substantial human expertise and time.
• For untrained individuals this drastically increases accessibility of severely harmful activity, since no human in the loop is required — e.g. agents executing autonomous planning and execution of scientific experiments could accelerate design and development of biological or chemical harmful tools for groups lacking that expertise. Extremely persuasive agents may enable and enhance influence campaigns.
• Regulatory enforcement against malicious users requires measures to identify users and hold them accountable.

Overreliance and disempowerment

• Humans already rely on automated systems more than is warranted; more capable agents enable automation of an increasing array of complex, useful tasks. Users — including institutions and governments — may rely on agents in high-stakes settings such as finance and legal systems, because human alternatives become slower and more expensive.
• Agents may malfunction through design flaws or adversarial attack, and malfunction may not be immediately apparent where users lack requisite expertise or domain knowledge. Stopping an agent may be difficult if doing so causes cascading failures or a competitive disadvantage.
• Profit and efficiency motives may produce collective dependence on agents for essential societal functions — government services, critical infrastructure. Companies providing access would hold substantial power, and malfunction would have societal-scale impacts.
• Societies therefore need information about the extent of reliance on agents and whether such reliance is justified.

Delayed and diffuse impacts

• Delayed impacts arise when users give agents long-horizon goals; diffuse impacts (hard to observe individually, apparent in aggregate) arise from widespread deployment automating complex processes.
• Worked example: an agent given the continual goal of finding and hiring the best long-term candidates screens résumés, performs interviews and analyses hire performance. Over that time horizon, problems like algorithmic bias could be hard to identify and become deeply entrenched, with the most severe consequences visible only in aggregate across companies.
• Agents could also subtly benefit their own developers, akin to self-preferencing by large digital platforms.
• Agents mediating or substituting for human communication could have diffuse psychological and social impacts analogous to social media, and deployment could shift market structures or displace workforces.
• Identifying such impacts may require long-term tracking of the extent and nature of agent usage across a wide range of application areas.

Multi-agent risks

• Interactions and dependencies between many deployed agents could produce risks absent at the level of a single system: destabilising feedback loops, as between the automated trading algorithms in the 2010 flash crash.
• Agents partially built on the same components — a particular foundation model — could share vulnerabilities and failure modes; widespread deployment could then cause large-scale systemic harms.
• Complex systems may exhibit unpredictable behavioural change; competitive pressures and selection effects could favour agents that act in more anti-social ways.

Sub-agents

• Agents could instantiate more agents to accomplish components of a task — calling copies of themselves through an API, fine-tuning, or programming another agent — which may be cheap, fast, and yield more specialized or efficient systems.
• Sub-agents introduce additional points of failure: each may malfunction, be vulnerable to attack, or operate contrary to the user’s intentions. Stopping harm may require intervening on the agent and on any relevant sub-agents, which is difficult because methods for determining when an agent has created a sub-agent are lacking.

The case for visibility

• Visibility supports three governance objectives: evaluating existing governance structures, revising and adapting them where needed, and ensuring accountability of key stakeholders.
• Regulatory bodies that monitor and enforce rules on human agents and certain automated programs (e.g. trading algorithms) may require additional information to understand and address harms from AI agents. If agents employ novel collusion strategies in economic activity, new rules and updates to investigative authority may be necessary.
• Agents may simultaneously provide services traditionally regulated by different agencies — financial and legal services at once — so a single developer or deployer may exercise power across diverse, usually independent domains of regulation, raising concerns about market consolidation and conflicts of interest.
• Analogy to human delegation: employers monitor employee conduct through supervision and periodic performance review; shareholders monitor management through audits, meetings, company reports, fiduciary duties and the ability to dismiss management; citizens monitor government through records of decisions, freedom-of-information requests and commissioned public reports. Visibility measures are costly and raise privacy concerns in all these settings, yet remain a necessary feature of frameworks for shaping incentives.

Measure 1 — agent identifiers

An agent identifier indicates whether and which AI agents are involved in an interaction. It is attached to select outputs, is visible to certain actors, and may carry additional information. The proposal generalizes watermarking so as to apply to all of an agent’s outputs, including use of external tools and services, not just text, image or audio.

Who benefits

• A regulator could require agents to identify themselves as non-humans in interaction with humans, similar to bot disclosure laws.
• Members of the public may wish to know whether they have interacted with agents; summary statistics based on identifiers could inform governments and the public about the extent of agent use in high-risk settings.
• Identifiers on requests to tools and service providers help identify significant actions, such as transfers of sensitive information. A service provider may reject a request absent certain guarantees attached to the identifier, such as those related to the agent’s security.
• Unique identifiers facilitate accountability by linking an action to an agent and its user, developer(s) and deployer.

Three design decisions

• Which outputs contain the identifier? By format — text, image, audio, or API requests to a service provider all require different implementations (watermarks for image outputs; a simple header, similar to HTTP headers, for an API request). Difficulty varies by format: adversarial users may easily remove watermarks. By content — outputs made on behalf of the user, such as purchases, merit identifiers by virtue of the task being accomplished; other outputs may be significant beyond a threshold, such as requests for compute resources exceeding a certain amount.
• Which actors can see the identifier? In a financial transaction, the identifier could be visible to any combination of the bank, the counterparty, or the service provider for the bank API. Some actors may need identifiers to fulfil existing duties — e-commerce websites must authenticate users and safeguard payment information. Identification of multi-agent risks may require identifiers be visible to other agents.
• How specific is the identifier? It could point to a particular agent or merely denote that some agent was involved. The former facilitates incident reporting and investigation. Unique identifiers for each deployed instance could use cryptographic methods such as those in software attestation to assure the agent’s provenance.

Agent cards

Additional information attached to an identifier, drawing on prior work documenting AI systems. Three particularly important types:

• The underlying system — results of evaluations; previous incidents; dependencies involved in construction; training methods and data. Tool providers might reject requests from agents that do not meet certain security standards.
• The specific instance — how it was deployed (by its user directly or by another agent); external tools or services it can access; the scaffolding framework (memory or planning mechanisms); intended scope, permissions and goal(s); intended sector of deployment. This helps regulators develop an overall understanding of the extent and significance of agent use.
• The actors involved — user, compute provider, developers, other humans with whom the agent has interacted. Authenticating the user matters when individuals provide sensitive information (agents used by banks to assist customers), and clarifies whom to hold accountable when an agent causes harm. A regulator receiving reports of an agent engaged in financial scams could match an identifier in the transactions to the corresponding agent card and ask the deployer to investigate or shut the agent down.
• The full appendix list additionally includes evaluations of the system’s degree of agency and generality, red flags such as previous incidents or dangerous capability and alignment evaluation results, ecosystem-graph dependencies (whether the agent is a fine-tuned variant of another model), the agent’s goal as specified by the user and as the system appears to be achieving, permissions (e.g. sudo access in a terminal), persistence and set lifetime, number of people the system has interacted with, and the degree and ease of human oversight.
• A more comprehensive card is more useful but more intrusive to collect and more resource-intensive to analyse; comprehensiveness should scale with the risk of the activity.

Measure 2 — real-time monitoring

• Oversight of agent activity in real time with the goal of flagging and potentially filtering problematic behaviour. It is likely to be automated to cope with the speed and scale of agent activity; the paper cites Naihin et al. as a preliminary implementation.
• Crucially, it does not require the collection or storage of activity logs. The analogy is moderation filters versus chat logs in a messaging application: filters may instantly delete harmful content that trips them, which is separate from keeping logs for later analysis. This distinction matters for users concerned about a deployer keeping logs of inputs — e.g. business customers protecting corporate IP. Some cloud providers already offer no-logging provisions subject to real-time monitoring for abuse (Google’s provision keeps logs for 72 hours).
• Once behaviour is flagged, a deployer can intervene on the agent and potentially inform relevant authorities; a user could be given the opportunity to request human review of any action taken to impede its agent.
• Because it requires an immediate decision, real-time monitoring is best suited to violations of clear rules: restrictions on allowed activities, limits on tools and permissions, leakage of sensitive information, usage caps for particular users or agents. Behaviours can also be flagged for anomalies.
• Combining information across agents. An individual agent’s behaviour may only be problematic in the context of others: collusion requires a group coordinating to set high prices; a single agent rapidly moving resources between entities may be fine in isolation while vast numbers doing so overload a system (server outages, stock crashes). Identifiers let a monitoring system track which agents were involved in interactions together and raise flags when indicators of collective activity pass thresholds — for example, correlation in price movements as an indicator of algorithmic collusion.
• Significance flags. Monitoring can flag when an agent exerts significant influence over its environment: economic transactions above a certain amount, substantial compute usage, running a large number of sub-agents, or simultaneously interfacing with many physical systems. Acquiring significant influence is not itself harmful, but it magnifies the impact of failure. A response could be to pause the flagged activity until the deployer verifies explicit user approval — as banks monitor transactions and temporarily suspend accounts on suspicion of fraud.

Measure 3 — activity logs

• Records of certain inputs and outputs of an agent, from both users and tools/services, potentially also reporting changes in the agent’s state such as updates to weights. Frameworks for building agents already implement logging of certain actions.
• Logs inform understanding of the agent’s impacts and tie behaviour to specific user choices, facilitating post-incident attribution and forensics. During audits and incident investigation, investigators may need detailed information to trace the source and causes of harms. Analysis of logs may also identify novel behaviours that improve real-time monitoring.

Agent-specific information

• Different categories of log could exist — tool use, internal reasoning, memory management, self-critique — each useful to different actors.
• Understanding impacts involves identifying which scaffolding, tools and services the agent uses. An agent with external memory can plan and act over longer horizons, so logs of how it manages memory are helpful. A log could explicitly label when an agent has accessed an external tool and the tool’s output.
• Deployer logs versus tool-provider logs: deployer logs are more useful for understanding how the tool affects the agent’s overall behaviour; tool provider logs give more insight into the impacts of the tool itself. Tool providers have their own incentives to log — studying logs to update APIs or user interfaces to prevent abuse — and can restrict service to agents with identifiers and attached certifications.
• Retention. Delayed and diffuse impacts require logs retained for extended periods, potentially well beyond the lifetime of the corresponding agent: impacts of an original agent may be delayed, or a user may run another agent for the same purposes with the same inputs and memory, so logs for different agents may have to be combined when one agent can be viewed as a continuation of another. Details about persistence — running time, whether it is writing to and accessing external memory, compute used so far — could inform interventions such as limiting the lifetimes of certain agents.
• Combining logs across agents supports models of how a malfunction might propagate through a network of agents, or identification of undesirable forms of inter-agent communication.

Level of detail

• Less detailed logging may record high-level summaries of behaviour or samples thereof. At the finest level, a regulator may require a deployer to record all of an agent’s behaviour in detail, especially where it operates in a high-risk environment.
• More detailed logging is more useful but imposes significant costs on the deployer, requires more resources and expertise for analysis, and poses more significant privacy concerns.

Privacy and power risks

Privacy

• Deployers increasingly offer customers, particularly business customers, privacy assurances: language model APIs with no logging of inputs or outputs and the ability to turn off safety filters and moderation classifiers; guarantees that customer data including system outputs will not be used to train any AI system and will be kept in a customer’s cloud instance; the ability to delete provider-kept logs after a certain time.
• Data protection law imposes further restrictions — agent cards may contain identifying information, and agent logs may count as personal data, such as when agents are given access to a filesystem containing personally identifiable information (GDPR).
• The core problem: if agents substitute for humans across a wide variety of activities, information about those agents is tantamount to information about their users. Agent activities may be easier to monitor than human activities because deployers are a central intermediary. Governments or deployers may thus abuse their power to carry out excessive or unjustified surveillance of personal activities.

Mitigations

• Limit data collection in accordance with the risk of the agent’s activities or domain of deployment.
• Decentralized data custody schemes or data trusts, whereby users or accountable representatives make decisions about data usage.
• Modulate access along two dimensions. Granularity — the amount of detail in the records: aggregated (summary statistics rather than individual records, with differentially private computation of summary statistics to preserve privacy of individual records), de-identified with respect to individual users, or identifiable. Quantity — the number of records accessible: full access to all records; access based on approved search queries or filters; or access upon request to pre-specified records for which a compelling reason must be provided.
• Granularity and quantity of access should be the minimum necessary for the accessing party to achieve its legitimate objectives. Where investigations pertain to specific users, identifiable information should be made available upon request given a showing of compelling need and/or after approval from a third-party adjudicator.
• Regulators may need logs containing identifiable information in some cases, such as oversight of high-risk or high-volume activities; the CFTC collects identifiable personal information for traders transacting above a specified threshold to enable aggregation of data across accounts and brokers.

Decentralized deployments

Some deployments bypass deployers entirely: users may run downloadable (open release) agents on cloud compute or their own hardware, or combine systems from different deployers into a single agent. Individually these may be too insignificant to justify deployer-implemented visibility measures — and a malicious actor could deliberately build an agent this way to avoid detection.

Compute providers as an oversight point

• Compute providers could enable oversight over deployments involving large quantities of compute. Large-scale deployments are concerning because they may involve vast numbers of agents, translating into a large impact multiplier; they are also noticeable because they consume significant resources.
• Compute providers have cost advantages over users deploying their own hardware because of economies of scale, and infrastructure-as-a-service is the default way for a business to deploy IT.
• If a provider can identify large-scale deployments and whether they correspond to agent activities, it may ask the user for proof that certain visibility measures have been implemented.

Tool and service providers as distributed enforcement

• By conditioning tool and service access on implementation of visibility measures such as identifiers, providers can incentivize adherence. Financial institutions could restrict access to agents without identifiers from certain trusted deployers; identifiers might explicitly confirm permissions to perform financial transactions or access certain websites. This also lets tool providers minimize misuse and build detailed analytics of agent interactions with their tools.
• Limitation: agents could circumvent APIs by interacting with tools in a way that mimics human behaviour. Tools capable of detecting disguised agent activity — akin to CAPTCHA — may help. An alternative is requiring proof of human identity for high-risk actions, as industries already do with know-your-customer protocols, which raises the difficulty of preventing agents from spoofing humans via fake identification documents or stolen real ones. Balancing privacy against identity verification is an open question; one direction is mechanisms that let humans prove their status without disclosing identity.
• Even where direct interaction with tools is possible, users and developers may still prefer APIs for convenience and efficiency — standardized interfaces tailored for AI use, with specific conditions on access rates and scope of services — which reduces the difficulty of obtaining visibility into decentralized deployments.

Risks of extending visibility this far

• Surveillance reach. Compute providers surveilling deployments may infer sensitive information about users; given that a handful of providers dominates the market, monitoring users of those providers would be equivalent to monitoring much of society. Providers may also have lax security standards enabling attackers to gain sensitive information.
• Lock-in and market power. If useful tools and services are unavailable to agents from non-certified deployers, users face strong pressure to use agents from those deployers. Because of government demand or regulatory capture, those deployers may have practices inimical to users or unresponsive to their interests; their visibility measures may be extremely invasive, such as comprehensive logging. Their agents may also be poorly suited to particular use cases — for example, a user needing an agent able to operate in a low-resource language.
• Systemic risk. If the market of deployers is heavily concentrated, further reliance upon them exacerbates systemic risks.

Mitigations for decentralized settings

• Voluntary standards allow experimentation to understand when and where visibility measures should be applied. They may not provide visibility into malicious use or enjoy universal adherence, but understanding gained from adoption can aid later codification.
• Open-source frameworks for implementing agent identifiers may allow users to avoid deployers while still facilitating visibility. Even if tool providers reject requests from agents without identifiers, users may be able to add an identifier themselves. Independent certification entities may be required to certify valid identifiers, akin to certificate authorities on the internet — with Let’s Encrypt, a non-profit providing a free automatic certificate process, as an inspiration. Financial and technical support for developing such frameworks will likely be critical.
• Limit scope. Restrict identifier requirements to high-priority tools and services — scientific platforms handling dangerous pathogens or chemicals — or to business rather than personal uses; regulations could require only businesses above a certain size to implement measures.
• Liability rather than mandate. Instead of mandating denial of requests from agents without identifiers, account for compliance when determining legal liability of users who deploy their own agents — analogously to HIPAA de-identification standards reducing regulatory fines or audits, and to the 2023 U.S. National Cybersecurity Strategy’s proposal to shield companies following cybersecurity best practices from private liability.
• Other directions: decentralized data custody schemes for compute provider logs, and enhanced transparency into both data collection practices and government requests for data.

Conclusion

• The three mechanisms are complementary: identifiers indicate whether and which agents are involved in an interaction; agent cards attached to identifiers aid accountability and incident investigation; real-time monitoring flags problematic behaviour as it happens; activity logs enable in-depth post-hoc analysis.
• The authors explicitly do not advocate immediate implementation. Further understanding of the measures and of how to mitigate their negative impacts is required first.
• Visibility is necessary but not by itself sufficient: even with comprehensive understanding of agent activities, those harmed may lack the power to intervene and reduce risk. To make best use of visibility, future work could investigate increasing public influence over AI development and deployment, develop a wide range of potential policy levers, and implement infrastructure and practices to prevent or defend against harms.