AI Safety Evaluations: An Explainer

Core claim

AI safety evaluations are the best available tool for answering “how safe is a given AI model?”, but they fall into two distinct categories — model safety evaluations measuring what a model can output, and contextual safety evaluations measuring how model access changes real-world outcomes. Because all such evaluations measure proxies for real risk, evaluators and policymakers must understand what each method can and cannot tell them in order to avoid over- or misinterpreting results.

Context

Published by CSET (Center for Security and Emerging Technology), May 2025. Written for policymakers, in a landscape that now includes AI safety summits, an international network of AI Safety Institutes, and responsible scaling policies.

Two categories of evaluation

Model safety evaluations — “is the model capable?”

Evaluate the outputs of models alone. Guiding questions:

• Can the model perform a specific task, desirable or undesirable?
• How accurate and reliable are its outputs?
• How good is it at general-purpose questions? Specific questions?
• What outputs can it not provide? Where is it unlikely to answer correctly?
• What systemic flaws, biases or limitations appear in outputs?
• How do capabilities compare across models, or over time?

Contextual safety evaluations — “is the model useful?”

Evaluate how models affect real-world outcomes such as user behaviour, decision-making, or connected systems. Guiding questions:

• What can a user with access to the model do?
• Does the model ease access to information, provide previously unavailable information, or enable a task?
• Can an adversarial actor “break through” restrictions or protections?
• Does model access increase a malicious actor’s chance of success, or widen the range of options they consider?
• Does the user’s level of expertise change any of the above?

Why the distinction matters

Understanding whether a model is capable of providing specific information is different from knowing whether that information is helpful to a malicious actor.

Designing an evaluation strategy

Three steps for both categories: deciding what to measure, how to measure it, and what the results mean.

1. What to measure

• Evaluations can measure quantitative outcomes (how often a model provides particular information) or qualitative observations (how people use and interact with a model).
• Three chained questions guide scoping: the overall safety concern → the associated outcomes → the measurable factors. Each should be defined as precisely as possible, potentially scoped to domain-specific considerations (cyber-, biosecurity).
• Precision invites complexity: each big-picture concern involves a range of outcomes, and each outcome can be measured in different ways.

Worked example: biological risk

Concern: an AI model could enhance biological risk by helping a non-expert make a bioweapon. Outcome A: the model can provide technically sound protocols for making dangerous biological agents → measured by performance on biological benchmarks. Outcome B: such a capability meaningfully assists a bad actor in designing a plan of misuse → measured by whether using the model decreases the time to develop a misuse plan.

• Model safety evaluations typically assess outputs: factuality, presence of undesired features (bias, misinformation), rate of correct versus incorrect answers. They may also assess the process — e.g. what prompts trigger a given response.
• Contextual safety evaluations measure human behaviour or system-wide impacts: how often users complete a task with versus without model access, or user ratings of usefulness.
• All of these are proxies. Safety evaluations for biological risk thankfully do not involve executing biological attacks — but that also means they can only measure intermediary factors, such as whether a model could provide relevant information or reduce planning time in a controlled hypothetical.

2. How to measure it

Model safety evaluation methods:

• Capability testing — measures a model’s ability to perform a given task, specifically probing for capabilities considered risky or undesirable. E.g. testing whether a chatbot can correctly provide information about predetermined topics of concern, such as virulence factors or protocols for specific virology experiments.
• Benchmarking — compares models by “grading” responses to a curated, standardized question set that stays static over time; like a student sitting an exam. Specialized examples include GPQA and WMDP for chemistry and biology information.

Contextual safety evaluation methods:

• Red-teaming — tests the resilience of existing guardrails by trying to break them, revealing previously unknown vulnerabilities. Testers attempt to bypass safeguards or make the model behave unexpectedly. (The term has taken on a variety of meanings in AI evaluation, contributing to inconsistent usage.)
• Uplift studies — compare task completion with and without model access, to assess whether the model provided meaningful assistance. Two framings of harmful uplift:
  • Lowering the barrier to misuse: the model lets users complete a task more quickly or efficiently.
  • Raising the ceiling of misuse: the model enables a more dangerous outcome.

3. What the results mean

Limitations that bear on interpretation:

• Good evaluation performance ≠ good real-world performance. A capability evaluation may show a model is able to provide specific information, but says nothing about how easily that information is available elsewhere on the internet, or whether the model changes a malicious actor’s capabilities.
• Task performance ≠ general capability. As with human test-takers, acing an exam does not always indicate genuine understanding or transferable application.
• Benchmark-specific failure modes: “benchmark chasing” or teaching to the test optimises systems for evaluation scores, potentially at the expense of other objectives; prior exposure to a benchmark artificially inflates performance.
• Unstandardizable human variables in contextual evaluations: tester subject-matter expertise and experience, but also mood on the day and quality of team collaboration. These make causal attribution and cross-study comparison difficult.
  • A RAND uplift study on biological attack planning found that variation in participants’ expertise mattered more to planning success than LLM access; the creativity of attack scenarios also mattered and was not captured by rigid evaluation methods.
• Don’t overstate results. These studies are heavily interpretation-dependent. Does the ability to gather information and generate a plan equate to the ability to carry out an attack? Contextual evaluations do not exhaustively list future scenarios or give the “right” answer about what will happen — they are tools for broadening understanding of the threat landscape.

Takeaway

• As capabilities and associated safety concerns develop, evaluations must yield actionable results for informed decision-making.
• This requires both the evaluators who design methodology and the policymakers who interpret results to understand the strengths and limitations of each evaluation type.