AI Agents That Matter

Abstract

Agent benchmarking practices are inherited from language model evaluation and are unsuited to agents, which makes it impossible to tell genuine advances from hype. Evaluations must be cost-controlled rather than accuracy-only, must separate model evaluation from downstream procurement evaluation, must use holdout sets matched to the agent’s intended generality, and must be standardised enough to reproduce.

Framing: what makes a system “agentic”

• The paper declines to define “agent” and instead identifies clusters of factors that make a system more agentic on a spectrum.
• Environment and goals — complex environments (many tasks and domains, multiple stakeholders, long horizons, unexpected changes) and complex goals pursued without instructions on how.
• User interface and supervision — natural-language instruction, acting on the user’s behalf, requiring less supervision.
• System design — tool use, planning, reflection, subgoal decomposition; control flow driven by an LLM rather than fixed.

Shortcoming 1: evaluations are not cost-controlled

• Accuracy has no upper bound on cost: repeated sampling and majority voting raise accuracy on GSM-8K, MATH, Chess and MMLU; AlphaCode goes from near 0% zero-shot to over 15% with 1,000 retries and over 30% with a million.
• Where the environment supplies a correctness signal (test cases in coding), an agent can simply keep sampling until it passes, so accuracy alone cannot identify scientific progress.
• Three simple baselines were built and run against HumanEval leaderboard agents (LDB, LATS, Reflexion):
  • Retry — re-invoke the model at temperature zero up to five times on test failure.
  • Warming — the same, but raising temperature from 0 to 0.5 across retries.
  • Escalation — start with Llama-3 8B and escalate to GPT-3.5, Llama-3 70B, GPT-4 on failure.
• Results: no significant accuracy difference between the warming baseline and the best agent architecture; costs differ by nearly two orders of magnitude for similar accuracy. Reflexion and LDB cost over 50% more than warming; LATS over 50 times more.
• Consequence: there is no evidence that “System 2” techniques (planning, reflection, debugging) are responsible for the reported accuracy gains, because papers proposing them have not tested simple baselines.

Shortcoming 2: joint optimisation of cost and accuracy

• Plotting agents on a cost–accuracy Pareto curve opens a design space: optimise both jointly rather than accuracy alone.
• Total cost splits into fixed costs (one-time hyperparameter/prompt optimisation) and variable costs (per-run tokens); variable cost dominates at scale.
• Demonstration: DSPy was modified to search jointly over temperature, number of few-shot examples, choice of examples, and formatting instructions using Optuna, evaluated on HotPotQA.
• Outcome: 53% lower variable cost for GPT-3.5 and 41% lower for Llama-3-70B at similar accuracy versus default DSPy; the higher fixed cost is recouped after roughly 1,350 tasks.

Shortcoming 3: model evaluation vs. downstream evaluation

• Model evaluation is a scientific question — researchers normalise for compute or parameter count, because dollar costs change over time and vary by provider.
• Downstream evaluation is an engineering/procurement question — dollar cost is the actual construct of interest, and the fact that inference costs fall over time is a feature rather than a bug.
• Proxies for cost mislead downstream developers: Mixtral 8x7B costs twice as much as Llama 2 13B on Anyscale despite appearing comparable on active-parameter counts. If every developer picks a flattering proxy, multi-dimensional evaluation loses its usefulness.
• Recommendation: report input/output token counts alongside dollar costs so evaluations can be recomputed at current prices; a prototype interface was built.
• NovelQA case study — a good model benchmark that misleads downstream developers, because it asks all questions about a novel at once rather than sequentially. This makes RAG look worse than it is: RAG actually costs more than 20 times less than long-context in a realistic scenario, but only half as much on NovelQA — a tenfold overestimate.

Shortcoming 4: benchmarks allow shortcuts

• Overfitting is a much more serious problem for agents than LLM training-data contamination, because knowledge of test samples can be programmed directly into the agent, and agent benchmarks typically hold only a few hundred samples.
• Four levels of generality are proposed, each requiring a different holdout:

Level	What should be held out	Benchmarks with appropriate holdouts
Distribution-specific	In-distribution samples	1 / 1
Task-specific	Out-of-distribution samples	3 / 6
Domain-general	Tasks	1 / 8
Fully general	Domains	0 / 2

• Of 17 agent benchmarks surveyed, most lack an appropriate held-out set; 7 have no holdout at all and no stated intent to add one. Holdout sets should also be kept secret.
• Responsibility sits with benchmark developers rather than agent developers, since designing shortcut-proof benchmarks is easier than auditing every agent.
• WebArena / STeP case study — STeP reaches 35.8% accuracy, more than double the original baseline, by hardcoding policies for the specific tasks in the benchmark (e.g. appending /user/user_name to a base URL). Such policies are brittle to drift, which WebArena does not model, and the leaderboard is therefore misleading for downstream developers.
• Benchmarks also ignore humans in the loop: current evaluations test either fully supervised chatbots or fully autonomous agents, whereas real deployments sit in between. Simple human feedback moved GPT-4 from 0% to over 86% on challenging programming problems — so the absence of human-in-the-loop evaluation causes underestimation, while the absence of holdouts causes overestimation.

Shortcoming 5: standardisation and reproducibility

• Five root causes identified:
  1. Evaluation scripts assume an agent design that not all agents satisfy, so developers write their own scripts and results become incomparable.
  2. Repurposing LLM benchmarks for agents introduces inconsistencies — HumanEval lacks example test cases for 3 of 164 problems and embeds them in docstrings, so Reflexion and LATS removed those problems while LDB added machine-readable tests. Aggregators like PapersWithCode then club incomparable results together.
  3. High evaluation cost makes confidence intervals infeasible — running SWE-Agent across SWE-bench could cost over $8,000 for a single run at the authors’ $4-per-task cap, so error bars are rarely reported.
  4. Environment interaction produces subtle errors — WebArena tasks are not independent, since Reddit rate limits mean consecutive posting tasks fail more often, which affected STeP’s evaluation.
  5. Lack of standardisation produces bugs — LATS and STeP both marked some incorrectly completed tasks as correct and removed tasks from the benchmark (1 and 8 respectively).
• Existing frameworks (HELM, LM Evaluation Harness) address this for model evaluation but do not suffice for agents; an agent evaluation framework is named as future work.

Recommendations

• Compare agents on cost as well as accuracy, and report token counts.
• Keep model-evaluation and downstream-evaluation benchmarks separate, or at least as variants.
• Match holdout sets to the intended level of generality, and keep them secret.
• Standardise evaluation scripts and evaluation practices at the benchmark-developer level.