Measuring AI Ability to Complete Long Software Tasks

Core claim

Benchmark scores are hard to translate into real-world capability. METR proposes measuring AI performance by the length of tasks an agent can complete, calibrated to how long the same task takes a skilled human. On a 170-task software/ML suite, the 50%-task-completion time horizon has grown exponentially since 2019 with a doubling time of about 7 months (207 days, 95% CI 166–240). If the trend continues and generalizes, AI systems will be able to automate many software tasks that currently take humans a month within roughly 5 years.

Publication

METR blog post (19 March 2025) and accompanying paper, arXiv:2503.14499, published at NeurIPS 2025. Paper v3 (Feb 2026) updates headline figures. METR notes the original blog text and static figures are now out of date; current methodology and results live on their time horizons page and in Time Horizon 1.1 (Jan 2026).

The problem with existing benchmarks

• Frontier systems went from basic text generation to autonomously executing multi-hour ML research projects in five years, yet the real-world meaning of benchmark performance remains unclear.
• Current models are vastly better than humans at text prediction and knowledge tasks and outperform experts on most exam-style problems — but the best agents still cannot carry out substantive projects independently, or reliably handle low-skill computer work like remote executive assistance.
• Specific limitations of existing benchmarks:
  • They often consist of artificial rather than economically valuable tasks.
  • They are often adversarially selected for tasks current models struggle with (e.g. HellaSwag, Humanity’s Last Exam), biasing the human comparison.
  • Individual benchmarks saturate increasingly quickly, and there is no general quantitative way to compare across them — preventing comparison of models of vastly different capability (GPT-2 vs Claude 3.7).
• Because frontier developers have committed to using capability measures to determine risk mitigations, benchmarks that track and forecast capability are foundational to AI governance.

The metric: X% time horizon

• Definition: the duration of tasks (as measured by how long humans with relevant expertise take) that a model completes with X% success probability.
• Rationale: agents seem to struggle with stringing together longer sequences of actions more than with the skills or knowledge needed for single steps.
• Human task time is strongly predictive of model success. Regressing model success rate on the logarithm of human time-to-complete gives R² ≈ 0.80; at publication, models had close to 100% success on tasks taking humans under 4 minutes but under 10% on tasks over ~4 hours.
• Related prior framings: Richard Ngo’s t-AGI, Ajeya Cotra’s Bio Anchors report, Carlsmith. Methodologically inspired by Item Response Theory in human psychometrics — but using difficulty ratings derived directly from human baseline times rather than learned from agent performance.

Methodology

Task suite (170 tasks)

Suite	Contents
HCAST (subset)	97 diverse software tasks, 1 minute to 30 hours
RE-Bench	7 difficult ML research engineering tasks, all 8 hours
SWAA (Software Atomic Actions)	66 new single-step tasks, 1–30 seconds, enabling measurement of pre-2023 models

Example tasks by length: a 3-second multiple choice (“which file is a shell script?”); 1-minute Wikipedia research; 9-minute bug fix in molecular dynamics input files; 56-minute JSON-transformation script inferring rules from examples; 8-hour CUDA kernel implementation to speed up a backtesting tool 30x.

• Tasks under 1 minute measure professional software knowledge; at 10 minutes, the easiest meaningful step of a real project; the shortest standalone economically relevant projects take about an hour; by 8 hours, tasks represent valuable software projects.

Baselining and model runs

• Over 800 human baselines totalling 2,529 hours; 148 of 169 tasks have human baselines, with researcher estimates for 21 HCAST tasks.
• Baseliners are skilled professionals in software engineering, ML and cybersecurity, mostly from top-100 universities, averaging ~5 years of relevant experience.
• 12 frontier models (plus 4 near-frontier) released 2019–2025, with ~8 runs per agent/task pair. Average correlation between which tasks different models succeed at is ~0.73.
• Time horizon is fitted by logistic regression of success probability on log human task time; t_task is the geometric mean time of successful human baselines, and the fitted parameter h_agent is the 50% horizon.

Results

• Doubling time of 207 days (~7 months), 95% bootstrapped CI 166–240 days, over 2019–2025. Errors calculated by three-level hierarchical bootstrap over task families, tasks and runs.
• Absolute range: GPT-2 has a 50% horizon of 2 seconds; in paper v3, o3 has a 110-minute horizon and succeeds at several tasks over 4 hours. o3 lies above the long-run trend (p = 0.006).
• Individual models’ horizon error bars are wide but highly correlated across models — sampling easy or hard tasks shifts every estimate together — so the authors are more confident in the slope than in any single model’s horizon.
• The 80% time horizon shows a similar trend but horizons are roughly 5x shorter.
• This helps resolve the apparent contradiction between superhuman benchmark performance and models not being robustly helpful in day-to-day work: frontier models are capable of some tasks taking expert humans hours, but can only reliably complete tasks of a few minutes.

Drivers of progress

• Greater reliability and ability to adapt to mistakes, combined with better logical reasoning and tool use.

External validity and robustness

• Similar (noisier) trends hold across subsets of the suite: very short software tasks vs HCAST vs RE-Bench, and subsets filtered by length or by qualitative “messiness” assessments.
• SWE-bench Verified replication, using independently collected human time estimates rather than baselines, shows an even faster doubling time of under 3 months. METR suspects this is partly an artefact of operationalization: those authors exclude codebase familiarization time, which is a large fraction of short-task time but not of long-task time, so their human time estimates rise more steeply across the same tasks.
• Results are not especially sensitive to which tasks or models are included, nor to curve-fitting choices (regularization, task weighting, WLS vs OLS).
• Acknowledged gap: the tasks do not perfectly represent the average segment of intellectual labour by researchers and software engineers. The supplementary experiments find little evidence of slower trends on more realistic tasks, but do not rule out meaningfully slower trends on the full distribution needed to automate software engineering jobs. Agent time horizons can also differ by a large factor across task domain and reference human population.
• Notable limitation: performance is much lower on less structured, “messier” tasks.

Extrapolation

• Naive extrapolation implies a >1 month horizon (167 work hours) between mid-2028 and mid-2031.
• The steepness of the trend makes forecasts robust to large measurement error: if the absolute measurements are off by a factor of 10, the arrival time shifts by only about 2 years.
• Model error remains a live concern in the other direction: recent trends may be more predictive than pre-2024 ones. The 2023–2025 growth rate is about 20% faster than the 2019–2025 rate, and fitting only 2024–2025 data shortens the estimate for month-long-task AI by about 2.5 years.
• Most of the overall uncertainty comes not from data noise but from future changes in the trend and external validity concerns, which the sensitivity analysis does not capture.

Implications

• For benchmark design: measuring performance in units of human task length allows a single scale to span a wide range of capability levels and diverse domains, and gives absolute rather than merely relative performance a meaningful interpretation. Most benchmarks fail this because they cover a narrow difficulty range, or use ”% questions correct” over multimodally distributed or impossible questions.
• For forecasting and risk management: a fairly robust exponential trend on a metric that matters for real-world impact. If it holds to the end of the decade, frontier systems will autonomously carry out month-long projects — with enormous stakes on both benefit and risk sides.
• Infrastructure (Vivaria), data and analysis code are open-sourced for replication and extension.