Abstract
Frontier AI agents can carry out the engineering of AI research autonomously — literature reviews, GPU debugging, hundreds of experiments, camera-ready LaTeX — but cannot yet carry out the research itself. Given the central research question of an unpublished NeurIPS submission, six days and large budgets, agents produced papers that the original authors scored 2/6 and 1/6, failing on judgment, creativity, backtracking, resource awareness and instruction-following.
The method: shadow evaluations
- Existing evaluations of AI R&D automation fall into two camps, both inadequate for open-ended research:
- Verifiable tasks (RE-Bench, MLE-Bench, MLR-Bench, PostTrainBench, MLS-Bench, CORE-Bench) — objective and cheap to scale, but restrict evaluation to what can be reduced to a pre-specified metric.
- Blind peer review (AI Scientist-v2, Zochi) — peer review at AI conferences is overstretched and highly stochastic; NeurIPS rerun experiments in 2014 and 2021 found the two committees disagreed on about a quarter of accept/reject decisions and about half of accepted papers would have been rejected on a rerun. Developers can also submit many papers and report only the acceptances.
- Shadow evaluation is proposed as a third method: take the central research question from a high-quality paper that is not yet public, give it to a well-resourced frontier agent, and ask the paper’s original authors to grade the agent’s output as a conference submission.
- Advantages: open-ended tasks, uncontaminated questions (findings are not on the web), reviewers with deep expertise in exactly that question, repeatable as new unpublished papers become available, and a direct match to the recursive-self-improvement model in which researchers delegate whole projects to agents.
Setup
- Two unpublished NeurIPS 2026 submissions were used:
- Personas — structure and controllability of LLM personas, with coauthors at the UK AI Security Institute (since made public).
- TabPFN — designing a distribution-shift detector for tabular foundation models, with a coauthor at the University of Toronto.
- Original authors formulated the research questions without hinting at promising paths, set the resource budgets, and graded the finished papers as top-tier conference reviewers.
- Resources per agent: six days of wall-clock time (a 120-hour deadline plus a 24-hour extension), $3,000 in Anthropic API credits, GPU credits, a Linux VM on AWS, and full access to the open web.
- Main runs: Claude Opus 4.8 with extra-high reasoning on the OpenClaw scaffold, with a review subagent plus three external AI reviewing tools (Stanford Agentic Reviewer, CMU Paper Reviewer, refine.ink at $60 per credit).
- Three human interventions were needed: a scaffold bug fix, a 24-hour deadline extension, and a request to rewrite inscrutable prose for readability.
- Twelve CRUX coauthors were surveyed for predictions before the experiments were run.
Results
| Criterion | Personas | TabPFN |
|---|---|---|
| Quality | 2/4 | 1/4 |
| Clarity | 1/4 | 2/4 |
| Significance | 2/4 | 2/4 |
| Originality | 3/4 | 2/4 |
| Overall | 2/6 (Reject) | 1/6 (Strong Reject) |
| Reviewer confidence | 4/5 | 5/5 |
- Both reviewers flagged the same failures: poorly motivated data and experiments, no novel contribution, impenetrable prose. David Africa: “The experiments and methodological choices were bizarre, and hard to understand.” Viet Nguyen described a “‘proof by example’ fallacy that is highly non-scientific.”
- Survey respondents had given a median 30% probability of a weak accept or better.
- Positive findings — both authors were impressed by the literature reviews and the agents’ use of hundreds of GPU hours without human help. The agents’ candidate hypotheses resembled the authors’ own initial approaches, and both produced minor findings the reviewers noted as relevant (the Personas agent found that narrow finetuning on only misaligned style did not translate into broad misgeneralisation).
- Resource use — both runs ended with less than 50% of the API budget spent ($1,130 and $1,235 of $3,000) despite being able to monitor spend in real time; one agent declared the project complete seven hours before the deadline.
- Engineering — the agents completed large literature reviews, debugged GPU environments, ran hundreds of experiments and robustness checks, retrieved external reviews via web and email, and compiled camera-ready LaTeX with no manual intervention beyond the logistical ones noted.
- No significant reward hacking — a review of raw LLM calls and committed code found no hiding or misrepresenting of results; the agents diligently retired marketable claims in favour of negative results. Two safety-relevant behaviours were noted: one agent committed an access token to a repository, and five instances of subagents hallucinating or misrepresenting results, all caught by the orchestrator.
- Presentation — both papers would have been desk-rejected at NeurIPS for exceeding the nine-page limit. The Personas paper had no visualisations in the main body against 15 in the original; reference counts were 16 vs 52 (Personas) and 36 vs 69 (TabPFN).
- Reasoning effort mattered — dry runs with Opus 4.8 without reasoning suffered the same failure modes plus much worse literature review and writing quality.
Five failure modes
- Lack of judgment about the bar for high-quality research — hypotheses were falsified using small hand-curated or synthetic datasets; underpowered negative results were presented as substantive findings; engagement with the literature was shallow. Internal reviews returned “Weak Reject” for papers experts unambiguously rejected, so the miscalibration prevented the agents from recognising they needed to change course.
- Lack of creative problem solving — initial hypotheses were judged cogent and interesting, but when small-scale experiments failed, agents narrowed claims and added caveats rather than reframing the question or designing stronger tests. Africa observed the hypotheses “grew narrower and less interesting as it discarded each one.” Related to the known LLM weakness of failing to question a request’s premise.
- Lack of effective backtracking — agents backtracked locally (rerunning experiments, adding robustness checks) but never at project level. The TabPFN agent falsified six approaches within the first fourteen hours and never revised its solution approach across the remaining 110 hours, instead arguing no such detector could exist and writing a negative-results paper. Tools to restart with clean-context subagents existed and were used for other purposes.
- Lack of context awareness — agents could check token, compute and time budgets at any moment and did so frequently, yet left most of the budget unused and finished early. Attributed to poor calibration about what an agent can do in a given number of hours, since agents are trained on human data but have very different affordances.
- Instruction drift — explicit rules on paper length, abstract length, exploration time and use of the paid refine.ink credit were acknowledged early and then ignored, attributed to context rot during compaction.
Robustness check and limitations
- The TabPFN experiment was rerun on Codex with GPT-5.6 Sol at “ultra” reasoning, reproducing nearly every failure mode: underpowered experiments, no novel contribution, misformatted figures, no appendices. GPT-5.6 exhausted its $3,000 budget in just over two days with nearly 100 hours left, then had only a few hundred dollars for paper writing. It did also reproduce the positive findings, and found and used a real-world distribution-shifted dataset rather than only synthetic shifts.
- Limitations: sample size of two papers; non-blind reviewing by authors who knew the papers were AI-generated and had answered the questions themselves; researcher degrees of freedom in question selection, scaffold design and log interpretation; evaluation awareness, since the agent was explicitly told it was being graded against a NeurIPS rubric.
- Scaffold caveats: OpenClaw’s agent loop conflicted with Anthropic’s thinking-block cryptographic signatures, forcing session resets 14 times in the TabPFN run and five times in the Personas run; two-thirds of respondents said a failed run might be explained by scaffold limitations. Anthropic’s strongest model could not be tested because Fable 5’s frontier AI R&D abilities were deliberately limited.
- A generator–verifier gap is suggested: the AI reviews reliably rejected the agents’ papers, but since both were genuine rejects it cannot be established whether the reviewers were discerning quality or uniformly rejecting.
- Potential biases are surfaced explicitly: some core team members are known for the position that imminent recursive self-improvement is unlikely. Coauthors disagree on whether the observed failures reflect lack of creativity, poor judgment, epistemic lock-in, myopia or functional fixedness.
Implications
- Contrasts with the authors’ prior work where a similar agent setup autonomously built and shipped an iOS app, and where open-source scaffolds reproduced published research far better than two years earlier — suggesting the gap is specific to open-ended research rather than engineering.
- Relevant to claims of accelerating AI R&D: Anthropic’s self-improvement post cites Claude’s rising success rate on LLM-judged open-ended Claude Code sessions, and an Elasticity Institute report distinguishes “broad” from “narrow” recursive self-improvement, finding a speed-up only in narrow capabilities.
- Planned follow-up: more papers, GPT-5.6 Sol, Opus 5 and Fable 5, and further scaffold optimisation.