Abstract
New results from ForecastBench — the leading AI forecasting benchmark, and the only one comparing AI forecasters to human superforecasters directly — show that several frontier AI systems are now statistically indistinguishable from superforecaster-level accuracy on the tournament leaderboard, and that one system (Cassi AI) has for the first time outranked the superforecaster median specifically on market questions; caveats about stale superforecaster baselines and overlapping confidence intervals mean this looks like parity rather than clear AI outperformance.
Background
- Prompted by a blog post Scott Alexander published in early July 2026 claiming “AI superforecasters are here,” which asked how current AI systems compare to human superforecasters.
- The Forecasting Research Institute (FRI) runs ForecastBench, described as the leading AI forecasting benchmark and the only one that compares AI forecasters to human superforecasters.
Tournament leaderboard: parity reached
- On the tournament leaderboard (as of July 16, 2026), several models are “statistically indistinguishable from superforecaster-level accuracy.”
- The top current system is from Cassi AI; models from xAI and Google DeepMind are also ranked as indistinguishable from superforecaster-level accuracy.
- The tournament leaderboard is meant to capture the frontier of LLM forecasting ability, so submissions may use tools, added context, fine-tuning, or ensembling.
- Statistical detail: bootstrap one-sided p-values against the null hypothesis of equal accuracy with superforecasters are 0.41 for Cassi AI, 0.16 and 0.15 for xAI’s two submissions, and 0.14 for Google DeepMind’s submission — none reject equal forecasting accuracy at conventional significance levels.
Cassi AI’s approach
- FRI described Cassi’s approach in an earlier post from January 2026: a multi-stage pipeline that generates sub-questions and search queries, retrieves up-to-date context, and filters for relevance and recency, then produces forecasts via an ensemble of models, with another LLM analysing the reasoning to produce a final forecast.
- Since January, the overall architecture has stayed similar, but Cassi made incremental refinements — improvements to how the agent gathers, organises, and synthesises information, and better integration of relevant data sources.
First-ever outperformance on market questions
- Cassi’s system now ranks higher than the superforecaster median on market questions — the first time any model has achieved this.
- ForecastBench splits questions into two kinds: dataset questions (auto-generated from real-world time series — ACLED, DBnomics, FRED, Yahoo! Finance, and Wikipedia — via pre-specified templates) and market questions (drawn from prediction platforms Manifold, Metaculus, Polymarket, and the RAND Forecasting Initiative).
- AI systems matched superforecasters on dataset questions back in May 2026, but market questions are considered a better test of human-like forecasting skill: dataset questions favour LLMs’ strength at looking up data and establishing base rates, while market questions require judgment about novel, one-off events.
- One anonymous submission (“Anomous 8, ensemble”) is also ranked above superforecasters on market questions.
- Example recent market questions cited: “Will Lionel Messi score more goals than Cristiano Ronaldo at the 2026 FIFA World Cup?” and “Will July 2026 be the warmest July on record, according to Copernicus?”
Dataset questions: broader outperformance
- On the preliminary leaderboard (which ranks models on resolved dataset questions), 17 submissions now rank above superforecasters — up from just one (Google DeepMind’s “green-tree”) until very recently.
- The preliminary leaderboard surfaces early results before a system reaches the main tournament leaderboard: tournament-leaderboard inclusion takes 50 days after submission, while preliminary-leaderboard inclusion needs at least 225 resolved dataset questions, typically around 10 days after submission.
- These early rankings aren’t final, but FRI’s stability analysis suggests they’re robust, so many of these systems are expected to also rank above superforecasters once they reach the tournament leaderboard.
- Three submissions from Torchcast currently top the preliminary leaderboard. Torchcast describes its approach as a proprietary multi-agent architecture that analyses each question, gathers relevant evidence, generates forecasting perspectives, and produces a final probability estimate, using a combination of Torchcast-trained models and commercially available LLMs.
Caveats
- Superforecaster predictions were last elicited in 2024, so the AI-vs-human comparison relies on a statistical extrapolation that grows less reliable over time.
- Results depend on which questions have resolved and are inherently stochastic, so they may shift as more questions resolve.
- The 95% confidence intervals for many submissions overlap substantially — more consistent with superforecaster parity than with clear outperformance.
- These findings don’t mean ForecastBench is “saturated”: the superforecaster median is just a human reference point, and it’s possible AI systems will go on to exceed human superforecaster accuracy.
Planned next steps
- A fresh superforecaster elicitation round (fall 2026).
- Updated dataset questions (fall 2026).
- New “quantile questions” (fall 2026).