ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

Summary

The standard dynamic benchmark for AI forecasting: 1,000 automatically generated and regularly updated questions about future events, with a 200-question subset used to compare expert forecasters, the general public, and LLMs on the same items. Expert forecasters outperform the top-performing LLM at p < 0.001. Because the questions are about not-yet-resolved events, it is the canonical solution to the contamination problem that Douven flags, and it is the natural reference point for calibrating how hard the paper's research-forecasting task is and how its agent populations compare to human benchmarks.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: LLM agents in forecasting). Verified against: Opened https://arxiv.org/abs/2409.19839 (arXiv abstract page, v5 28 Feb 2025)

Critique / open questions

Queued — not yet assessed.

Key quotes

Forecasts of future events are essential inputs into informed decision-making. (abstract)

Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. (abstract)