PaperBench: Evaluating AI's Ability to Replicate AI Research

Summary

OpenAI's benchmark requiring agents to replicate 20 ICML 2024 Spotlight/Oral papers from scratch, with 8,316 individually gradable rubric tasks co-developed with the original authors plus a separate benchmark for validating the LLM judge. Best tested agent reached 21.0% and top ML PhDs still beat the models on a subset. OLDER than the window but flagged because it is the canonical paper-replication benchmark, referenced constantly by the newer harnesses already in the catalog, and the dedupe file shows no entry for it.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: research-agent benchmarks). Verified against: Opened https://arxiv.org/abs/2504.01848, read full abstract, 13 authors, v1 2025-04-02, cs.AI/cs.CL. Dedupe: no 'paperbench' and no '2504.01848' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

We introduce PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research. (abstract)

Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. (abstract)