CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

Summary

270 tasks over 90 papers in computer science, social science and medicine at three difficulty levels, testing whether an agent can reproduce a study's results from the provided code and data, with a parallelised evaluation harness and both AutoGPT and a task-specific CORE-Agent baseline; best accuracy on the hardest level was 21%. OLDER than the window but flagged because it is the founding computational-reproducibility agent benchmark that ReplicatorBench, MLReplicate and ReproRepo in the catalog all position against, and the dedupe file shows no entry for it.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: replication and reproducibility). Verified against: Opened https://arxiv.org/abs/2409.11363, read full abstract, 5 authors, 2024-09-17, cs.CL/cs.AI/cs.MA. Dedupe: no 'core-bench' and no '2409.11363' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. (abstract)

To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. (abstract)