REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?

Summary

112 task instances built from social science papers that have public reproduction reports; agents receive the paper PDF plus the reproduction package and must judge reproducibility, not merely re-run code, which is what distinguishes it from result-reproduction benchmarks. Leading agents scored 21.4%; the authors' REPRO-Agent improved 71% over the best prior approach. OLDER than the window but flagged because it is the reference benchmark for social-science reproducibility assessment (and the target that PaperRepro, 2603.00058, and 2602.08561 are scored on), and the dedupe file shows no entry for it.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: replication and reproducibility). Verified against: Opened https://arxiv.org/abs/2507.18901, read full abstract, 6 authors, 2025-07-25, cs.CL. Dedupe: no 'repro-bench' and no '2507.18901' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. (abstract)

With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. (abstract)