ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

Summary

Peer-reviewed (KDD 2026) benchmark that moves beyond code-and-data reproduction to genuine replication in the social and behavioural sciences, with human-verified replicable AND non-replicable claims so agents can be tested on identifying failures. Evaluates three stages — retrieving replication data, designing and executing experiments, interpreting results — and finds that agents design and run experiments competently but fail at sourcing the new data a replication requires. Built with the Center for Open Science; the most directly transferable replication benchmark for behavioural IS research.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: replication and reproducibility). Verified against: arXiv abstract page opened 2026-08-29 (title, 11 authors, submitted 11 Feb 2026, KDD 2026 AI4Sciences camera-ready note and full abstract read)

Critique / open questions

Queued — not yet assessed.

Key quotes

The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. (abstract)

Existing benchmarks focus primarily on the computational aspect of this task, testing agents' ability to reproduce or replicate research outcomes when having access to the code and data. (abstract)