MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

Summary

End-to-end benchmark that asks six autonomous research systems (AI Scientist v1/v2, Agent Laboratory, CycleResearcher, AI Researcher, Tiny Scientist) to reproduce ICML 2025 outstanding papers, yielding 45 manuscripts judged by both automated conference-style review and structured expert human review. Automated review accepted 10 of 37 valid submissions, but human reviewers found methodological flaws, hallucinated results and reproducibility failures in all systems, and 59% of the automatically accepted papers contained fabricated or unsupported claims. Also shows compute does not buy quality: the cheapest system beat the most expensive despite a 38x token gap.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: replication and reproducibility). Verified against: arXiv abstract page opened 2026-08-29 (title, 5 authors, submitted 15 May 2026, full abstract with all statistics read)

Critique / open questions

Queued — not yet assessed.

Key quotes

Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks have failed to keep pace. (abstract)

To bridge this gap, we introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning reproducibility. (abstract)