Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review

Summary

Head-to-head quality benchmark of four AI-Scientist systems (Sakana AI Scientist v1 and v2, CycleResearcher, Data-to-Paper) on a common set of 15 research proposals, producing 60 machine-written papers scored by three frontier LLM reviewers on originality, rigour, clarity and significance. Human/benchmark papers score roughly 2x higher than every autonomous system, and inter-model agreement is high between Gemini and Claude but weak for GPT-5.4. Directly useful evidence for the 'AI-generated paper quality / slop' argument.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: autonomous AI scientists and their evaluation). Verified against: arXiv abstract page opened 2026-08-29 (title, 2 authors, abstract and score statistics read)

Critique / open questions

Queued — not yet assessed.

Key quotes

AI Scientist systems capable of autonomous research have the potential to significantly accelerate scientific discovery. (abstract)

However, evaluating and comparing the quality of AI-generated papers remains an open challenge. (abstract)