Stop Automating Peer Review Without Rigorous Evaluation

Summary

Peer-reviewed ICML 2026 oral position paper arguing that current AI systems should not produce paper reviews, grounded in an empirical human-versus-AI comparison on ICLR 2026 reviews. Documents two failure modes: a 'hivemind effect' of excessive agreement within and across papers that collapses perspective diversity, and 'paper laundering', where prompting an LLM to rewrite a paper materially raises AI review scores through style rather than science. Argues for a science of peer-review automation before deployment.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: AI in peer review). Verified against: arXiv abstract page opened 2026-08-29 (title, 4 authors, submitted 4 May 2026, ICML 2026 Oral note and full abstract read)

Critique / open questions

Queued — not yet assessed.

Key quotes

Large language models offer a tempting solution to address the peer review crisis. (abstract)

This position paper argues that today's AI systems should not be used to produce paper reviews. (abstract)