How Closely Do LLM Reviews Align with Human Peer Review?

Summary

Controlled cross-provider comparison of GPT-5.4, Gemini 3.1 Pro and Claude Opus 4.6 against human reviews and final decisions on 300 topic-matched ICLR 2026 submissions (oral / poster / rejected, decision info stripped). All three models separate accept from reject but none reproduces the oral-versus-poster gradient; scoring patterns are provider-specific, and LLMs emphasise missing baselines where humans raise computational efficiency. Shows that coarse decision alignment does not imply agreement with fine-grained human reviewing priorities.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: AI in peer review). Verified against: arXiv abstract page opened 2026-08-29 (title, 2 authors, submitted 4 Aug 2026, full abstract read)

Critique / open questions

Queued — not yet assessed.

Key quotes

Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting. (abstract)

We compare reviews from OpenAI GPT-5.4, Google Gemini 3.1 Pro Preview, and Anthropic Claude Opus 4.6 with human reviews and final decisions for 300 topic-matched ICLR 2026 submissions, equally divided among oral, poster, and rejected papers. (abstract)