AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot

Summary

The largest deployed field study of LLM reviewing to date: AI-generated reviews for all 22,977 AAAI-26 submissions, with survey evidence that authors and reviewers found the AI reviews useful and preferred them to human reviews on technical accuracy and research suggestions. Reports the safeguard architecture and shows the system beats naive LLM baselines at surfacing scientific weaknesses. The reference point for any IS argument about institutionalising AI in peer review.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: AI in peer review). Verified against: arXiv abstract page opened 2026-08-29 (title, 13 authors, abstract read)

Critique / open questions

Queued — not yet assessed.

Key quotes

Scientific peer review faces mounting strain as submission volumes surge, making it increasingly difficult to sustain review quality, consistency, and timeliness. (abstract)

Recent advances in AI have led the community to consider its use in peer review, yet a key unresolved question is whether AI can generate technically sound reviews at real-world conference scale. (abstract)