Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

Summary

A pairwise-battle evaluation of research ideas from 14 frontier LLMs and 5 research-agent architectures, with all systems given identical shared literature contexts drawn from papers the human judges already know. Over 6,000 double-blind comparisons from 105 active CS researchers yield an Elo leaderboard of proposal-stage expert preference that is stable across annotator composition and domain. Two findings matter for anyone assessing AI research output: agent scaffolding sometimes helps and sometimes underperforms its own backbone, and LLM judges still cannot reproduce expert preference (best judge 72.56% soft accuracy on overall quality).

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: autonomous AI scientists and their evaluation). Verified against: Opened https://arxiv.org/abs/2608.29696 on 2026-09-07; verbatim abstract read (14 LLMs, 5 agent architectures, 6,000+ comparisons, 105 researchers, 72.56% best-judge soft accuracy), 11 authors, submitted 30 Aug 2026. Not in dedupe_titles.txt.

Critique / open questions

Queued — not yet assessed.

Key quotes

Evaluating research ideas generated by LLMs is difficult because their scientific value cannot be fully determined by objective criteria, and no single reference answer specifies what counts as a good idea. (abstract)

To address this challenge, we introduce Ideation Arena, a battle style platform that evaluates research ideas through pairwise human assessment. (abstract)