The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas

Summary

Recruits 43 expert researchers to execute randomly assigned research ideas, either expert-written or LLM-generated, each spending over 100 hours and writing a 4-page paper, then has the projects blind-reviewed. Review scores of LLM-generated ideas fall significantly more than expert-written ideas on novelty, excitement, effectiveness and overall (p < 0.05), closing and in several metrics reversing the ranking observed at the ideation stage. The strongest existing warning that apparent novelty of LLM-generated research output is not the same as epistemic quality - directly relevant both to why this paper measures justification soundness and traceability rather than surface novelty, and to designing a research-forecasting task whose outcome is verifiable.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the Architecture as Epistemology (ISR) paper (angle: epistemic quality of LLM outputs; LLM agents in forecasting). Verified against: Opened https://arxiv.org/abs/2506.20803 (arXiv abstract page, v1 25 Jun 2025)

Critique / open questions

Queued — not yet assessed.

Key quotes

Large Language Models (LLMs) have shown promise in accelerating the scientific research pipeline. (abstract)

A key capability for this process is the ability to generate novel research ideas, and prior studies have found settings in which LLM-generated research ideas were judged as more novel than human-expert ideas. (abstract)