Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Summary

Separates novelty from usefulness in autonomous-discovery claims by scoring agent trajectories on P-Creativity (novel relative to the agent's own prior solutions in a run), H-Creativity (novel relative to the human solution corpus) and task performance, over AIDE and AIRA-Dojo on 10 MLE-Bench tasks. An LLM-as-judge pipeline is validated against human creativity judgments. Results: P-Creativity declines as agents shift from exploration to exploitation, and agents show higher H-Creativity than medal-winning humans while performing worse - novelty that does not convert into results.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: autonomous AI scientists and their evaluation). Verified against: Opened https://arxiv.org/abs/2608.30047 on 2026-09-07; verbatim abstract read (P-/H-Creativity, AIDE and AIRA-Dojo, 10 MLE-Bench tasks), three authors, submitted 30 Aug 2026, comment confirms COLM 2026. Not in dedupe_titles.txt.

Critique / open questions

Queued — not yet assessed.

Key quotes

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. (abstract)

We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). (abstract)