The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

Summary

Controlled audit of the internal workflow of two prominent open-source AI-scientist systems, isolating four failure modes: inappropriate benchmark selection, data leakage, metric misuse and post-hoc selection bias. Finds real instances of each across a severity spectrum that are easy to miss from the finished paper alone, and shows that trace logs plus code make detection far more effective. Concludes with a concrete policy recommendation that venues evaluating AI-generated research mandate submission of traces and code — directly actionable for an IS argument about verifiable research pipelines.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-08-29 literature sweep as a high-priority candidate for the RISE knowledge base (angle: autonomous AI scientists and their evaluation). Verified against: arXiv abstract page opened 2026-08-29 (title, 3 authors, submitted 10 Sep 2025, NeurIPS 2025 AI4Science Spotlight note and full abstract read)

Critique / open questions

Queued — not yet assessed.

Key quotes

AI scientist systems, capable of autonomously executing the full research workflow from hypothesis generation and experimentation to paper writing, hold significant potential for accelerating scientific discovery. (abstract)

However, the internal workflow of these systems have not been closely examined. (abstract)