TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents

Summary

Argues that existing AI-scientist benchmarks are configured for reproduction (recovering a hidden target study's result) rather than discovery, and builds 40 blind tasks from 40 peer-reviewed studies across 10 domains that expose only a neutral objective and frozen data. A fixed LLM judge scores evidentiary maturity on six dimensions via 29 artifact-grounded items with no per-instance human grading. Four coding agents on one frozen base model cluster in a narrow 58.4-60.3/100 plateau with no reliable separation: competent execution and documentation, but missing controls, robustness, falsifiability and cross-dataset generalization - the bottleneck is scientific judgment, not coding.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: autonomous AI scientists and their evaluation). Verified against: Opened https://arxiv.org/abs/2609.05079 on 2026-09-07; verbatim abstract read (40 blind tasks, six dimensions/29 items, 58.4-60.3 plateau), four authors, submitted 4 Sep 2026. Not in dedupe_titles.txt.

Critique / open questions

Queued — not yet assessed.

Key quotes

Autonomous coding agents are increasingly proposed as AI-scientist systems that conduct analyses and write research reports, but executing a prescribed analysis is not the same as making a discovery. (abstract)

Existing benchmarks are configured for reproduction: tasks, data, and rubrics are built around a hidden target study, and recovery of its result is rewarded. (abstract)