AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

Summary

AstaBench is Ai2's benchmark suite for measuring how well AI agents perform scientific research. The abstract lists five shortcomings of existing agent benchmarks — no reproducible agent tools, no accounting for confounders such as model cost and tool access, no standardized interfaces for agent prototyping, no holistic product-informed measures of real-world use, and no comprehensive baselines — and responds with "principles and tooling" for more rigorous benchmarking. The suite comprises 2400+ problems spanning the scientific discovery process and multiple domains, many inspired by real user requests to deployed Asta agents. It ships with a research environment with production-grade search tools for controlled, reproducible evaluation, nine science-optimized classes of Asta agents, and numerous baselines. An evaluation of 57 agents across 22 agent classes leads the authors to conclude that AI "remains far from solving the challenge of science research assistance."

Contribution

Claimed: a set of benchmarking principles, a broad task suite, the first research environment with production-grade search tools, a family of science-optimized agents, and a large comparative evaluation. What the abstract actually supports: the scale claims (2400+ problems, 57 agents, 22 classes) and one headline qualitative finding. The "rigor" claims (reproducible tools, confounder accounting) are design intents stated in the abstract; how they are operationalized and whether they succeed cannot be judged from the abstract. No per-task or per-agent numbers are given.

Method

As far as the abstract states: 2400+ problems across the discovery process and several scientific domains, partly derived from user requests to Asta agents; a controlled environment with search tools; nine agent classes plus baselines; 57 agents in 22 classes evaluated. The abstract does not specify how problems are scored (rubric, exact match, LLM judge, human judgment), which domains are covered, how cost is measured or normalized, which underlying models the agents use, or whether any scores were validated against human performance. The arXiv v2 comment (April 2026) states the paper was published as a conference paper at ICLR 2026.

Relevance to RISE

The abstract names literature reviews, replicating experiments, data analysis and proposing new directions as the capabilities benchmarked, so the suite informs the literature-discovery, literature-synthesis, replication, data-analysis and hypothesis-generation stages. Catalog slug: asta-bench, which the catalog places in the evaluation-infrastructure layer alongside aviary and mlgym. Relative to airs-bench (20 tasks from ML papers) and naturebench (90 tasks from Nature-family papers), AstaBench's abstract claims breadth across "the entire scientific discovery process" rather than SOTA-anchored execution on a narrow task set. For the ISR question of how multi-agent structure and aggregation rules shape epistemic quality, the abstract describes no debate, tournament or review mechanism inside the benchmark itself, but its explicit accounting for "model cost and tool access" as confounders is exactly what a fair comparison of aggregation structures at fixed compute requires, making AstaBench a candidate outcome measure for such comparisons.

Critique / open questions

From the abstract one cannot assess what "holistic" means operationally, how the nine Asta agent classes differ structurally, whether problems inspired by Asta user requests bias the suite toward Ai2's own product, or how valid the scorers are. The "first" claim for the search environment is not verifiable here. The abstract concedes only the headline limitation — that AI is far from solving science research assistance — and does not name limitations of the suite itself. Whether the 22 agent classes include multi-agent configurations, and whether structure (rather than model) explains performance differences, is not stated.

Key quotes

"Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances." (abstract)

"Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents." (abstract)

"Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance." (abstract)