ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction

Summary

Addresses the acceptance problem in building scientific agent benchmarks: LLM-proposed items are kept only if an executable solution script reproduces the answer in the real scientific software, randomized no-tool screening rejects items solvable from the prompt alone, and a tool-using agent solves the survivor within a time budget. Instantiated in FEniCSx, 500 generation attempts yield 128 unique protocol survivors. Turns benchmark construction into an auditable process rather than expert hand-curation, which matters for the credibility of every tool-use research-agent benchmark.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a medium-priority candidate for the RISE knowledge base (angle: research-agent benchmarks). Verified against: Opened https://arxiv.org/abs/2609.02067, read full abstract, 4 authors, 2026-09-02, cs.AI/cs.MS/cs.SE. Dedupe: no 'toolgate' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textbooks, published papers, and online resources. (abstract)

These routes can produce strong evaluations, but they require substantial per-item labor. (abstract)