Toward Generalist Autonomous Research via Hypothesis-Tree Refinement

Summary

Arbor is a framework for running the exploration-experimentation- abstraction loop of research autonomously over long horizons. A long-lived coordinator manages global strategy over a persistent Hypothesis Tree (Hypothesis Tree Refinement, HTR) that links hypotheses, artifacts, evidence, and distilled insights across time, while short-lived executors implement and test individual hypotheses in isolated worktrees. As results return, the tree is updated, reusable lessons are propagated, the search frontier is refined, and verified improvements are admitted. The system is evaluated under "Autonomous Optimization" (AO), in which an agent improves an initial research artifact by iterative experimentation without step-level human supervision. Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six, with more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and budget. On MLE-Bench Lite it reaches 86.36% Any Medal with GPT-5.5.

Contribution

Claimed: "generalist autonomous research" through a cumulative, tree-structured process rather than a sequence of local attempts. What the abstract supports: superior held-out gains over two general coding agents on six optimization tasks under matched budgets, and a strong MLE-Bench Lite score. The evaluation setting (AO) is artifact optimization against a measurable metric, which is narrower than research in the sense of producing knowledge claims; "generalist" is supported only across ML-engineering task types. No comparison with other AI-scientist systems appears in the abstract.

Method

Six research tasks with dev/held-out evaluation; baselines are Codex and Claude Code run through the same task interface and resource budget; plus MLE-Bench Lite with GPT-5.5. The abstract does not specify the number of runs or seeds, variance across runs, the base model used for the six tasks (GPT-5.5 is named only for MLE-Bench Lite), the criterion by which improvements are "verified" and admitted (the catalog page describes held-out-test merge gates), or any ablation separating the tree structure from the coordinator-executor split or from plain memory.

Relevance to RISE

Informs hypothesis-generation, research-design, code-generation, and data-analysis; it is the paper behind the catalog entry arbor, which notes there is no drafting or review stage and that the headline numbers are self-reported. Its nearest catalog neighbor is qu2026coral: both are experiment-driven optimization systems with persistent memory, but Arbor uses a coordinator-executor hierarchy over a tree while CORAL uses peer agents over shared memory, and Arbor benchmarks against coding agents while CORAL benchmarks against evolutionary search. The tree-of-experiments idea also echoes the manager-controlled agentic tree search of yamada2025aiscientistv2. For the multi-agent structure/aggregation question: the abstract's aggregation mechanism is hierarchical — a coordinator steering executors over a persistent hypothesis tree, with a held-out metric acting as the arbiter that "admits verified improvements" — and no debate, review, or adversarial role is described; epistemic quality is operationalized entirely as held-out gain.

Critique / open questions

A held-out metric is a sound arbiter for optimization but says nothing about whether the "distilled insights" propagated through the tree are true or transferable. Averaging relative gains across six tasks can be dominated by tasks with small baseline gains, and the abstract gives no dispersion. The baselines are general coding agents rather than research systems, so the comparison shows the value of the harness, not of Arbor over its direct peers. The MLE-Bench Lite figure lacks a same-model baseline in the abstract. The abstract concedes no limitations explicitly; its own definition of AO as an "operational setting" marks the gap between optimization and open-ended discovery.

Key quotes

"We introduce Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time." (abstract)

"As results return, Arbor updates the tree, propagates reusable lessons, refines the search frontier, and admits verified improvements." (abstract)

"Across six real research tasks in model training, harness engineering, and data synthesis, Arbor achieves the best held-out result on all six tasks, attaining more than 2.5x the average relative held-out gain of Codex and Claude Code under the same task interface and resource budget." (abstract)