Skip to content

NatureBench

external · status: active · focus: analysis · discipline: general · started: 2026

Project page: https://github.com/FrontisAI/NatureBench

Licence: MIT

Source: projects/landscape/naturebench.yml

Positioning

A cross-discipline benchmark (arXiv:2606.24530) of 90 tasks distilled from peer-reviewed Nature-family papers across six scientific domains, asking whether AI coding agents can match — or surpass — the published state of the art. Each task is a containerized package (task brief, the paper's dataset, a held-out test set with hidden ground truth, an automated evaluator) built by NatureGym, an automated Claude-Code-skills pipeline that converts a published paper into an executable Docker task. Sits in the RISE evaluation-infrastructure layer alongside AstaBench, MLGym, and EconCS Bench, but targets empirical scientific ML with executable, SOTA-anchored scoring.

Distinctive contribution

Scores agents against each source paper's reported SOTA (Surpass-SOTA rate) rather than mere reproduction, with an information firewall that strips the source method from the task brief so agents must discover solutions. Ships a full harness with built-in adapters for Claude Code, Codex CLI, and Gemini CLI, a post-hoc validity judge, and a public leaderboard: by September 2026 the leaderboard had grown to sixteen harness-model configurations (up from twelve at launch), with the best reaching a 23.3% Surpass-SOTA rate (57.8% Match-SOTA), with failures still dominated by method-selection errors.

Evaluation scores

Dimension Score (0–3) Note
Lifecycle coverage 0 Benchmark / evaluation infrastructure; produces task packages and scores, not scholarship.
Autonomy level 0 Evaluation harness — agency lives in the agents under test; NatureGym automates task construction but is a build pipeline, not a research agent.
Architectural transparency 3 Full code: harness, agent adapters, evaluators, post-hoc judge, NatureGym construction skills, docs, HuggingFace dataset, and public leaderboard.
Inputs supported 1 Single input form (containerized task package) with each task bundling the source paper's dataset; no broader literature or private-corpus access.
Outputs / reproducibility 2 Dockerized tasks, hidden test sets, versioned dataset, and persisted results directories; agent nondeterminism precludes exact reruns.
Internal evaluation 2 Systematic evaluation grown from twelve to sixteen harness-model configurations by September 2026, including externally submitted runs (e.g. HELIX + Claude Opus 5, Luria 1.0 + DeepSeek-V4-Pro) alongside the original failure analysis; still not yet peer-reviewed.
Openness 2 MIT for original work (NOTICE-scoped); third-party task data under heterogeneous per-task licenses; full runs need agent API keys and 24-80 GB GPUs.
Maturity / traction 2 Launched June 2026 (77 stars in the first month) and sustained since: a NatureBench-25 25-task track (Aug 28), a Harbor/Hugging Face task-conversion integration (Aug 31), and externally submitted leaderboard runs (Claude Opus 5, DeepSeek-V4-Pro, Qwen 3.8 Max, GLM-5.3 Flash) kept current through Sept 23, 2026 — external users and regular releases, though still short of sustained production adoption.
Cross-family policy 1 Post-hoc validity judge is configured independently of the executing agent, so cross-family judging is possible via config but neither default nor required.
Runtime assurance 1 Hidden-ground-truth evaluator plus a post-hoc validity judge screen scored outputs; no in-flight gating while the agent runs.
Cross-platform portability 2 Built-in adapters for Claude Code, Codex CLI, and Gemini CLI plus a documented custom-agent interface.

Scored on 2026-10-01. See the evaluation rubric.

Tags

Pipeline stages: data-analysis code-generation

Architectural features: tool-use

Inputs: nature-family-papers coding-agent-configurations

Outputs: containerized-task-packages surpass-sota-scores leaderboard

Data sources: paper-datasets huggingface-dataset

Knowledge sources: source-papers

Limitations

  • Heavy compute footprint: gpu_high tasks assume A100-class (80 GB) GPUs, so full benchmark runs are beyond commodity hardware.
  • The information firewall removes the source method from task briefs, but the underlying Nature-family papers are public, so pretraining contamination cannot be ruled out.
  • Third-party data bundled per task is governed by heterogeneous per-task license notices, complicating redistribution.

Papers describing this project

  • NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? — Wang, Y., Cheng, L., Zuo, Y., Zeng, S., He, B., Jiang, C., et al. (2026). arXiv. arXiv:2606.24530