HeurekaBench¶

external · status: active · focus: analysis · discipline: biomedical · started: 2025

Project page: https://github.com/mlbio-epfl/HeurekaBench

Licence: MIT

Source: projects/landscape/heureka-bench.yml

Positioning¶

A framework for constructing AI-co-scientist benchmarks rather than a fixed benchmark: a semi-automated pipeline uses multiple LLMs to mine validated insights out of published papers, reformulate them as open-ended research questions over the papers' own experimental datasets, and validate the resulting question-answer pairs against the published findings. The instantiation shipped in the repo, sc-HeurekaBench, does this for single-cell biology — 44 GB of datasets, six question-set variants (multiple-choice and open-ended, each in lite, full and tool-usage form), an LLM judge, and baseline runners for plain LLMs, CellVoyager and Biomni 0.0.6. Sits in the RISE evaluation-infrastructure layer alongside AstaBench, AIRS-Bench and LifeSciBench, but one level up: it is the recipe, not the target set.

Distinctive contribution¶

The only entry in the catalog that publishes the benchmark- construction pipeline itself (benchmark_creation/, benchmark_validation/, geval_prompts/), so the recipe — published finding plus the study's raw data becomes a graded open-ended agent task — is portable to any field with data-backed papers, including economics replication packages, rather than being locked to the one domain it was instantiated in. Its headline empirical claim is architectural rather than a ranking: adding a critic module improves ill-formed responses from open-weight LLM agents by up to 22%, isolating response well-formedness from research ability.

Evaluation scores¶

Dimension	Score (0–3)	Note
Lifecycle coverage	0	Evaluation infrastructure: the declared stages describe the tasks it constructs and grades, not scholarship it produces itself — no paper, code artifact, or finding leaves the framework.
Autonomy level	0	The construction pipeline is explicitly semi-automated (LLM generation with validation against published results) and the eval harness is a set of scripts; all research agency lives in the agents being measured.
Architectural transparency	3	Repo ships the construction pipeline, the validation stage, the G-Eval judge prompts (`geval_prompts/`), the answer-extraction and scoring scripts, and both baseline runners (`run_baselines/`, `run_biomni/`) — code and prompts, not just an artifact drop.
Inputs supported	2	Six question-set variants (mcq/oeq × lite/full/tool-usage) over 44 GB of real experimental data, runnable against three model paths (vLLM open-weights, closed APIs) and two agent adapters (CellVoyager, Biomni).
Outputs / reproducibility	3	As of 2026-09-14 the 44 GB sc-HeurekaBench dataset moved from six hand-reassembled Google Drive parts to a single versioned Hugging Face dataset repo (huggingface.co/datasets/sibasmarakp/sc-HeurekaBench), alongside fixed JSON question sets with a SHA256 checksum — a reproducible, versioned data manifest; judge-scored open-ended answers remain not bit-reproducible.
Internal evaluation	3	Accepted to ICLR 2026 (verified on the arXiv listing and the repo's own `@inproceedings` citation block) — peer-reviewed external validation, with baseline agents scored in the paper.
Openness	2	A LICENSE file is now present in the repo (MIT, copyright Panigrahi/Videnović/Brbić 2026; confirmed 2026-10-01, reversing the 'no license file' finding from 2026-09-08) — permissive license, but the 44 GB dataset's fragile six-part Google Drive distribution still limits full end-to-end reproducibility on commodity hardware.
Maturity / traction	1	23 stars, 1 fork, 14 commits as of 2026-11-01; light but ongoing maintenance (dataset migrated to Hugging Face 2026-09-14, README tidied 2026-10-30) rather than new features — still a single-lab research artifact with no leaderboard.
Cross-family policy	0	Model-agnostic evaluation target with no executor/reviewer pairing of its own; the construction pipeline is described as using multiple LLMs but the README does not document which families, so cross-family cannot be credited.
Runtime assurance	1	Generated questions are validated against the published findings at construction time and an optional critic module screens ill-formed agent answers — single-pass checks, with no gating on failure during an agent run.
Cross-platform portability	2	Runs against open-weight models via vLLM plus OpenAI/Anthropic APIs and two agent adapters, but the harness assumes a conda environment and the sc-HeurekaBench data layout.

Scored on 2026-11-01. See the evaluation rubric.

Tags¶

Pipeline stages: literature-synthesis hypothesis-generation research-design data-analysis code-generation

Architectural features: multi-agent tool-use

Inputs: source-papers experimental-datasets agent-implementation

Outputs: question-answer-sets evaluation-scores

Data sources: single-cell-datasets

Knowledge sources: source-papers

Limitations¶

A LICENSE file (MIT) has been confirmed present in the repo since 2026-09-12 — reuse terms are resolved and permissive.
The framework claim is domain-general but the only instantiation is single-cell biology; portability to economics or other fields is argued, not demonstrated.
The 44 GB dataset moved to a single Hugging Face dataset repo on 2026-09-14, resolving the earlier fragile six-part Google Drive distribution.
README does not state how many questions each of the six variants contains, so benchmark size cannot be verified from the repo.
No leaderboard; activity since the ICLR 2026 acceptance has been maintenance-only (license, hosting, README) rather than new benchmark content or baselines.
Open-ended grading depends on an LLM judge whose own accuracy is not separately validated against human raters in the repo.

Papers describing this project¶

HeurekaBench: A Benchmarking Framework for AI Co-scientist — Panigrahi, S. S., Videnović, J., Brbić, M. (2026). ICLR 2026. arXiv:2601.01678

panigrahi2026heurekabench (BibTeX)