Projects catalog¶
This catalog evaluates agentic-research systems against the
standard rubric.
Vocabularies for stages, architectural features, focus, and
disciplinary scope are defined in
projects/VOCABULARY.md.
The matrix and per-project pages below are auto-generated from
projects/*.yml and projects/landscape/*.yml by
scripts/build_indexes.py. Do not edit by hand — edit the YAML
sources.
Comparison matrix¶
| Project | Type | Focus | LC | AUT | ARC | IN | OUT | EVAL | OPEN | MAT | XF | RUN | PORT | Discipline |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E2ER — End-to-End Research | owned | end-to-end |
3 | 2 | 2 | 3 | 2 | 1 | 2 | 1 | 0 | 2 | 1 | economics |
| AARRI-Bench | external | end-to-end |
0 | 0 | 3 | 1 | 2 | 2 | 1 | 1 | 0 | 1 | 2 | computer-science |
| Academic Research Skills (ARS) | external | end-to-end |
2 | 1 | 3 | 3 | 3 | 3 | 2 | 3 | 1 | 3 | 3 | general |
| academic-writing-agents | external | revision |
2 | 1 | 3 | 1 | 2 | 0 | 2 | 1 | 0 | 2 | 0 | general |
| AgentEconomist | external | analysis |
2 | 1 | 1 | 1 | 2 | 2 | 0 | 1 | 0 | 1 | 0 | economics |
| Agent Laboratory | external | end-to-end |
3 | 2 | 3 | 2 | 2 | 2 | 3 | 0 | 0 | 1 | 1 | computer-science |
| Agon | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 1 | 2 | 1 | 1 | 2 | 2 | general |
| AI Co-Mathematician (Google DeepMind) | external | end-to-end |
2 | 2 | 1 | 2 | 2 | 2 | 0 | 1 | 0 | 2 | 0 | general |
| AI Peer Review (poldrack) | external | review |
0 | 2 | 3 | 0 | 2 | 0 | 2 | 2 | 2 | 1 | 1 | general |
| ai-peer-review-skill | external | review |
0 | 2 | 3 | 0 | 2 | 0 | 2 | 1 | 0 | 1 | 0 | general |
| AI Research Feedback | external | review |
1 | 2 | 3 | 1 | 1 | 0 | 2 | 2 | 0 | 1 | 0 | economics |
| AI-Researcher (HKUDS) | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 3 | 1 | 2 | 0 | 1 | 2 | computer-science |
| AIRS-Bench (Meta FAIR) | external | end-to-end |
0 | 0 | 3 | 2 | 2 | 2 | 1 | 2 | 0 | 1 | 2 | computer-science |
| AlphaEvolve (Google DeepMind) | external | end-to-end |
1 | 3 | 1 | 1 | 2 | 3 | 0 | 2 | 0 | 3 | 0 | mathematics |
| Project APE | external | end-to-end |
3 | 3 | 3 | 2 | 3 | 3 | 3 | 1 | 1 | 2 | 1 | economics |
| Arbor | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 2 | 3 | 2 | 0 | 2 | 3 | general |
| ARIS (Auto-Research-In-Sleep) | external | end-to-end |
3 | 3 | 3 | 3 | 2 | 3 | 3 | 3 | 2 | 3 | 3 | computer-science |
| ARK (idea2paper) | external | end-to-end |
2 | 2 | 2 | 1 | 2 | 1 | 2 | 1 | 0 | 1 | 1 | computer-science |
| Asta AutoDiscovery | external | ideation |
1 | 3 | 3 | 1 | 2 | 3 | 2 | 2 | 0 | 1 | 1 | general |
| AstaBench (AI2) | external | end-to-end |
0 | 2 | 3 | 3 | 3 | 3 | 3 | 3 | 1 | 1 | 2 | general |
| Auto-Empirical Research Skills (AERS) | external | end-to-end |
3 | 2 | 3 | 2 | 2 | 2 | 2 | 2 | 0 | 1 | 3 | social-sciences |
| Automated Alignment Researchers (Anthropic) | external | analysis |
2 | 3 | 3 | 1 | 2 | 2 | 2 | 1 | 0 | 2 | 2 | computer-science |
| AutoRA (Automated Research Assistant) | external | end-to-end |
2 | 3 | 3 | 1 | 2 | 2 | 3 | 2 | 0 | 1 | 1 | social-sciences |
| AutoResearchClaw | external | end-to-end |
3 | 2 | 2 | 3 | 2 | 2 | 3 | 3 | 1 | 3 | 3 | general |
| AutoSurvey | external | literature |
1 | 3 | 2 | 1 | 2 | 3 | 1 | 0 | 0 | 1 | 1 | general |
| Aviary (FutureHouse) | external | end-to-end |
0 | 2 | 3 | 2 | 3 | 2 | 3 | 2 | 1 | 1 | 2 | general |
| BixBench3 | external | analysis |
0 | 0 | 3 | 1 | 2 | 2 | 1 | 1 | 1 | 2 | 1 | biomedical |
| Claude Science | external | end-to-end |
2 | 2 | 1 | 3 | 3 | 1 | 1 | 2 | 0 | 2 | 1 | general |
| Clo-Author | external | end-to-end |
3 | 2 | 3 | 2 | 2 | 2 | 1 | 1 | 0 | 2 | 1 | economics |
| Coarse (coarse.ink) | external | review |
0 | 2 | 2 | 2 | 1 | 2 | 3 | 1 | 1 | 1 | 2 | general |
| CORAL | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 2 | 3 | 2 | 1 | 2 | 2 | general |
| CORE-Bench | external | replication |
0 | 2 | 3 | 1 | 2 | 3 | 2 | 3 | 0 | 1 | 2 | general |
| DARE (De-Anthropocentric Research Engine) | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 0 | 2 | 2 | 0 | 2 | 2 | general |
| data-to-paper | external | end-to-end |
2 | 3 | 3 | 2 | 3 | 3 | 3 | 0 | 0 | 2 | 1 | general |
| DeepResearch Bench | external | end-to-end |
0 | 0 | 3 | 1 | 2 | 2 | 2 | 2 | 0 | 0 | 1 | general |
| DeepResearcher (GAIR-NLP) | external | literature |
1 | 3 | 3 | 2 | 2 | 3 | 3 | 2 | 0 | 2 | 1 | general |
| DeepScientist | external | end-to-end |
2 | 3 | 2 | 2 | 2 | 3 | 2 | 2 | 1 | 1 | 2 | computer-science |
| EconAgentSkills | external | analysis |
2 | 1 | 3 | 2 | 2 | 0 | 2 | 1 | 0 | 1 | 2 | economics |
| Econ Paper Review Skill (econ-review) | external | review |
1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 0 | 2 | 1 | economics |
| econ-skills (zbsaygin) | external | analysis |
2 | 1 | 3 | 2 | 2 | 0 | 2 | 1 | 0 | 2 | 0 | economics |
| Econ Writing Skill | external | drafting |
2 | 0 | 3 | 1 | 1 | 1 | 3 | 2 | 0 | 1 | 1 | economics |
| EconCS Bench | external | end-to-end |
0 | 0 | 3 | 1 | 0 | 1 | 1 | 1 | 0 | 0 | 3 | economics |
| EconTools | external | review |
2 | 1 | 3 | 2 | 2 | 1 | 2 | 1 | 1 | 2 | 1 | economics |
| EvoScientist | external | end-to-end |
3 | 3 | 3 | 2 | 2 | 3 | 3 | 3 | 1 | 2 | 3 | general |
| FAROS (Foundation AutoResearch Operating System) | external | end-to-end |
3 | 2 | 3 | 2 | 2 | 2 | 1 | 2 | 0 | 3 | 1 | computer-science |
| ForeSci | external | ideation |
0 | 0 | 3 | 1 | 1 | 2 | 1 | 1 | 0 | 0 | 2 | computer-science |
| AI Co-Scientist (Google DeepMind) | external | ideation |
2 | 3 | 1 | 2 | 1 | 3 | 1 | 3 | 0 | 3 | 0 | general |
| GPT Researcher | external | literature |
1 | 3 | 3 | 2 | 2 | 1 | 3 | 3 | 0 | 1 | 2 | general |
| HeurekaBench | external | analysis |
0 | 0 | 3 | 2 | 3 | 3 | 2 | 1 | 0 | 1 | 2 | biomedical |
| InnovatorBench | external | end-to-end |
0 | 0 | 3 | 1 | 2 | 2 | 1 | 1 | 0 | 1 | 1 | computer-science |
| autoresearch (Karpathy) | external | analysis |
1 | 2 | 2 | 0 | 1 | 1 | 1 | 2 | 0 | 1 | 2 | computer-science |
| Kosmos (jimmc414 implementation) | external | end-to-end |
2 | 3 | 3 | 2 | 2 | 2 | 1 | 2 | 1 | 2 | 1 | general |
| LifeSciBench | external | end-to-end |
0 | 0 | 1 | 2 | 0 | 2 | 0 | 1 | 0 | 0 | 1 | biomedical |
| LitReview Arena | external | literature |
0 | 0 | 3 | 1 | 2 | 3 | 2 | 1 | 0 | 0 | 2 | computer-science |
| Luxas | external | end-to-end |
3 | 3 | 3 | 1 | 2 | 1 | 2 | 2 | 1 | 3 | 2 | general |
| MARG (Multi-Agent Review Generation) | external | review |
0 | 2 | 3 | 1 | 3 | 2 | 3 | 1 | 0 | 1 | 0 | general |
| mcp-stata (Stata agentic toolkit) | external | analysis |
1 | 0 | 3 | 1 | 2 | 1 | 1 | 2 | 0 | 1 | 3 | economics |
| MLE-bench (OpenAI) | external | analysis |
0 | 0 | 3 | 2 | 2 | 3 | 2 | 3 | 0 | 1 | 2 | computer-science |
| MLGym (Meta) | external | end-to-end |
0 | 2 | 3 | 2 | 3 | 2 | 2 | 0 | 0 | 1 | 1 | computer-science |
| nano-scientist | external | end-to-end |
3 | 3 | 2 | 1 | 2 | 1 | 1 | 1 | 1 | 2 | 2 | general |
| NatureBench | external | analysis |
0 | 0 | 3 | 1 | 2 | 2 | 2 | 2 | 1 | 1 | 2 | general |
| Open CoScientist Agents | external | ideation |
1 | 3 | 3 | 2 | 1 | 1 | 3 | 1 | 3 | 2 | 1 | general |
| OpenScholar (AI2) | external | literature |
0 | 2 | 3 | 2 | 2 | 3 | 3 | 3 | 1 | 1 | 1 | general |
| OpenEcon Data | external | analysis |
0 | 0 | 2 | 1 | 1 | 0 | 1 | 2 | 0 | 1 | 2 | economics |
| OpenScience | external | end-to-end |
2 | 2 | 3 | 3 | 2 | 1 | 2 | 2 | 1 | 1 | 2 | general |
| p-hacking-skills (phack) | external | analysis |
0 | 0 | 3 | 1 | 3 | 2 | 3 | 1 | 0 | 3 | 2 | economics |
| pAI-Econ-claude | external | end-to-end |
2 | 1 | 3 | 1 | 2 | 2 | 2 | 2 | 0 | 3 | 0 | economics |
| PaperOrchestra | external | drafting |
2 | 2 | 3 | 2 | 2 | 2 | 2 | 1 | 0 | 1 | 1 | computer-science |
| PaperQA2 (FutureHouse) | external | literature |
0 | 2 | 3 | 2 | 2 | 3 | 3 | 3 | 1 | 3 | 3 | general |
| Paper-WorkFlow | external | end-to-end |
3 | 2 | 2 | 2 | 3 | 1 | 2 | 1 | 0 | 2 | 2 | economics |
| PaperCoder (Paper2Code) | external | replication |
1 | 3 | 3 | 2 | 3 | 3 | 3 | 3 | 0 | 2 | 2 | computer-science |
| PaperBench (OpenAI) | external | replication |
0 | 0 | 3 | 2 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | computer-science |
| PaperDoctor | external | review |
1 | 2 | 3 | 2 | 1 | 2 | 2 | 1 | 0 | 2 | 0 | general |
| PARNESS | external | end-to-end |
3 | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 2 | 2 | general |
| Prism | external | drafting |
2 | 1 | 1 | 2 | 2 | 0 | 1 | 2 | 0 | 1 | 0 | general |
| ralph-wiggum-asset-pricing | external | drafting |
2 | 2 | 3 | 2 | 2 | 1 | 1 | 1 | 1 | 3 | 1 | finance |
| RD-Agent (R&D-Agent) | external | analysis |
2 | 3 | 3 | 2 | 2 | 3 | 2 | 3 | 1 | 2 | 2 | general |
| RECAST (Replication and Extension with Causal AI Statistical Toolkit) | external | replication |
2 | 2 | 1 | 2 | 2 | 2 | 1 | 1 | 0 | 3 | 1 | econometrics |
| RECLAIM | external | replication |
1 | 3 | 3 | 1 | 3 | 2 | 2 | 1 | 0 | 2 | 2 | computer-science |
| Refine (refine.ink) | external | review |
0 | 2 | 1 | 1 | 1 | 2 | 0 | 3 | 0 | 1 | 0 | general |
| ReplicatorBench / ReplicatorAgent | external | replication |
2 | 2 | 3 | 2 | 2 | 2 | 2 | 1 | 0 | 1 | 1 | social-sciences |
| REPRO-Bench | external | replication |
0 | 2 | 1 | 2 | 1 | 3 | 1 | 2 | 0 | 0 | 1 | social-sciences |
| ReproAgent | external | replication |
1 | 3 | 3 | 1 | 2 | 3 | 2 | 1 | 1 | 2 | 1 | computer-science |
| ReproRepo | external | replication |
0 | 0 | 3 | 1 | 1 | 3 | 2 | 1 | 1 | 1 | 2 | computer-science |
| Claude Code Research Harness Workflow | external | analysis |
2 | 2 | 3 | 2 | 3 | 2 | 1 | 1 | 0 | 2 | 0 | social-sciences |
| Research Paper Writing Skills | external | drafting |
1 | 0 | 3 | 1 | 1 | 0 | 3 | 2 | 0 | 1 | 3 | computer-science |
| ResearchTown | external | ideation |
2 | 3 | 3 | 2 | 2 | 2 | 3 | 0 | 0 | 1 | 1 | general |
| ResearchAgent (NAACL 2025) | external | ideation |
1 | 2 | 3 | 2 | 2 | 2 | 1 | 0 | 0 | 2 | 0 | general |
| ResearchClawBench | external | end-to-end |
0 | 0 | 3 | 2 | 2 | 2 | 2 | 2 | 0 | 1 | 3 | general |
| ResearchGym | external | end-to-end |
0 | 0 | 3 | 2 | 2 | 2 | 1 | 1 | 1 | 2 | 1 | computer-science |
| Reviewer (Ingar30) | external | review |
0 | 2 | 3 | 1 | 2 | 1 | 3 | 1 | 0 | 2 | 1 | economics |
| Robin (FutureHouse) | external | end-to-end |
2 | 2 | 3 | 2 | 1 | 2 | 2 | 2 | 1 | 2 | 2 | biomedical |
| Sakana AI Scientist v2 | external | end-to-end |
2 | 3 | 3 | 1 | 2 | 3 | 1 | 3 | 0 | 1 | 0 | computer-science |
| Sakana AI Scientist (v1) | external | end-to-end |
2 | 3 | 3 | 1 | 2 | 3 | 1 | 3 | 0 | 1 | 0 | computer-science |
| Sakana AI Agentic Scientist (AI-Scientist-v3) | external | end-to-end |
2 | 2 | 3 | 1 | 2 | 1 | 2 | 1 | 1 | 2 | 1 | computer-science |
| ScholarEval | external | review |
1 | 2 | 3 | 1 | 1 | 2 | 2 | 1 | 1 | 2 | 2 | general |
| Scholar Loop | external | end-to-end |
2 | 3 | 3 | 1 | 2 | 1 | 3 | 1 | 0 | 3 | 0 | computer-science |
| Social Science Replicability Infrastructure | external | replication |
1 | 2 | 2 | 2 | 2 | 2 | 3 | 1 | 0 | 2 | 1 | social-sciences |
| SocSci-Repro-Bench | external | replication |
0 | 0 | 2 | 1 | 0 | 2 | 2 | 1 | 0 | 0 | 3 | social-sciences |
| stata-code | external | analysis |
1 | 0 | 3 | 1 | 2 | 2 | 2 | 1 | 0 | 2 | 2 | economics |
| Stata MCP | external | analysis |
1 | 0 | 2 | 1 | 2 | 1 | 2 | 2 | 0 | 1 | 2 | economics |
| StatsPAI | external | analysis |
0 | 0 | 3 | 1 | 2 | 3 | 3 | 2 | 0 | 1 | 2 | economics |
| STORM / Co-STORM | external | literature |
1 | 2 | 3 | 2 | 2 | 2 | 3 | 3 | 0 | 1 | 2 | general |
| SurveyX | external | literature |
1 | 3 | 2 | 1 | 1 | 2 | 1 | 0 | 0 | 1 | 1 | general |
| SwarmResearch | external | end-to-end |
1 | 3 | 2 | 0 | 2 | 2 | 1 | 1 | 1 | 1 | 2 | computer-science |
| Terminal-Bench-Science | external | analysis |
1 | 0 | 3 | 1 | 1 | 2 | 3 | 2 | 0 | 0 | 2 | general |
| Tongyi DeepResearch | external | literature |
1 | 3 | 3 | 2 | 2 | 3 | 3 | 3 | 0 | 1 | 2 | general |
| ToolUniverse | external | end-to-end |
0 | 2 | 3 | 3 | 2 | 2 | 3 | 2 | 1 | 2 | 2 | biomedical |
| Writing-Driven Autoresearch | external | end-to-end |
2 | 3 | 3 | 0 | 2 | 2 | 2 | 0 | 0 | 3 | 1 | computer-science |
| zeropaper (Auto AI Research Template) | external | end-to-end |
3 | 3 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 3 | 1 | finance |
| Claude Code Skills for Empirical Research (zirui-song) | external | analysis |
2 | 1 | 3 | 2 | 2 | 1 | 2 | 1 | 1 | 3 | 1 | economics |
| Zochi (Intology) | external | end-to-end |
3 | 3 | 2 | 2 | 2 | 3 | 2 | 2 | 0 | 2 | 1 | computer-science |
Score columns: LC = lifecycle coverage, AUT = autonomy, ARC = architectural transparency, IN = inputs supported, OUT = outputs/reproducibility, EVAL = internal evaluation, OPEN = openness, MAT = maturity/traction, XF = cross-family policy, RUN = runtime assurance, PORT = cross-platform portability. Scale 0–3. See the evaluation rubric.
One-line summaries¶
- E2ER — End-to-End Research — E2ER is a strategist-driven agentic research pipeline that takes a research idea (human- or agent-supplied) and carries it through literature synthesis, identification, data acquisition, analysis, and paper drafting.
- AARRI-Bench — "Act As a Real Research Intern" (arXiv:2606.07462) — 82 containerized scenarios in standardized Harbor task format, each with an assertion-based verifier, probing whether LLM agents show the professionalism of human researchers in granular research situations (citation integrity, ablation-completeness audits, dead-end recognition, contradictory-advisor merging) rather than end-to-end execution.
- Academic Research Skills (ARS) — A comprehensive Claude Code plugin suite (v3.9.0 at scoring date) for the academic research pipeline: literature → write → review → revise → finalize.
- academic-writing-agents — A Claude Code plugin that puts a manuscript-audit orchestrator over twelve specialist agents: five read-only reviewers (consistency-checker, logic-reviewer, technical-reviewer, writing-reviewer, latex-layout-auditor), a bibliography-auditor, two web-enabled research agents (research-analyst for related work, novelty and positioning; brainstormer for alternative framings), a paper-crawler over the DBLP and OpenAlex APIs, and three agents that modify the source (prose-polisher, section-drafter, latex-figure-specialist).
- AgentEconomist — An interactive research copilot (arXiv:2604.27725) that turns a human researcher's abstract economic intuition into a runnable agent-based computational experiment.
- Agent Laboratory — An end-to-end autonomous research workflow (arXiv:2501.04227) that guides a research idea through three phases — literature review, experimentation, and report writing — with specialized LLM-driven agents and external tools (arXiv, Hugging Face, Python, LaTeX).
- Agon — A Claude Code plugin that drives a research project from a one-line topic through idea → proposal → running experiments with no human-written experimental code.
- AI Co-Mathematician (Google DeepMind) — A closed, agentic multi-agent workbench (arXiv:2605.06651) built on Gemini 3.1 for open-ended mathematics research.
- AI Peer Review (poldrack) — A CLI tool for AI-assisted meta-review of scientific papers: submits a manuscript to up to six proprietary LLMs (GPT-4o, GPT-4o-mini, Claude 3.7 Sonnet, Gemini 2.5 Pro, DeepSeek R1, Llama 4 Maverick) for independent review, then synthesizes a meta-review plus a concerns-by-reviewer matrix.
- ai-peer-review-skill — A single-purpose Claude Code skill sitting squarely in the referee-simulation stage: hand it a manuscript (PDF, DOCX, TXT or MD) and it extracts the text, spawns N parallel Claude subagents (default 5, named with NATO codenames alfa through echo) that each return an independent structured review — summary, major concerns, minor concerns, verdict — with one slot optionally given to an AI-Alignment-Forum-style critic red-teaming narrative, novelty, baselines, ablations and reproducibility, then synthesises a meta-review separating shared from unique concerns and issuing a final verdict.
- AI Research Feedback — Ten Claude Code skills that referee economics and finance research artifacts before they leave the author's hands.
- AI-Researcher (HKUDS) — An autonomous multi-agent system for ML research (arXiv 2505.18705, NeurIPS 2025 spotlight) that takes either a detailed idea (Level 1) or only a set of reference papers (Level 2) and runs resource collection from arXiv, GitHub and Hugging Face, idea generation, algorithm design, implementation and experiments inside a Docker container, an iterative validate-and-refine cycle, and hierarchical paper writing.
- AIRS-Bench (Meta FAIR) — A benchmark (arXiv:2602.06855) quantifying the end-to-end AI research abilities of LLM agents: 20 tasks sourced from 17 state-of-the-art ML papers across language modeling, code generation, mathematics, biochemical modeling, and time-series forecasting.
- AlphaEvolve (Google DeepMind) — A Gemini-powered evolutionary coding agent that combines LLM generative capabilities with automated evaluators in an iterative propose-test-refine loop.
- Project APE — An autonomous system that generates empirical economic policy research papers end-to-end from publicly available data, then scores them via a TrueSkill tournament in which AI-generated papers compete head-to-head against peer-reviewed human benchmarks from AER and AEJ:Policy (judged by Gemini 3.1 Flash Lite).
- Arbor — A generalist autonomous research agent (arXiv:2606.11926) from the WebThinker group at Renmin University: given a goal, a benchmark directory, and a metric, a Coordinator grows a persistent hypothesis tree ("Idea Tree") while Executors implement each idea in isolated git worktrees, run real experiments on a dev split, and merge only gains that survive a held-out test split.
- ARIS (Auto-Research-In-Sleep) — An open-source research harness for autonomous ML research (arXiv:2605.03042) built around cross-model adversarial collaboration: an executor model drives forward progress while a reviewer from a different model family critiques intermediate artifacts and requests revisions.
- ARK (idea2paper) — A CS-research pipeline ("Automatic Research Kit") that orchestrates six specialized agents — Researcher, Reviewer, Planner, Writer, Experimenter, Coder — through a three-phase Research / Dev / Review loop to turn a research idea into a submission-formatted LaTeX paper with DBLP-verified references, controllable via CLI, a web dashboard, or Telegram.
- Asta AutoDiscovery — Ai2's autonomous data-driven discovery agent (formerly AutoDS; relaunched inside AstaLabs on 2026-02-12): pointed at a structured dataset, it generates natural-language hypotheses, proposes experiment plans, writes and executes Python analyses — up to 500 experiments in a session — and ranks the resulting findings by Bayesian surprise, the shift from the LLM's prior to posterior belief in each hypothesis.
- AstaBench (AI2) — An evaluation framework from AI2 for measuring scientific-research abilities of AI agents.
- Auto-Empirical Research Skills (AERS) — A Claude-plugin-structured mega-catalog of agent skills for empirical social-science research: 78 collections / 1,151 vendored skills — 7 first-party Stanford REAP × CoPaper.AI collections (including the StatsPAI causal engine and the Paper-WorkFlow meta-orchestrator) plus curated, security-audited community collections — spanning topic refinement, literature review, data acquisition, identification strategy, estimation (Python/Stata/R), robustness audit, publication tables, writing, review simulation, AI-trace removal, and journal submission.
- Automated Alignment Researchers (Anthropic) — Anthropic's open-sourced sandbox and agent harness for automating a specific slice of alignment research — weak-to-strong generalization.
- AutoRA (Automated Research Assistant) — A Python framework for closing the empirical research loop in the behavioural and brain sciences: an experimentalist proposes novel experimental conditions, an experiment runner collects the corresponding observations from participants or a synthetic generator, and a theorist fits a model that explains the data — then the cycle repeats with the new evidence, all coordinated through a serializable
StandardStateobject. - AutoResearchClaw — An autonomous research pipeline taking a chat-level idea to a full paper via ACP-compatible agent back-ends (Claude Code, Codex CLI, Copilot CLI, Gemini CLI, Kimi CLI).
- AutoSurvey — A NeurIPS 2024 framework (arXiv:2406.10252) for automatically generating comprehensive literature surveys from a topic and a paper database.
- Aviary (FutureHouse) — A gymnasium for defining custom language-agent environments (arXiv:2412.21154), with pre-built environments for math, general knowledge, biological sequences, scientific literature search, and protein stability.
- BixBench3 — Twenty computational-biology tasks at the scale of a whole research study, framed explicitly as delegation rather than autonomy: the scientist keeps the research question and the high-level method choice, and the agent implements every analysis step from the raw data of a published study up to a set of exactly specified output artifacts (138 artifacts across the 20 tasks).
- Claude Science — Anthropic's AI workbench for researchers ("Claude Code for science"), announced 2026-06-30: a desktop app (macOS, Linux, and, natively since v0.1.47 on 2026-09-10, Windows) in which a generalist coordinating agent with 60+ curated skills and connectors spawns specialist agents to run literature analysis, database queries, multistep data analyses, figure iteration, and manuscript drafting, with compute spanning the local machine, HPC clusters via SSH, and Modal GPUs.
- Clo-Author — A Claude Code scaffold for empirical economics research, spanning literature review through journal submission.
- Coarse (coarse.ink) — A web-based AI peer-review service: users upload academic papers (up to 50 MB) and receive AI-generated referee reports with 20+ detailed comments.
- CORAL — Infrastructure (arXiv:2604.01658) for multi-agent autonomous self-evolution — organizations of AI agents that run experiments, share knowledge through persistent stores, and continuously improve solutions against a user-supplied grading script.
- CORE-Bench — A computational-reproducibility benchmark: 270 tasks built from 90 published papers (Code Ocean capsules) in computer science, social science and medicine, written in Python or R, each asking an agent to reproduce the reported results and then answer written and vision-based questions about the output into a supplied
report.json. - DARE (De-Anthropocentric Research Engine) — A pure-markdown research-skill corpus, substantially rearchitected in a "v4" rewrite merged 2026-09-23: what had been a four-layer hierarchy of 900+ skills (Campaign -> Strategy -> Tactic -> SOP) grouped into ten packages is now a two-layer research graph of 267 nodes — 51 Tactics (complete research transformations) composed from 216 SOPs (single-purpose steps) — reached through four product shells (entry, catalog, write-spec, execute-spec); the old
cli/anddsh-plugin/components were dropped in the same merge. - data-to-paper — An end-to-end framework that takes annotated data and produces backward-traceable scientific manuscripts: every numeric value in the output can be click-traced to the specific code line that generated it.
- DeepResearch Bench — An ICLR 2026 benchmark of 100 PhD-level research tasks (50 Chinese, 50 English) across 22 fields, written by domain experts to match real user query distributions, for evaluating "Deep Research Agent" (DRA) systems — i.e., web-search-and-synthesis agents rather than wet-lab or formal-modeling systems.
- DeepResearcher (GAIR-NLP) — An end-to-end RL-trained deep-research agent (arXiv:2504.03160) that learns to plan, retrieve, cross-validate, and self-reflect via reinforcement learning in real-world web environments rather than in simulated retrieval.
- DeepScientist — A local-first autonomous research studio for ML/AI work: you hand it a paper, a repository, or a bare research objective, and it reproduces a baseline, then runs long-horizon hypothesise → verify → analyse cycles under Bayesian optimisation, accumulating findings across sessions and ending in figures, reports and a LaTeX/PDF draft.
- EconAgentSkills — Ten opinionated agent skills covering the applied-econometrics workbench in four stages: data (working-with-data, api-data-fetcher, stata-data-cleaning), analysis (r-econometrics, stata-regression, python-panel-data), writing (academic-paper-writer, latex-tables) and communication (econ-visualization, beamer-presentation).
- Econ Paper Review Skill (econ-review) — A referee-simulation Claude Code/Codex skill for economics (and adjacent social-science) manuscripts: it reads a paper, reconstructs its argument and evidence chain, checks quotations, calculations and citations against the source and against live literature, and produces a typeset referee report plus a prioritized, agent-ready revision plan.
- econ-skills (zbsaygin) — A personal Claude Code skill pack for academic economics research workflows: reading papers by citekey, multi-agent literature review with novelty assessment, a Delphi-consensus correctness audit of math and econometrics, Econometrica-style notation cleanup, and LaTeX document/Beamer-slide/figure production.
- Econ Writing Skill — A portable agent skill (
skills/econ-write/) that packages economics paper-writing craft — section formulas, identification-strategy-aware phrasing, LaTeX conventions, and a pre-submission checklist — distilled from 50+ guides by named economists (Cochrane, McCloskey, Shapiro, Head, Bellemare, Goldin, Kremer). - EconCS Bench — A benchmark suite of open research challenges in Economics and Computation (EconCS), associated with the AI-Driven Research in EconCS workshop at EC 2026.
- EconTools — A personal Claude Code skill pack for economic-history and economics research workflows: simulated peer review with named referee personas (Diebolt), Economic History Review journal house-style automation (EHRstyle), a research-idea sounding board modelled on a real doctoral mentor (Janluiten), a dual-model adversarial reference auditor (Kris), a LaTeX-Word round-trip for language editing (Tanniedi), and a PDF-to-markdown literature converter (Tyler).
- EvoScientist — A self-evolving AI scientist system (arXiv:2603.08127) built on the DeepAgents framework.
- FAROS (Foundation AutoResearch Operating System) — A self-hosted, web-app multi-agent system spanning the full research lifecycle: topic coaching → literature-gated idea selection → a structured
PlanPackage(hypothesis, variables, acceptance criteria, evidence citations) → sandboxed code generation and execution → experiment/metric tracking → paper drafting with citation management and LaTeX/PDF output → ReviewX, a claim-evidence-experiment consistency auditor. - ForeSci — A temporally-controlled benchmark of 500 tasks testing whether LLM agents can make forward-looking research judgments — direction forecasting, bottleneck/opportunity discovery, strategic research planning, and venue-aware positioning — from a frozen, cutoff-dated knowledge base (~7,500 documents across four fast-moving AI subfields: LLM agents, fine-tuning, RAG systems, visual generative modeling), with post-cutoff papers held out purely for validation.
- AI Co-Scientist (Google DeepMind) — Google's closed multi-agent research partner (announced Feb 2025, published in Nature 2026-05-19) that generates, debates, and evolves novel research hypotheses.
- GPT Researcher — An autonomous "deep research" agent that produces long-form, cited reports on any topic from web and local sources.
- HeurekaBench — A framework for constructing AI-co-scientist benchmarks rather than a fixed benchmark: a semi-automated pipeline uses multiple LLMs to mine validated insights out of published papers, reformulate them as open-ended research questions over the papers' own experimental datasets, and validate the resulting question-answer pairs against the published findings.
- InnovatorBench — An ICLR 2026 benchmark-platform pair (arXiv:2510.27598) for end-to-end innovative LLM research: 20 tasks built from the open-source code of 14 published papers, spanning six research areas — data construction, data filtering, data augmentation, loss design, reward design, and scaffold construction.
- autoresearch (Karpathy) — The origin point of the "autoresearch" category rather than a research tool: a ten-file scaffold in which an AI coding agent repeatedly edits train.py, trains a single-GPU nanochat language model for exactly five minutes, scores the result by validation bits-per-byte, and keeps or discards the change.
- Kosmos (jimmc414 implementation) — An open-source implementation of the Kosmos AI scientist architecture (Lu et al., arXiv:2511.02824), adapted to run via Claude Code or the Anthropic / OpenAI APIs.
- LifeSciBench — OpenAI's expert-authored benchmark (announced 2026-06-17) for measuring how well AI models support real-world life-science research.
- LitReview Arena — A benchmark and evaluation platform for literature-review-writing agents: it collects blind pairwise ("battle-style") expert preferences between AI-generated and human/baseline literature-review drafts across five dimensions (coverage, claim support, structure, research suggestions, overall utility), then converts them into a reproducible offline benchmark.
- Luxas — An unattended question-to-manuscript harness: a Claude Opus "brain" agent coordinates 13 specialised sub-agent kinds (search, reader, worker, experiment, tool_impl, tool_review, experiment_reviewer, math, illustrator, illustrator_write, typesetter, reviewer, fixer) through literature discovery over OpenAlex/arXiv/CrossRef, experiment design, parallel implementation and testing, figure generation, LaTeX writing and a three-layer adversarial review, ending in a compiled PDF.
- MARG (Multi-Agent Review Generation) — A research artifact (arXiv:2401.04259) and reusable demo for generating peer reviews of scientific papers using multiple specialized agents.
- mcp-stata (Stata agentic toolkit) — A stdio MCP server, distributed on PyPI and run through uvx, that gives an AI agent control of a local licensed Stata 17+ installation: run commands or .do files (including background jobs), inspect and lint data and code, read r()/e()/s() stored results as structured JSON, export graphs, and diff session state.
- MLE-bench (OpenAI) — Seventy-five Kaggle competitions repackaged as an offline agent benchmark: an agent gets the competition description and data and must produce a submission CSV, graded deterministically against the held-out test set and scored against the human medal thresholds from that competition's original leaderboard.
- MLGym (Meta) — A gym-style framework and benchmark (MLGym-Bench, arXiv:2502.14499) for advancing AI research agents on 13 diverse ML research tasks (CV, NLP, RL, game theory).
- nano-scientist — A budget-first autonomous report generator:
python main.py "topic" --budget 2.00runs four self-terminating loops — literature, experimentation, writing with an internal peer-review pass, and compilation — and lands a LaTeX source, a deduplicated CrossRef-checked BibTeX file and a compiled PDF inoutputs/<uuid>/alongsidecost_log.json,history.jsonandsummary.json. - NatureBench — A cross-discipline benchmark (arXiv:2606.24530) of 90 tasks distilled from peer-reviewed Nature-family papers across six scientific domains, asking whether AI coding agents can match — or surpass — the published state of the art.
- Open CoScientist Agents — An open-source implementation of Google DeepMind's AI co-scientist (arXiv:2502.18864), built on LangGraph and GPT Researcher.
- OpenScholar (AI2) — A retrieval-augmented LM designed to answer scientific queries by searching the literature and generating responses grounded in sources.
- OpenEcon Data — An MCP server plus web app giving AI agents natural-language access to 330K+ economic indicators indexed across 11 providers (FRED, World Bank, IMF, Eurostat, UN Comtrade, BIS, Statistics Canada, OECD, ExchangeRate-API, CoinGecko, ChinaMacro).
- OpenScience — An open-source (Apache-2.0), model-agnostic AI workbench for scientific research, launched 2026-07-03 as an explicit open alternative to Claude Science: a Bun/TypeScript monorepo whose CLI starts a local server hosting a browser workspace, an agent runtime (default
researchagent plusbiology/physics/mlspecialists with critique and literature-review sub-agents), 292 bundled markdown skills, and ~30 scientific-database connectors (UniProt, PDB, ChEMBL, PubChem, arXiv, OpenAlex, Semantic Scholar). - p-hacking-skills (phack) — An instrumented specification-search engine, packaged both as a
phackPyPI CLI and as 11 Claude Code/Codex skills, that measures how quickly and how often realistic agent-driven specification searches manufacture p < .05 on econometric data whose true effect is exactly zero by construction (OLS/RCT, DiD, staggered DiD, RDD, IV). - pAI-Econ-claude — A gated, human-in-the-loop multi-agent Claude Code Skill for AI-assisted economic THEORY development: a 10-stage sequential pipeline (research intake → puzzle refinement → literature → canonical-model matching → model primitives → assumption audit → propositions → proof sketches → counterexample search → economic interpretation → manuscript generation) with an optional numerical- simulation stage, 8 quality gates, and 6+ mandatory human decision points.
- PaperOrchestra — A write-up-only multi-agent pipeline from Google Research: point it at a directory of raw research materials (the idea plus experimental logs) and a LaTeX template directory, and five specialised agents — outline generation, literature review, section writing, content refinement and plotting — return a submission-ready manuscript with a synthesised related-work section and generated plots and conceptual diagrams.
- PaperQA2 (FutureHouse) — A high-accuracy retrieval-augmented generation package focused on scientific PDFs (and Office docs, source code).
- Paper-WorkFlow — A meta-orchestrator for the empirical-economics paper pipeline: Stage 0-9 take a project from topic selection through literature grounding, data, identification/estimation, exhibits, drafting, structural and sentence-level polish, simulated referee review, and journal shortlisting/submission packaging.
- PaperCoder (Paper2Code) — An ICLR 2026 multi-agent system (arXiv:2504.17192) that transforms a machine-learning paper into a working code repository via a three-stage pipeline (planning, analysis, code generation) with specialized agents per stage.
- PaperBench (OpenAI) — The reference replication benchmark for AI research: agents must reproduce 20 ICML 2024 Spotlight and Oral papers from scratch — understanding the contribution, writing the codebase, and executing the experiments — with no starter code.
- PaperDoctor — An agent framework that gives pre-submission papers an evidence-grounded diagnostic rather than a verdict: it checks writing, layout, references, code, theory, prior work, and experiments, and for high-priority claims reruns the paper's own code to verify them.
- PARNESS — A declarative "paper harness": research pipelines are composed in YAML over a thin (~900-line) DAG kernel that does topological sorting and data flow, while control flow is agent-driven — modules emit
_routesignals to branch, iterate or stop. - Prism — OpenAI's free AI-native LaTeX workspace for scientific writing and collaboration, launched late January 2026 on Crixet, a cloud LaTeX platform OpenAI acquired.
- ralph-wiggum-asset-pricing — Geoff Huntley's "Ralph Wiggum" loop applied to an academic asset-pricing paper: the human writes a paper specification (the paper in bullet points), an economic-background file and a test selection; then Claude Code or Codex, running with permissions disabled inside a firewalled dev container, repeats plan → improve → test on the LaTeX paper and R code until every PASS/FAIL test passes.
- RD-Agent (R&D-Agent) — Microsoft's framework for automating data-driven R&D as a two-role loop: a Research agent proposes a hypothesis, a Development agent implements it as executable code, the result is run against real data, and the measured feedback drives the next iteration.
- RECAST (Replication and Extension with Causal AI Statistical Toolkit) — An end-to-end autonomous pipeline for the replication + extension + peer-review arc of the RISE concept diagram.
- RECLAIM — A reproducibility benchmark of 100 NeurIPS 2025 papers that asks whether autonomous agents can recover a paper's pinned central empirical result under a fixed GPU-hour budget.
- Refine (refine.ink) — A commercial AI peer-review service that produces reviewer-grade feedback on academic papers within ~20–40 minutes by running multi-hour parallel compute jobs (~2+ hours per review).
- ReplicatorBench / ReplicatorAgent — A benchmark plus reference agent for evaluating whether LLM agents can replicate empirical findings in the social and behavioral sciences end-to-end: extracting replication-relevant metadata and data from a paper's PDF, designing and generating a replication analysis, executing it in a sandboxed Docker environment, interpreting the statistical results, and validating the verdict against expert-annotated human-coded ground truth.
- REPRO-Bench — A benchmark of 112 task instances, each a published social-science paper paired with its author-supplied reproduction package and a public reproduction report, where the agent must return a verdict on whether the paper reproduces rather than merely re-run its code.
- ReproAgent — A contract-guided paper-to-code reproduction agent: given a research paper, a four-stage Prepare-Plan-Generate-Repair pipeline extracts implementation requirements, binds them to evidence from related repositories, and generates a dependency-ordered, repair-validated codebase.
- ReproRepo — A framework (arXiv:2606.18237) for building issue-grounded reproducibility-audit benchmarks from paper-repository pairs: it collects conference paper metadata and repo links, curates reproducibility-related GitHub issues via LLM review, pins fixed paper/repository snapshots, runs blind static-audit agents, and scores whether agent findings match hidden human-reported reproduction blockers.
- Claude Code Research Harness Workflow — A seven-stage controlled-execution framework (setup -> audit -> clean -> plan -> work -> review -> release) that constrains Claude Code to produce auditable, reproducible empirical research rather than general-purpose "vibe coding." Raw data is read-only, every cleaning and merge decision must leave a script and a log, every analysis task needs a script/log/output triple before it can be marked complete, and causal claims are checked against the declared identification strategy before a replication package is released.
- Research Paper Writing Skills — A single portable skill package (
research-paper-writing/) for ML/CV/NLP paper writing: a SKILL.md core workflow plus eight section-specific reference guides (abstract, introduction, related work, method, experiments, conclusion, paper review, paragraph flow) and an example bank, curated and adapted from Prof. - ResearchTown — An ICML 2025 multi-agent platform for community-level automatic research simulation.
- ResearchAgent (NAACL 2025) — The NAACL 2025 reference implementation (arXiv:2404.07738) of iterative research idea generation over scientific literature.
- ResearchClawBench — A benchmark for end-to-end autonomous research: 40 tasks (4 each across astronomy, chemistry, earth science, energy, information science, life science, materials science, mathematics, neuroscience and physics), each anchored to a real published paper and shipped with that paper's raw data and related literature.
- ResearchGym — A benchmark and execution environment (arXiv:2602.15112, ICLR 2026 Agents in the Wild Workshop) that turns five oral/spotlight papers from ICML, ICLR, and ACL into 39 containerized sub-tasks: each paper's dataset, evaluation harness, and baseline code are kept, but its proposed method is hidden, so an agent must propose a hypothesis, implement it, and beat the paper's own metric.
- Reviewer (Ingar30) — A reproducible multi-agent reviewer for academic economics papers.
- Robin (FutureHouse) — A multi-agent system for automating scientific discovery (arXiv:2505.13400), with explicit support for hypothesis generation, experiment design, and data analysis.
- Sakana AI Scientist v2 — An autonomous "AI scientist" pipeline that ideates, runs experiments (primarily ML), drafts a paper, and self-reviews.
- Sakana AI Scientist (v1) — The original AI Scientist release (arXiv:2408.06292): an end-to-end agentic pipeline that ideates, runs experiments, and writes a paper with self-review on a fixed set of CS templates (NanoGPT, 2D Diffusion, Grokking).
- Sakana AI Agentic Scientist (AI-Scientist-v3) — Third generation of Sakana AI's "AI Scientist" lineage, reframed as an orchestrator that drives off-the-shelf coding-agent CLIs (Claude Code, Codex, Cursor, OpenCode, Gemini, Pi) through a six-phase pipeline — task drafting, investigation, ablations, sealed final evaluation, post-hoc analysis, and paper writing — against a human-authored task.md plus an evaluator, rather than hand-rolling its own agent.
- ScholarEval — A retrieval-augmented system that reviews a research idea before any work is done, along two declared dimensions — Soundness (are the proposed methods empirically valid given what the literature already establishes?) and Contribution (does it advance on prior work?) — with every judgement grounded in papers retrieved via the Semantic Scholar API under a user-declared literature cutoff date rather than in model priors.
- Scholar Loop — A single-maintainer "autonomous AI scientist" that runs the full PhD loop on a single-GPU budget: literature scouting (arXiv + OpenAlex, citation-ranked), grounded hypothesis generation, debate-gated real PyTorch experiments in a smoke -> verify -> full funnel, reflection into a time-decaying skill library, and a number-grounded write-up with self-review — all under a self-stopping budget governor.
- Social Science Replicability Infrastructure — Infrastructure aimed at the replication stage of the RISE pipeline: given a published paper, attempt to reproduce its empirical results in an automated or semi-automated fashion.
- SocSci-Repro-Bench — A benchmark of 221 reproduction tasks drawn from 54 published social-science papers across four disciplines (political science, sociology, economics, psychology) and roughly a dozen substantive domains, where an agent must execute the authors' own analysis code in R, Python or Stata and extract the target quantitative result.
- stata-code — An agent-native Python bridge to Stata offering four parallel interfaces onto one core — an MCP server (21 tools), a Jupyter kernel, a VS Code extension, and a CLI — for running econometric analyses (DiD, IV, RDD) and producing publication-ready tables with token-efficient, structured output.
- Stata MCP — An MCP server plus VS Code / Cursor / Antigravity IDE extension that lets AI coding assistants (GitHub Copilot, Claude Code, Claude Desktop, Cline, Cursor AI) run Stata .do-file selections, view results and graphs, and interact with datasets directly inside the editor.
- StatsPAI — An "agent-native" Python library for causal inference and applied econometrics — a Stata/R-replacement workbench (regress, ivreg, feols, Callaway-Sant'Anna DiD, rdrobust, synthetic control, matching, DML, meta-learners, causal forests, structural estimation; 1,145 registered functions across 87 submodules) whose structured result objects, machine-readable schemas, and MCP server are designed for LLM agents to call.
- STORM / Co-STORM — An LLM-powered knowledge-curation system that writes Wikipedia-style long-form articles from web search.
- SurveyX — An academic survey-automation system (arXiv:2502.14776) that generates domain-specific surveys from a paper title plus retrieval keywords.
- SwarmResearch — An orchestrator-subagent harness for open-ended discovery, from a four-author group at UIUC.
- Terminal-Bench-Science — A benchmark of 70 (growing toward 100+) expert-curated computational research workflows spanning life, physical, earth, mathematical, and engineering sciences, run through the Harbor agent-evaluation framework.
- Tongyi DeepResearch — An agentic large language model purpose-built for long-horizon deep-information-seeking tasks (arXiv:2510.24701), shipped both as open weights (30.5B total / 3.3B active) and as inference code with ReAct and 'Heavy' (IterResearch) modes.
- ToolUniverse — A curated tool registry and MCP server (arXiv:2509.23426) that packages biomedical, chemical, and general scientific APIs into a uniform agent-callable surface.
- Writing-Driven Autoresearch — A multi-agent harness that inverts the usual autoresearch order: instead of experimenting and then writing up, it drafts a complete submittable paper first — claims, tables and all, with the numbers standing in as placeholders — and then drives every experiment from the claims in that draft in a modify → measure → verify → revise loop.
- zeropaper (Auto AI Research Template) — An autonomous research-paper pipeline that uses Claude Code, Codex, or Gemini CLI as the subagent dispatcher.
- Claude Code Skills for Empirical Research (zirui-song) — A personal Claude Code skill pack of 21 skills for empirical economics/finance research, split into six legacy "/command"-style skills (robustness checklist, literature review, coding guidelines, data documentation, project structure, referee response) and fifteen newer agent-skill-format skills organised into four groups: an empirical pipeline group (table regeneration, pipeline refresh, Stata-preflight, robustness-battery, spec-curve-sweep, power-first), a verification group (verify-claims, verify-numbers-pipeline, referee-panel), a LaTeX/Overleaf sync group, and a cross-model-review group built on Claude-Codex plan debate.
- Zochi (Intology) — An end-to-end "artificial scientist" system from Intology, claimed to span hypothesis generation through to peer-reviewed publication.
How to add a project¶
- Copy
projects/landscape/sakana-ai-scientist.ymlas a template. - Fill in fields per
projects/schema.md. - Score it against
projects/EVALUATION.md. - Open a pull request.