PaperGym: Rubric-Centered Evolution for Research-Plan Generation

Summary

Turns each scientific paper into a training-and-evaluation environment for research planning by synthesizing the question from goal and background while deriving grading criteria from method and experiments, cutting criterion leakage to 3.7% against 11.9-34.1% in existing rubric datasets. Releases PaperGym-20k plus the PaperGym-Innov and PaperGym-Design benchmarks covering methodological innovation and experimental design. Important for the leakage problem that quietly inflates most paper-derived research-agent benchmarks.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: research-agent benchmarks). Verified against: Opened https://arxiv.org/abs/2608.31119, read full abstract, 9 authors, 2026-08-31, cs.CL. Dedupe: no 'papergym' or 'rubric' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

Research planning is the decisive capability of AI scientists. (abstract)

Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. (abstract)