SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

Summary

Names and measures 'semantic drift': paper-reproduction code that runs but silently diverges from the paper's specification. Decomposes 30 ICLR/ICML/NeurIPS 2025 papers into 1,491 atomic Semantic Alignment Units across numerical, methodological, protocol and ordering drift, then scores 12 generator configurations; the best (Claude + PaperCoder) reaches only 0.301 of 1.0, and the failure taxonomy shows agents attempt most requirements but implement them wrongly, with mismatches and stubs dominating zero scores. The key finding for replication agents is that scaffolds optimised for executability buy almost nothing for scientific faithfulness.

Contribution

Queued — not yet read.

Method

Queued — not yet read.

Relevance to RISE

Surfaced by the 2026-09-07 literature sweep as a high-priority candidate for the RISE knowledge base (angle: replication and reproducibility). Verified against: Opened https://arxiv.org/abs/2608.24252, read full abstract, 5 authors, 2026-08-25, cs.AI/cs.SE. Dedupe: no 'sa-bench' or 'semantic alignment' match.

Critique / open questions

Queued — not yet assessed.

Key quotes

LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. (abstract)

We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. (abstract)