External pipelines · RISE project catalogue
ReproRepo
A framework (arXiv:2606.18237) for building issue-grounded reproducibility-audit benchmarks from paper-repository pairs: it collects conference paper metadata and repo links, curates reproducibility-related GitHub issues via LLM review, pins fixed paper/repository snapshots, runs blind static-audit agents, and scores whether agent findings match hidden human-reported reproduction blockers. Evaluated at scale on 1,149 ML papers. Sits in the RISE evaluation-infrastructure layer and speaks directly to the replication-infrastructure interest.
Where it sits
left: what it builds on · right: what builds on it · pale: exampleContributed by Shanda Li and collaborators
How studies reach it
No published study reaches it yet.
Disciplines it reaches
No study reaches it yet.
Solid: published studies. Light: examples.
Computed from the records on this site: what each study, template and specialist names as used, which study extends which, and who contributed what. 0 studies in total.
What it does
Uses human-raised GitHub issues as naturally occurring ground truth for reproducibility auditing — sidestepping both expensive expert annotation and full re-execution — under a blind protocol in which agents audit paper + repo snapshots without ever seeing the hidden issues. The best baseline (Codex with GPT-5.5) surfaces at least one semantically related human-reported blocker for roughly 90% of papers; the CMU team includes Shah, Talwalkar, Dettmers, and Yang.
- Focus
- replication
- Inputs
- paper-repository-pairs, github-issues
- Outputs
- benchmark-case-sets, audit-findings, alignment-scores
- Architecture
- tool-use
- Maintained by
- Shanda Li and collaborators (Carnegie Mellon University)
- Started
- 2026
Description
Data model- Discipline
- Computer science
- Method family
- Replication
- Design
- not specified
- Research stage
- Replication
- Contributors
- Shanda Li and collaborators
- Usage
- not used in published research yet
- Source
- RISE project catalogue · projects/landscape · @4c17bae
- Record
- pipeline:reprorepo · JSON
Solid tags are declared by the source or mapped from its terms; dashed tags are inferred by a published rule. Hover a tag for its provenance.
Bring it into the standard
A pipeline built outside E2ER can meet the standard by describing its steps as a template, attaching the floor of checks and publishing evaluation records. Its authors keep ownership and credit.