Skip to content

Social Science Replicability Infrastructure

external · status: active · focus: replication · discipline: social-sciences · started: 2025

Project page: https://github.com/benjamin-kohler/social_science_replicability

Licence: none

No licence declared

The repository declares no licence, so its code cannot be reused, modified or redistributed without the maintainer's permission. RISE describes and links to the project; nothing from it is reproduced here.

Source: projects/landscape/social-science-replicability.yml

Positioning

Infrastructure aimed at the replication stage of the RISE pipeline: given a published paper, attempt to reproduce its empirical results in an automated or semi-automated fashion. Sits squarely in the replication block of the RISE diagram.

Distinctive contribution

Focuses the agentic-research conversation on replication of existing papers rather than generation of new ones — a complementary axis to the AI-scientist line.

Evaluation scores

Dimension Score (0–3) Note
Lifecycle coverage 1 Four stages, all in the replication arc; does not cover ideation/drafting.
Autonomy level 2 Supervised: user supplies target, system attempts replication.
Architectural transparency 2 Open source; architecture documented in README; prompts visible.
Inputs supported 2 Accepts target paper + target dataset; integrates external sources.
Outputs / reproducibility 2 Reports + code persisted; reproducibility-by-design as a stated goal.
Internal evaluation 2 Systematic benchmark across 48 papers with human-verified reproducibility, run against 4 agent scaffolds and 4 LLMs, with root-cause analysis of failures; paper now forthcoming at EMNLP 2026 but not yet third-party replicated.
Openness 3 Open source under permissive license.
Maturity / traction 1 Active prototype; single-developer-led; 29 stars, 7 forks, 42 commits, most recent 2026-08-31; no further commits since.
Cross-family policy 0 Single-LLM-family pipeline; methodology extractor + replicator within one family.
Runtime assurance 2 Code-execution + output-match comparison against target paper is the runtime assurance.
Cross-platform portability 1 Python-CLI tool; back-end LLM swappable but not multi-IDE.

Scored on 2026-10-01. See the evaluation rubric.

Tags

Pipeline stages: replication data-acquisition data-analysis code-generation

Architectural features: tool-use dag-orchestration artifact-versioning

Inputs: target-paper target-dataset

Outputs: replication-report code comparison-tables

Data sources: target-paper-data

Knowledge sources: target-paper

Limitations

  • Success rate depends heavily on availability of target paper's data + code.
  • The 48-paper benchmark (4 scaffolds x 4 LLMs) lives in the companion paper, not as a runnable harness in this repo; success varies substantially by model, scaffold and paper.

Papers describing this project

  • Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results — Köhler, B., Zollikofer, D., Einsiedler, A., Hoyle, A., Ash, E. (2026). arXiv (forthcoming at EMNLP 2026). arXiv:2604.21965