Assessing Reproducibility in Economics Using Standardized Crowd-Sourced Analysis

Summary

Brodeur, Sung, Miguel, Vilhuber and Hoces de la Guardia present a framework to standardise crowd-sourced computational reproductions in economics through the Social Science Reproduction Platform (SSRP), and summarise the first 487 reproductions uploaded to the platform.

Contribution

The framework addresses four challenges for computational reproductions: lack of standardisation, aggregation issues, existing incentives for "adversarial" interactions, and the loss of knowledge from analyses that are never published. The empirical headline: "approximately 30% of recent studies meeting at least a basic definition of being computationally reproducible," consistent with a prior literature reporting between 14% and 43%.

Method

Empirical: descriptive analysis of the first 487 reproductions on the SSRP platform, framed by a methodological proposal for standardised crowd-sourced reproduction protocols.

Relevance to RISE

A direct anchor for RISE work on replication and reproducibility: provides a platform-level baseline (~30% reproducibility) and a standardisation framework that any RISE replication-oriented project — most notably catalog entry social-science-replicability — should benchmark itself against. Also relevant for evaluating whether agentic systems can raise the reproducibility rate.

Critique / open questions

The 487-reproduction sample is non-representative (self-selected SSRP contributors); the "basic definition" of reproducibility used for the ~30% headline figure is the lowest of several bars in the broader literature.

Key quotes

"This paper presents a framework to standardize crowd-sourced computational reproductions in economics through the Social Science Reproduction Platform (SSRP)."

"The results show substantial heterogeneity in the ability to successfully reproduce empirical results in economics research, with approximately 30% of recent studies meeting at least a basic definition of being computationally reproducible."