We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: Statistical Methods for Replicability Assessment

Abstract: Large-scale replication studies like the Reproducibility Project: Psychology (RP:P) provide invaluable systematic data on scientific replicability, but most analyses and interpretations of the data fail to agree on the definition of "replicability" and disentangle the inexorable consequences of known selection bias from competing explanations. We discuss three concrete definitions of replicability based on (1) whether published findings about the signs of effects are mostly correct, (2) how effective replication studies are in reproducing whatever true effect size was present in the original experiment, and (3) whether true effect sizes tend to diminish in replication. We apply techniques from multiple testing and post-selection inference to develop new methods that answer these questions while explicitly accounting for selection bias. Our analyses suggest that the RP:P dataset is largely consistent with publication bias due to selection of significant effects. The methods in this paper make no distributional assumptions about the true effect sizes.
Subjects: Methodology (stat.ME); Applications (stat.AP)
MSC classes: 62F03, 62P25
Journal reference: Ann. Appl. Stat. 14(3): 1063-1087 (September 2020)
DOI: 10.1214/20-AOAS1336
Cite as: arXiv:1903.08747 [stat.ME]
  (or arXiv:1903.08747v2 [stat.ME] for this version)

Submission history

From: Kenneth Hung [view email]
[v1] Wed, 20 Mar 2019 21:15:12 GMT (223kb,D)
[v2] Thu, 27 Feb 2020 00:04:45 GMT (221kb,D)

Link back to: arXiv, form interface, contact.