We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

econ

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Economics > Econometrics

Title: Critical Values Robust to P-hacking

Abstract: P-hacking occurs when researchers engage in various behaviors that increase their chances of reporting statistically significant results. P-hacking is problematic because it reduces the informativeness of hypothesis tests -- by making significant results much more common than they are supposed to be in the absence of true significance. Despite its prevalence, p-hacking is not taken into account in hypothesis testing theory: the critical values used to determine significance assume no p-hacking. To address this problem, we build a model of p-hacking and use it to construct critical values such that, if these values are used to determine significance, and if researchers adjust their behavior to these new significance standards, then significant results occur with the desired frequency. Because such robust critical values allow for p-hacking, they are larger than classical critical values. As an illustration, we calibrate the model with evidence from the social and medical sciences. We find that the robust critical value for any test is the classical critical value for the same test with one fifth of the significance level -- a form of Bonferroni correction. For instance, for a $z$-test with a significance level of $5\%$, the robust critical value is $2.31$ instead of $1.65$ if the test is one-sided and $2.57$ instead of $1.96$ if the test is two-sided.
Subjects: Econometrics (econ.EM); Methodology (stat.ME)
Cite as: arXiv:2005.04141 [econ.EM]
  (or arXiv:2005.04141v4 [econ.EM] for this version)

Submission history

From: Pascal Michaillat [view email]
[v1] Fri, 8 May 2020 16:37:11 GMT (1634kb,D)
[v2] Thu, 23 Dec 2021 18:26:56 GMT (142kb,D)
[v3] Tue, 11 Jan 2022 03:15:17 GMT (144kb,D)
[v4] Thu, 16 Jun 2022 02:58:47 GMT (144kb,D)

Link back to: arXiv, form interface, contact.