References & Citations
Statistics > Methodology
Title: Implementing Monte Carlo Tests with P-value Buckets
(Submitted on 27 Mar 2017 (v1), last revised 4 Nov 2019 (this version, v5))
Abstract: Software packages usually report the results of statistical tests using p-values. Users often interpret these by comparing them to standard thresholds, e.g. 0.1%, 1% and 5%, which is sometimes reinforced by a star rating (***, **, *). We consider an arbitrary statistical test whose p-value p is not available explicitly, but can be approximated by Monte Carlo samples, e.g. by bootstrap or permutation tests. The standard implementation of such tests usually draws a fixed number of samples to approximate p. However, the probability that the exact and the approximated p-value lie on different sides of a threshold (the resampling risk) can be high, particularly for p-values close to a threshold. We present a method to overcome this. We consider a finite set of user-specified intervals which cover [0,1] and which can be overlapping. We call these p-value buckets. We present algorithms that, with arbitrarily high probability, return a p-value bucket containing p. We prove that for both a bounded resampling risk and a finite runtime, overlapping buckets need to be employed, and that our methods both bound the resampling risk and guarantee a finite runtime for such overlapping buckets. To interpret decisions with overlapping buckets, we propose an extension of the star rating system. We demonstrate that our methods are suitable for use in standard software, including for low p-value thresholds occurring in multiple testing settings, and that they can be computationally more efficient than standard implementations.
Submission history
From: Georg Hahn [view email][v1] Mon, 27 Mar 2017 20:47:25 GMT (70kb,D)
[v2] Fri, 20 Oct 2017 00:23:16 GMT (212kb,D)
[v3] Fri, 11 May 2018 16:24:40 GMT (200kb,D)
[v4] Tue, 7 May 2019 04:43:44 GMT (210kb,D)
[v5] Mon, 4 Nov 2019 15:32:36 GMT (210kb,D)
Link back to: arXiv, form interface, contact.