Current browse context:
math.ST
Change to browse by:
References & Citations
Mathematics > Statistics Theory
Title: Always Valid Inference: Bringing Sequential Analysis to A/B Testing
(Submitted on 15 Dec 2015 (v1), revised 17 Feb 2016 (this version, v2), latest version 16 Jul 2019 (v3))
Abstract: A/B tests are typically analyzed via p-values and confidence intervals; but these inferences are wholly unreliable if users make decisions while continuously monitoring their tests. We define always valid p-values that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid p-values can be interpreted as the natural p-values corresponding to a sequential hypothesis test. Through this connection we derive always valid p-values with good detection properties. Notably, we also extend our approach to address multiple hypothesis testing in the sequential setting. Our methodology has been implemented in a large scale commercial A/B testing platform, from which we present empirical results.
Submission history
From: Ramesh Johari [view email][v1] Tue, 15 Dec 2015 20:33:31 GMT (326kb,D)
[v2] Wed, 17 Feb 2016 07:12:05 GMT (1077kb,D)
[v3] Tue, 16 Jul 2019 19:42:42 GMT (2192kb,D)
Link back to: arXiv, form interface, contact.