Current browse context:
math.ST
Change to browse by:
References & Citations
Mathematics > Statistics Theory
Title: Always Valid Inference: Bringing Sequential Analysis to A/B Testing
(Submitted on 15 Dec 2015 (this version), latest version 16 Jul 2019 (v3))
Abstract: Nearly all technology platforms (e.g., web applications, mobile applications, etc.) use randomized controlled trials, or A/B tests, as a means to optimize their product offering. Such tests are generally analyzed using classical frequentist statistical measures: p-values and confidence intervals. These measures serve as a transparent, interpretable interface between the data and the user, allowing valid inference. However, these reported values cease to be valid if users make decisions based on continuous monitoring of their tests. Users try to take advantage of data as fast as it becomes available, but current testing practice prevents them from doing so while maintaining valid statistical inference.
Through connections to sequential hypothesis testing, we present analogues of classical frequentist statistical measures that are always valid, regardless of when users choose to look at the test. We discuss how to optimally choose such a sequential test. We also discuss applications to bandits, and extensions to multiple hypothesis testing in the sequential setting. Finally, we discuss implementation and deployment of our approach in a large scale commercial A/B testing platform.
Submission history
From: Ramesh Johari [view email][v1] Tue, 15 Dec 2015 20:33:31 GMT (326kb,D)
[v2] Wed, 17 Feb 2016 07:12:05 GMT (1077kb,D)
[v3] Tue, 16 Jul 2019 19:42:42 GMT (2192kb,D)
Link back to: arXiv, form interface, contact.