References & Citations
Statistics > Methodology
Title: Percent Change Estimation in Large Scale Online Experiments
(Submitted on 1 Nov 2017 (v1), last revised 22 Aug 2019 (this version, v2))
Abstract: Online experiments are a fundamental component of the development of web-facing products. Given their large user-bases, even small product improvements can have a large impact on user engagement or profits on an absolute scale. As a result, accurately estimating the relative impact of these changes is extremely important. I propose an approach based on an objective Bayesian model to improve the sensitivity of percent change estimation in A/B experiments. Leveraging pre-period information, this approach produces more robust and accurate point estimates and up to 50% tighter credible intervals than traditional methods. The R package abpackage provides an implementation of the approach.
Submission history
From: Jacopo Soriano [view email][v1] Wed, 1 Nov 2017 23:28:46 GMT (349kb,D)
[v2] Thu, 22 Aug 2019 04:35:38 GMT (404kb,D)
Link back to: arXiv, form interface, contact.