We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Abstract: Online controlled experiments (also known as A/B Testing) have been viewed as a golden standard for large data-driven companies since the last few decades. The most common A/B testing framework adopted by many companies use "average treatment effect" (ATE) as statistics. However, it remains a difficult problem for companies to improve the power of detecting ATE while controlling "false discovery rate" (FDR) at a predetermined level. One of the most popular FDR-control algorithms is BH method, but BH method is only known to control FDR under restrictive positive dependence assumptions with a conservative bound. In this paper, we propose statistical methods that can systematically and accurately identify ATE, and demonstrate how they can work robustly with controlled low FDR but a higher power using both simulation and real-world experimentation data. Moreover, we discuss the scalability problem in detail and offer comparison of our paradigm to other more recent FDR control methods, e.g., knockoff, AdaPT procedure, etc.
Comments: 9 pages, 5 figures
Subjects: Methodology (stat.ME); Applications (stat.AP)
Cite as: arXiv:2110.07279 [stat.ME]
  (or arXiv:2110.07279v1 [stat.ME] for this version)

Submission history

From: Yihan Bao [view email]
[v1] Thu, 14 Oct 2021 11:31:45 GMT (2567kb,D)

Link back to: arXiv, form interface, contact.