We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.AP

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Applications

Title: Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

Abstract: A/B tests have been widely adopted across industries as the golden rule that guides decision making. However, the long-term true north metrics we ultimately want to drive through A/B test may take a long time to mature. In these situations, a surrogate metric which predicts the long-term metric is often used instead to conclude whether the treatment is effective. However, because the surrogate rarely predicts the true north perfectly, a regular A/B test based on surrogate metrics tends to have high false positive rate and the treatment variant deemed favorable from the test may not be the winning one. In this paper, we discuss how to adjust the A/B testing comparison to ensure experiment results are trustworthy. We also provide practical guidelines on the choice of good surrogate metrics. To provide a concrete example of how to leverage surrogate metrics for fast decision making, we present a case study on developing and evaluating the predicted confirmed hire surrogate metric in LinkedIn job marketplace.
Subjects: Applications (stat.AP)
Journal reference: Proceedings of the 14th ACM International Conference on Web Search and Data Mining (2021)
DOI: 10.1145/3437963.3441737
Cite as: arXiv:2106.01421 [stat.AP]
  (or arXiv:2106.01421v1 [stat.AP] for this version)

Submission history

From: Weitao Duan [view email]
[v1] Wed, 2 Jun 2021 19:03:43 GMT (1716kb)

Link back to: arXiv, form interface, contact.