We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.MM

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Networking and Internet Architecture

Title: Unbiased Experiments in Congested Networks

Abstract: When developing a new networking algorithm, it is established practice to run a randomized experiment, or A/B test, to evaluate its performance. In an A/B test, traffic is randomly allocated between a treatment group, which uses the new algorithm, and a control group, which uses the existing algorithm. However, because networks are congested, both treatment and control traffic compete against each other for resources in a way that biases the outcome of these tests. This bias can have a surprisingly large effect; for example, in lab A/B tests with two widely used congestion control algorithms, the treatment appeared to deliver 150% higher throughput when used by a few flows, and 75% lower throughput when used by most flows-despite the fact that the two algorithms have identical throughput when used by all traffic.
Beyond the lab, we show that A/B tests can also be biased at scale. In an experiment run in cooperation with Netflix, estimates from A/B tests mistake the direction of change of some metrics, miss changes in other metrics, and overestimate the size of effects. We propose alternative experiment designs, previously used in online platforms, to more accurately evaluate new algorithms and allow experimenters to better understand the impact of congestion on their tests.
Comments: 16 pages, to appear in ACM Internet Measurement Conference (IMC '21)
Subjects: Networking and Internet Architecture (cs.NI); Multimedia (cs.MM)
DOI: 10.1145/3487552.3487851
Cite as: arXiv:2110.00118 [cs.NI]
  (or arXiv:2110.00118v1 [cs.NI] for this version)

Submission history

From: Bruce Spang [view email]
[v1] Thu, 30 Sep 2021 23:12:58 GMT (1128kb,D)

Link back to: arXiv, form interface, contact.