We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.ST

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Mathematics > Statistics Theory

Title: Comparing distributions by multiple testing across quantiles or CDF values

Abstract: When comparing two distributions, it is often helpful to learn at which quantiles or values there is a statistically significant difference. This provides more information than the binary "reject" or "do not reject" decision of a global goodness-of-fit test. Framing our question as multiple testing across the continuum of quantiles $\tau\in(0,1)$ or values $r\in\mathbb{R}$, we show that the Kolmogorov--Smirnov test (interpreted as a multiple testing procedure) achieves strong control of the familywise error rate. However, its well-known flaw of low sensitivity in the tails remains. We provide an alternative method that retains such strong control of familywise error rate while also having even sensitivity, i.e., equal pointwise type I error rates at each of $n\to\infty$ order statistics across the distribution. Our one-sample method computes instantly, using our new formula that also instantly computes goodness-of-fit $p$-values and uniform confidence bands. To improve power, we also propose stepdown and pre-test procedures that maintain control of the asymptotic familywise error rate. One-sample and two-sample cases are considered, as well as extensions to regression discontinuity designs and conditional distributions. Simulations, empirical examples, and code are provided.
Comments: under review
Subjects: Statistics Theory (math.ST); Econometrics (econ.EM); Methodology (stat.ME)
Journal reference: Journal of Econometrics 206 (2018) 143-166
DOI: 10.1016/j.jeconom.2018.04.003
Cite as: arXiv:1708.04658 [math.ST]
  (or arXiv:1708.04658v1 [math.ST] for this version)

Submission history

From: David Kaplan [view email]
[v1] Tue, 15 Aug 2017 19:30:50 GMT (193kb,D)

Link back to: arXiv, form interface, contact.