We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Statistics > Methodology

Title: Distributed Simultaneous Inference in Generalized Linear Models via Confidence Distribution

Abstract: We propose a distributed method for simultaneous inference for datasets with sample size much larger than the number of covariates, i.e., N >> p, in the generalized linear models framework. When such datasets are too big to be analyzed entirely by a single centralized computer, or when datasets are already stored in distributed database systems, the strategy of divide-and-combine has been the method of choice for scalability. Due to partition, the sub-dataset sample sizes may be uneven and some possibly close to p, which calls for regularization techniques to improve numerical stability. However, there is a lack of clear theoretical justification and practical guidelines to combine results obtained from separate regularized estimators, especially when the final objective is simultaneous inference for a group of regression parameters. In this paper, we develop a strategy to combine bias-corrected lasso-type estimates by using confidence distributions. We show that the resulting combined estimator achieves the same estimation efficiency as that of the maximum likelihood estimator using the centralized data. As demonstrated by simulated and real data examples, our divide-and-combine method yields nearly identical inference as the centralized benchmark.
Comments: This paper is published in Journal of Multivariate Analysis
Subjects: Methodology (stat.ME)
Journal reference: Journal of Multivariate Analysis, 176, 104567 (2020)
DOI: 10.1016/j.jmva.2019.104567
Cite as: arXiv:1611.06208 [stat.ME]
  (or arXiv:1611.06208v3 [stat.ME] for this version)

Submission history

From: Lu Tang [view email]
[v1] Fri, 18 Nov 2016 19:51:07 GMT (366kb,D)
[v2] Mon, 11 Nov 2019 03:17:48 GMT (372kb,D)
[v3] Tue, 21 Jul 2020 21:05:02 GMT (373kb,D)

Link back to: arXiv, form interface, contact.