We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Abstract: We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.
Comments: 29 pages, 8 figures, 8 tables
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Econometrics (econ.EM); Statistics Theory (math.ST)
MSC classes: Primary 62J10, 62M02, secondary 60K35
ACM classes: G.3
Cite as: arXiv:2106.03156 [stat.ML]
  (or arXiv:2106.03156v3 [stat.ML] for this version)

Submission history

From: Youngki Shin [view email]
[v1] Sun, 6 Jun 2021 15:38:37 GMT (4376kb,D)
[v2] Mon, 21 Jun 2021 13:52:59 GMT (4068kb,D)
[v3] Thu, 7 Oct 2021 02:05:38 GMT (7157kb,D)

Link back to: arXiv, form interface, contact.