We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models

Abstract: Momentum methods have been shown to accelerate the convergence of the standard gradient descent algorithm in practice and theory. In particular, the minibatch-based gradient descent methods with momentum (MGDM) are widely used to solve large-scale optimization problems with massive datasets. Despite the success of the MGDM methods in practice, their theoretical properties are still underexplored. To this end, we investigate the theoretical properties of MGDM methods based on the linear regression models. We first study the numerical convergence properties of the MGDM algorithm and further provide the theoretically optimal tuning parameters specification to achieve faster convergence rate. In addition, we explore the relationship between the statistical properties of the resulting MGDM estimator and the tuning parameters. Based on these theoretical findings, we give the conditions for the resulting estimator to achieve the optimal statistical efficiency. Finally, extensive numerical experiments are conducted to verify our theoretical results.
Comments: 45 pages, 5 figures
Subjects: Methodology (stat.ME); Statistics Theory (math.ST); Computation (stat.CO)
Cite as: arXiv:2111.01507 [stat.ME]
  (or arXiv:2111.01507v1 [stat.ME] for this version)

Submission history

From: Yuan Gao [view email]
[v1] Tue, 2 Nov 2021 11:25:18 GMT (68kb,D)

Link back to: arXiv, form interface, contact.