We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learning

Abstract: In this paper, we consider both first- and second-order techniques to address continuous optimization problems arising in machine learning. In the first-order case, we propose a framework of transition from deterministic or semi-deterministic to stochastic quadratic regularization methods. We leverage the two-phase nature of stochastic optimization to propose a novel first-order algorithm with adaptive sampling and adaptive step size. In the second-order case, we propose a novel stochastic damped L-BFGS method that improves on previous algorithms in the highly nonconvex context of deep learning. Both algorithms are evaluated on well-known deep learning datasets and exhibit promising performance.
Comments: 29 pages, 8 figures. arXiv admin note: text overlap with arXiv:2012.05783
Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC)
MSC classes: 68T07, 90C15, 90C30, 90C53
ACM classes: G.1.6; G.3; G.4; I.2.6
Cite as: arXiv:2111.14761 [cs.LG]
  (or arXiv:2111.14761v1 [cs.LG] for this version)

Submission history

From: Sanae Lotfi [view email]
[v1] Mon, 29 Nov 2021 18:10:00 GMT (250kb,D)

Link back to: arXiv, form interface, contact.