Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Oracle Complexity of Second-Order Methods for Finite-Sum Problems
(Submitted on 15 Nov 2016 (v1), last revised 8 Mar 2017 (this version, v2))
Abstract: Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require much fewer iterations than first-order methods, and hold the promise for more efficient algorithms. Although computing and manipulating Hessians is prohibitive for high-dimensional problems in general, the Hessians of individual functions in finite-sum problems can often be efficiently computed, e.g. because they possess a low-rank structure. Can second-order information indeed be used to solve such problems more efficiently? In this paper, we provide evidence that the answer -- perhaps surprisingly -- is negative, at least in terms of worst-case guarantees. However, we also discuss what additional assumptions and algorithmic approaches might potentially circumvent this negative result.
Submission history
From: Yossi Arjevani [view email][v1] Tue, 15 Nov 2016 18:41:55 GMT (24kb)
[v2] Wed, 8 Mar 2017 11:05:59 GMT (29kb)
Link back to: arXiv, form interface, contact.