Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent
(Submitted on 15 Jan 2020 (v1), last revised 30 Jan 2020 (this version, v2))
Abstract: Deep unfolding is a promising deep-learning technique in which an iterative algorithm is unrolled to a deep network architecture with trainable parameters. In the case of gradient descent algorithms, as a result of the training process, one often observes the acceleration of the convergence speed with learned non-constant step size parameters whose behavior is not intuitive nor interpretable from conventional theory. In this paper, we provide a theoretical interpretation of the learned step size of deep-unfolded gradient descent (DUGD). We first prove that the training process of DUGD reduces not only the mean squared error loss but also the spectral radius related to the convergence rate. Next, we show that minimizing the upper bound of the spectral radius naturally leads to the Chebyshev step which is a sequence of the step size based on Chebyshev polynomials. The numerical experiments confirm that the Chebyshev steps qualitatively reproduce the learned step size parameters in DUGD, which provides a plausible interpretation of the learned parameters. Additionally, we show that the Chebyshev steps achieve the lower bound of the convergence rate for the first-order method in a specific limit without learning parameters or momentum terms.
Submission history
From: Satoshi Takabe [view email][v1] Wed, 15 Jan 2020 05:58:07 GMT (333kb,D)
[v2] Thu, 30 Jan 2020 11:27:52 GMT (331kb,D)
Link back to: arXiv, form interface, contact.