Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Tune smarter not harder: A principled approach to tuning learning rates for shallow nets
(Submitted on 22 Mar 2020 (v1), last revised 30 Sep 2020 (this version, v3))
Abstract: Effective hyper-parameter tuning is essential to guarantee the performance that neural networks have come to be known for. In this work, a principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. We associate the learning rate with the gradient Lipschitz constant of the objective to be minimized while training. An upper bound on the mentioned constant is derived and a search algorithm, which always results in non-divergent traces, is proposed to exploit the derived bound. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods such as Tree Parzen Estimators (TPE). The proposed method is applied to three different existing applications: a) channel estimation in OFDM systems, b) prediction of the exchange currency rates and c) offset estimation in OFDM receivers, and it is shown to pick better learning rates than the existing methods using the same or lesser compute power.
Submission history
From: Thulasi Tholeti [view email][v1] Sun, 22 Mar 2020 09:38:35 GMT (108kb)
[v2] Wed, 12 Aug 2020 21:06:50 GMT (141kb)
[v3] Wed, 30 Sep 2020 02:18:30 GMT (142kb)
Link back to: arXiv, form interface, contact.