References & Citations
Computer Science > Machine Learning
Title: Towards Stability of Parameter-free Optimization
(Submitted on 7 May 2024 (v1), last revised 27 May 2024 (this version, v3))
Abstract: Hyperparameter tuning, particularly the selection of an appropriate learning rate in adaptive gradient training methods, remains a challenge. To tackle this challenge, in this paper, we propose a novel parameter-free optimizer, \textsc{AdamG} (Adam with the golden step size), designed to automatically adapt to diverse optimization problems without manual tuning. The core technique underlying \textsc{AdamG} is our golden step size derived for the AdaGrad-Norm algorithm, which is expected to help AdaGrad-Norm preserve the tuning-free convergence and approximate the optimal step size in expectation w.r.t. various optimization scenarios. To better evaluate tuning-free performance, we propose a novel evaluation criterion, \textit{reliability}, to comprehensively assess the efficacy of parameter-free optimizers in addition to classical performance criteria. Empirical results demonstrate that compared with other parameter-free baselines, \textsc{AdamG} achieves superior performance, which is consistently on par with Adam using a manually tuned learning rate across various optimization tasks.
Submission history
From: Yijiang Pang [view email][v1] Tue, 7 May 2024 14:58:12 GMT (909kb,D)
[v2] Thu, 23 May 2024 00:31:34 GMT (893kb,D)
[v3] Mon, 27 May 2024 14:46:21 GMT (893kb,D)
Link back to: arXiv, form interface, contact.