References & Citations
Computer Science > Machine Learning
Title: Scrap Your Schedules with PopDescent
(Submitted on 23 Oct 2023 (v1), last revised 24 Apr 2024 (this version, v2))
Abstract: In contemporary machine learning workloads, numerous hyper-parameter search algorithms are frequently utilized to efficiently discover high-performing hyper-parameter values, such as learning and regularization rates. As a result, a range of parameter schedules have been designed to leverage the capability of adjusting hyper-parameters during training to enhance loss performance. These schedules, however, introduce new hyper-parameters to be searched and do not account for the current loss values of the models being trained.
To address these issues, we propose Population Descent (PopDescent), a progress-aware hyper-parameter tuning technique that employs a memetic, population-based search. By merging evolutionary and local search processes, PopDescent proactively explores hyper-parameter options during training based on their performance. Our trials on standard machine learning vision tasks show that PopDescent converges faster than existing search methods, finding model parameters with test-loss values up to 18% lower, even when considering the use of schedules. Moreover, we highlight the robustness of PopDescent to its initial training parameters, a crucial characteristic for hyper-parameter search techniques.
Submission history
From: Abhinav Pomalapally [view email][v1] Mon, 23 Oct 2023 08:11:17 GMT (172kb,D)
[v2] Wed, 24 Apr 2024 18:18:30 GMT (443kb,D)
Link back to: arXiv, form interface, contact.