Current browse context:
stat
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
(Submitted on 11 Jul 2020 (v1), last revised 16 Dec 2020 (this version, v2))
Abstract: The evaluation of hyperparameters, neural architectures, or data augmentation policies becomes a critical model selection problem in advanced deep learning with a large hyperparameter search space. In this paper, we propose an efficient and robust bandit-based algorithm called Sub-Sampling (SS) in the scenario of hyperparameter search evaluation. It evaluates the potential of hyperparameters by the sub-samples of observations and is theoretically proved to be optimal under the criterion of cumulative regret. We further combine SS with Bayesian Optimization and develop a novel hyperparameter optimization algorithm called BOSS. Empirical studies validate our theoretical arguments of SS and demonstrate the superior performance of BOSS on a number of applications, including Neural Architecture Search (NAS), Data Augmentation (DA), Object Detection (OD), and Reinforcement Learning (RL).
Submission history
From: Yimin Huang [view email][v1] Sat, 11 Jul 2020 03:15:21 GMT (1072kb,D)
[v2] Wed, 16 Dec 2020 10:28:43 GMT (21135kb,D)
Link back to: arXiv, form interface, contact.