References & Citations
Computer Science > Machine Learning
Title: Adaptive Weight Decay: On The Fly Weight Decay Tuning for Improving Robustness
(Submitted on 30 Sep 2022 (this version), latest version 2 Dec 2023 (v2))
Abstract: We introduce adaptive weight decay, which automatically tunes the hyper-parameter for weight decay during each training iteration. For classification problems, we propose changing the value of the weight decay hyper-parameter on the fly based on the strength of updates from the classification loss (i.e., gradient of cross-entropy), and the regularization loss (i.e., $\ell_2$-norm of the weights). We show that this simple modification can result in large improvements in adversarial robustness -- an area which suffers from robust overfitting -- without requiring extra data. Specifically, our reformulation results in 20% relative robustness improvement for CIFAR-100, and 10% relative robustness improvement on CIFAR-10 comparing to traditional weight decay. In addition, this method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms, which the latter contributes to robustness to overfitting to label noise, and pruning.
Submission history
From: Amin Ghiasi [view email][v1] Fri, 30 Sep 2022 21:13:00 GMT (1852kb,D)
[v2] Sat, 2 Dec 2023 01:27:27 GMT (3375kb,D)
Link back to: arXiv, form interface, contact.