Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Why are Adaptive Methods Good for Attention Models?
(Submitted on 6 Dec 2019 (v1), last revised 23 Oct 2020 (this version, v2))
Abstract: While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models. The settings under which SGD performs poorly in comparison to adaptive methods are not well understood yet. In this paper, we provide empirical and theoretical evidence that a heavy-tailed distribution of the noise in stochastic gradients is one cause of SGD's poor performance. We provide the first tight upper and lower convergence bounds for adaptive gradient methods under heavy-tailed noise. Further, we demonstrate how gradient clipping plays a key role in addressing heavy-tailed gradient noise. Subsequently, we show how clipping can be applied in practice by developing an \emph{adaptive} coordinate-wise clipping algorithm (ACClip) and demonstrate its superior performance on BERT pretraining and finetuning tasks.
Submission history
From: Jingzhao Zhang [view email][v1] Fri, 6 Dec 2019 15:58:29 GMT (287kb)
[v2] Fri, 23 Oct 2020 08:07:04 GMT (800kb,D)
Link back to: arXiv, form interface, contact.