Title: On the Influence of Momentum Acceleration on Online Learning

Abstract: The article examines in some detail the convergence rate and mean-square-error performance of momentum stochastic gradient methods in the constant step-size case and slow adaptation regime. The results establish that momentum methods are equivalent to the standard stochastic gradient method with a re-scaled (larger) step-size value. The size of the re-scaling is determined by the value of the momentum parameter. The equivalence result is established for all time instants and not only in steady-state. The analysis is carried out for general risk functions, and is not limited to quadratic risks. One notable conclusion is that the well-known benefits of momentum constructions for deterministic optimization problems do not necessarily carry over to the stochastic (online) setting when adaptation becomes necessary and when the true gradient vectors are not known beforehand. The analysis also suggests a method to retain some of the advantages of the momentum construction by employing a decaying momentum parameter, as opposed to a decaying step-size. In this way, the enhanced convergence rate during the initial stages of adaptation is preserved without the often-observed degradation in MSD performance.