Title: A Smoothing Riemannian Gradient Algorithm Framework for Nonsmooth Optimization Problem on Manifold

Abstract: In this paper, we introduce the notion of generalized $\epsilon$-stationarity for a class of nonconvex and nonsmooth composite minimization problems on Riemannian manifolds embedding in Euclidean space. To find a generalized $\epsilon$-stationarity point, we develop a family of Riemannian gradient type methods based on the Moreau envelope technique with a decreasing sequence of smoothing parameters, namely Riemannian smoothing gradient and Riemannian smoothing stochastic gradient methods. We prove that the Riemannian smoothing gradient method has the iteration complexity of $\mathcal{O}(\epsilon^{-3})$ for driving a generalized $\epsilon$-stationary point. To our knowledge, this is the best-known computational complexity result for the nonconvex and nonsmooth composite problem on manifolds. For the Riemannian smoothing stochastic gradient method, one can achieve the iteration complexity of $\mathcal{O}(\epsilon^{-5})$ for driving a generalized $\epsilon$-stationary point.