Title: Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis

Abstract: We analyze linear regression problem with a nonconvex regularization called smoothly clipped absolute deviation (SCAD) under overcomplete Gaussian basis for Gaussian random data. We develop a message passing algorithm SCAD-AMP and analytically show that the stability condition is corresponding to the AT condition in spin glass literature. As asymptotic analysis, we show the correspondence between density evolution of SCAD-AMP and replica symmetric solution. Numerical experiments confirm that for sufficiently large system size, SCAD-AMP achieves the optimal performance predicted by replica method. From replica analysis, phase transition between replica symmetric (RS) and replica symmetry breaking (RSB) region is found in the parameter space of SCAD. The appearance of RS region for nonconvex penalty is a great advantage which indicate the region of smooth landscape of the optimization problem. Furthermore, we analytically show that the statistical representation performance of SCAD penalty is improved compared with $\ell_1$-based methods, and the minimum representation error under RS assumption is obtained at the edge of RS/RSB phase. The correspondence between the convergence of the existing coordinate descent algorithm and RS/RSB transition is also indicated.