Title: PUDLE: Implicit Acceleration of Dictionary Learning by Backpropagation

Abstract: The dictionary learning problem, representing data as a combination of a few atoms, has long stood as a popular method for learning representations in statistics and signal processing. The most popular dictionary learning algorithm alternates between sparse coding and dictionary update steps, and rich literature has studied its theoretical convergence. The growing popularity of neurally plausible unfolded sparse coding networks has led to the empirical finding that backpropagation through such networks performs dictionary learning. This paper offers the theoretical proof for these empirical results through PUDLE, a Provable Unfolded Dictionary LEarning method. We provide sufficient conditions on the network initialization and data distribution for model recovery. We highlight the impact of loss, unfolding, and backpropagation on convergence. We discuss proper initializations and strategies to resolve the instability issue concerning backpropagated gradient. This is achieved through tuning network parameters and modification of loss function. We complement our findings through synthetic and image denoising experiments. Finally, we highlight the interpretability of PULDE by deriving a mathematical relation between network weights, its output, and the training data.