Title: Scalable and Optimal Generalized Canonical Correlation Analysis via Alternating Optimization

Abstract: This paper considers generalized (multiview) canonical correlation analysis (GCCA) for large-scale datasets. A memory-efficient and computationally lightweight algorithm is proposed for the classic MAX-VAR GCCA formulation, which is gaining renewed interest in various applications, such as speech recognition and natural language processing. The MAX-VAR GCCA problem can be solved optimally via eigen-decomposition of a matrix that compounds the (whitened) correlation matrices of the views. However, this route can easily lead to memory explosion and a heavy computational burden when the size of the views becomes large. Instead, we propose an alternating optimization (AO)-based algorithm, which avoids instantiating the correlation matrices of the views and thus can achieve substantial saving in memory. The algorithm also maintains data sparsity, which can be exploited to alleviate the computational burden. Consequently, the proposed algorithm is highly scalable. Despite the non-convexity of the MAX-VAR GCCA problem, the proposed iterative algorithm is shown to converge to a globally optimal solution under certain mild conditions. The proposed framework ensures global convergence even when the subproblems are inexactly solved, which can further reduce the complexity in practice. Simulations and large-scale word embedding tasks are employed to showcase the effectiveness of the proposed algorithm.