References & Citations
Computer Science > Computation and Language
Title: Numerical Optimizations for Weighted Low-rank Estimation on Language Model
(Submitted on 2 Nov 2022 (v1), last revised 15 Dec 2022 (this version, v2))
Abstract: Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.
Submission history
From: Ting Hua [view email][v1] Wed, 2 Nov 2022 00:58:02 GMT (558kb,D)
[v2] Thu, 15 Dec 2022 22:45:46 GMT (558kb,D)
Link back to: arXiv, form interface, contact.