We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Numerical Optimizations for Weighted Low-rank Estimation on Language Model

Abstract: Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.
Comments: long paper EMNLP 2022
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Journal reference: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Cite as: arXiv:2211.09718 [cs.CL]
  (or arXiv:2211.09718v2 [cs.CL] for this version)

Submission history

From: Ting Hua [view email]
[v1] Wed, 2 Nov 2022 00:58:02 GMT (558kb,D)
[v2] Thu, 15 Dec 2022 22:45:46 GMT (558kb,D)

Link back to: arXiv, form interface, contact.