We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.NI

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Networking and Internet Architecture

Title: Compressing LSTM Networks by Matrix Product Operators

Abstract: Long Short Term Memory(LSTM) models are the building blocks of many state-of-the-art natural language processing(NLP) and speech enhancement(SE) algorithms. However, there are a large number of parameters in an LSTM model. This usually consumes a large number of resources to train the LSTM model. Also, LSTM models suffer from computational inefficiency in the inference phase. Existing model compression methods (e.g., model pruning) can only discriminate based on the magnitude of model parameters, ignoring the issue of importance distribution based on the model information. Here we introduce the MPO decomposition, which describes the local correlation of quantum states in quantum many-body physics and is used to represent the large model parameter matrix in a neural network, which can compress the neural network by truncating the unimportant information in the weight matrix. In this paper, we propose a matrix product operator(MPO) based neural network architecture to replace the LSTM model. The effective representation of neural networks by MPO can effectively reduce the computational consumption of training LSTM models on the one hand, and speed up the computation in the inference phase of the model on the other hand. We compare the MPO-LSTM model-based compression model with the traditional LSTM model with pruning methods on sequence classification, sequence prediction, and speech enhancement tasks in our experiments. The experimental results show that our proposed neural network architecture based on the MPO approach significantly outperforms the pruning approach.
Comments: 2 figures, 5 tables
Subjects: Networking and Internet Architecture (cs.NI); Computational Physics (physics.comp-ph); Quantum Physics (quant-ph)
Cite as: arXiv:2012.11943 [cs.NI]
  (or arXiv:2012.11943v3 [cs.NI] for this version)

Submission history

From: Ze-Feng Gao [view email]
[v1] Tue, 22 Dec 2020 11:50:06 GMT (186kb,D)
[v2] Mon, 4 Jan 2021 10:48:12 GMT (141kb,D)
[v3] Thu, 31 Mar 2022 05:54:05 GMT (143kb,D)

Link back to: arXiv, form interface, contact.