Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
(Submitted on 15 Dec 2016 (v1), last revised 3 Apr 2017 (this version, v3))
Abstract: Using unitary (instead of general) matrices in artificial neural networks (ANNs) is a promising way to solve the gradient explosion/vanishing problem, as well as to enable ANNs to learn long-term correlations in the data. This approach appears particularly promising for Recurrent Neural Networks (RNNs). In this work, we present a new architecture for implementing an Efficient Unitary Neural Network (EUNNs); its main advantages can be summarized as follows. Firstly, the representation capacity of the unitary space in an EUNN is fully tunable, ranging from a subspace of SU(N) to the entire unitary space. Secondly, the computational complexity for training an EUNN is merely $\mathcal{O}(1)$ per parameter. Finally, we test the performance of EUNNs on the standard copying task, the pixel-permuted MNIST digit recognition benchmark as well as the Speech Prediction Test (TIMIT). We find that our architecture significantly outperforms both other state-of-the-art unitary RNNs and the LSTM architecture, in terms of the final performance and/or the wall-clock training speed. EUNNs are thus promising alternatives to RNNs and LSTMs for a wide variety of applications.
Submission history
From: Li Jing [view email][v1] Thu, 15 Dec 2016 20:39:15 GMT (1624kb,D)
[v2] Sun, 26 Feb 2017 19:00:50 GMT (3179kb,D)
[v3] Mon, 3 Apr 2017 17:13:38 GMT (3180kb,D)
Link back to: arXiv, form interface, contact.