Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
(Submitted on 5 Nov 2016 (v1), last revised 27 Feb 2017 (this version, v2))
Abstract: In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of $6.28\%$. This is comparable to the state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.
Submission history
From: Adji Bousso Dieng [view email][v1] Sat, 5 Nov 2016 21:25:07 GMT (4054kb,D)
[v2] Mon, 27 Feb 2017 03:03:38 GMT (4850kb,D)
Link back to: arXiv, form interface, contact.