We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling

Abstract: For sequence models with large vocabularies, a majority of network parameters lie in the input and output layers. In this work, we describe a new method, DeFINE, for learning deep token representations efficiently. Our architecture uses a hierarchical structure with novel skip-connections which allows for the use of low dimensional input and output layers, reducing total parameters and training time while delivering similar or better performance versus existing methods. DeFINE can be incorporated easily in new or existing sequence models. Compared to state-of-the-art methods including adaptive input representations, this technique results in a 6% to 20% drop in perplexity. On WikiText-103, DeFINE reduces the total parameters of Transformer-XL by half with minimal impact on performance. On the Penn Treebank, DeFINE improves AWD-LSTM by 4 points with a 17% reduction in parameters, achieving comparable performance to state-of-the-art methods with fewer parameters. For machine translation, DeFINE improves the efficiency of the Transformer model by about 1.4 times while delivering similar performance.
Comments: Accepted at ICLR 2020
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:1911.12385 [cs.CL]
  (or arXiv:1911.12385v2 [cs.CL] for this version)

Submission history

From: Sachin Mehta [view email]
[v1] Wed, 27 Nov 2019 19:09:41 GMT (7780kb,D)
[v2] Thu, 6 Feb 2020 01:32:06 GMT (7778kb,D)

Link back to: arXiv, form interface, contact.