Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Self-Normalized Importance Sampling for Neural Language Modeling
(Submitted on 11 Nov 2021 (v1), last revised 17 Jun 2022 (this version, v2))
Abstract: To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform well, as long as an extra correction step is done, where the intended class posterior probability is recovered from the raw model outputs. In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. Through self-normalized language model training as well as lattice rescoring experiments, we show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.
Submission history
From: Yingbo Gao [view email][v1] Thu, 11 Nov 2021 16:57:53 GMT (264kb,D)
[v2] Fri, 17 Jun 2022 09:42:59 GMT (505kb,D)
Link back to: arXiv, form interface, contact.