Current browse context:
cs.AI
Change to browse by:
References & Citations
Computer Science > Artificial Intelligence
Title: Generative Bridging Network in Neural Sequence Prediction
(Submitted on 28 Jun 2017 (v1), revised 20 Aug 2017 (this version, v3), latest version 29 Nov 2018 (v6))
Abstract: Maximum Likelihood Estimation(MLE) has been known to pose data sparsity challenge in sequence prediction tasks, in order to alleviate data sparseness, we propose a novel framework to train sequence model via a bridging process. Unlike MLE which optimizes the sequence generator by directly maximizing the likelihood of ground truth sequence given the input, our proposed framework designs a bridge to connect generator with ground truth. During training, we first follow certain constraints to transform the pointwise ground truth as a bridge distribution, then match the generator's output distribution with the transformed bridge distribution by minimizing their KL-divergence. By imposing different constraints, bridge distribution will adopt different properties. In order to increase output diversity, enhance language smoothness and lower learning burden, we design three different regularization constraints to construct different bridge distributions. Combining these bridges with sequence generator, we can build three parallel generative bridging networks, namely uniform GBN, language-model GBN and coaching GBN. Experimental results on two recognized sequence prediction tasks have shown that GBN can yield significant improvements over the baseline system. Furthermore, we draw samples from three bridge distributions to analyze their different properties and verify their influences on the sequence model learning.
Submission history
From: Wenhu Chen [view email][v1] Wed, 28 Jun 2017 07:44:17 GMT (908kb,D)
[v2] Sun, 13 Aug 2017 16:24:41 GMT (0kb,I)
[v3] Sun, 20 Aug 2017 11:17:13 GMT (1714kb,D)
[v4] Tue, 31 Oct 2017 17:49:11 GMT (1401kb,D)
[v5] Sat, 17 Mar 2018 22:03:58 GMT (3845kb,D)
[v6] Thu, 29 Nov 2018 22:29:53 GMT (1901kb,D)
Link back to: arXiv, form interface, contact.