References & Citations
Computer Science > Computation and Language
Title: Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
(Submitted on 29 Dec 2020 (v1), last revised 18 Mar 2021 (this version, v2))
Abstract: Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embedding. Source code is freely available at this https URL
Submission history
From: Xuebo Liu [view email][v1] Tue, 29 Dec 2020 14:26:59 GMT (185kb,D)
[v2] Thu, 18 Mar 2021 11:46:55 GMT (186kb,D)
Link back to: arXiv, form interface, contact.