References & Citations
Computer Science > Computation and Language
Title: Alleviating the Inequality of Attention Heads for Neural Machine Translation
(Submitted on 21 Sep 2020 (v1), last revised 31 Aug 2022 (this version, v2))
Abstract: Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.
Submission history
From: Zewei Sun [view email][v1] Mon, 21 Sep 2020 08:14:30 GMT (7171kb,D)
[v2] Wed, 31 Aug 2022 11:50:22 GMT (102kb,D)
Link back to: arXiv, form interface, contact.