References & Citations
Computer Science > Computation and Language
Title: Span Fine-tuning for Pre-trained Language Models
(Submitted on 29 Aug 2021 (v1), last revised 15 Sep 2021 (this version, v2))
Abstract: Pre-trained language models (PrLM) have to carefully manage input units when training on a very large text with a vocabulary consisting of millions of words. Previous works have shown that incorporating span-level information over consecutive words in pre-training could further improve the performance of PrLMs. However, given that span-level clues are introduced and fixed in pre-training, previous methods are time-consuming and lack of flexibility. To alleviate the inconvenience, this paper presents a novel span fine-tuning method for PrLMs, which facilitates the span setting to be adaptively determined by specific downstream tasks during the fine-tuning phase. In detail, any sentences processed by the PrLM will be segmented into multiple spans according to a pre-sampled dictionary. Then the segmentation information will be sent through a hierarchical CNN module together with the representation outputs of the PrLM and ultimately generate a span-enhanced representation. Experiments on GLUE benchmark show that the proposed span fine-tuning method significantly enhances the PrLM, and at the same time, offer more flexibility in an efficient way.
Submission history
From: Rongzhou Bao [view email][v1] Sun, 29 Aug 2021 14:11:38 GMT (1228kb,D)
[v2] Wed, 15 Sep 2021 14:19:57 GMT (614kb,D)
Link back to: arXiv, form interface, contact.