References & Citations
Computer Science > Computation and Language
Title: Optimizing Deeper Transformers on Small Datasets
(Submitted on 30 Dec 2020 (v1), last revised 31 May 2021 (this version, v4))
Abstract: It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during fine-tuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Text-to-SQL semantic parsing and logical reading comprehension. In particular, we successfully train $48$ layers of transformers, comprising $24$ fine-tuned layers from pre-trained RoBERTa and $24$ relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider. We achieve this by deriving a novel Data-dependent Transformer Fixed-update initialization scheme (DT-Fixup), inspired by the prior T-Fixup work. Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.
Submission history
From: Peng Xu [view email][v1] Wed, 30 Dec 2020 22:53:49 GMT (417kb,D)
[v2] Wed, 19 May 2021 17:12:23 GMT (459kb,D)
[v3] Thu, 27 May 2021 16:53:14 GMT (912kb,D)
[v4] Mon, 31 May 2021 16:45:47 GMT (913kb,D)
Link back to: arXiv, form interface, contact.