We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning

Abstract: Parameter-efficient fine-tuning approaches have recently garnered a lot of attention. Having considerably lower number of trainable weights, these methods can bring about scalability and computational effectiveness. In this paper, we look for optimal sub-networks and investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module in BERT can act as a winning ticket: fine-tuning each specific module while keeping the rest of the network frozen can lead to comparable performance to the full fine-tuning. Among different modules, LayerNorms exhibit the best capacity for knowledge transfer with limited trainable weights, to the extent that, with only 0.003% of all parameters in the layer-wise analysis, they show acceptable performance on various target tasks. On the reasons behind their effectiveness, we argue that their notable performance could be attributed to their high-magnitude weights compared to that of the other modules in the pre-trained BERT.
Comments: Accepted at EMNLP 2022 (main conference), this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2302.00378 [cs.CL]
  (or arXiv:2302.00378v2 [cs.CL] for this version)

Submission history

From: Mohammad Akbar-Tajari [view email]
[v1] Wed, 1 Feb 2023 11:20:18 GMT (1016kb,D)
[v2] Wed, 22 Feb 2023 16:56:58 GMT (344kb,D)

Link back to: arXiv, form interface, contact.