We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

Abstract: State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in this https URL
Comments: accepted in ACL, 2021
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2106.04489 [cs.CL]
  (or arXiv:2106.04489v1 [cs.CL] for this version)

Submission history

From: Rabeeh Karimi Mahabadi [view email]
[v1] Tue, 8 Jun 2021 16:16:40 GMT (5589kb,D)

Link back to: arXiv, form interface, contact.