References & Citations
Computer Science > Computation and Language
Title: Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts
(Submitted on 25 May 2022 (v1), last revised 22 Nov 2022 (this version, v2))
Abstract: Recent works suggest that transformer models are capable of multi-tasking on diverse NLP tasks and adapting to new tasks efficiently. However, the potential of these multi-task models may be limited as they use the same set of parameters for all tasks. In contrast, humans tackle tasks in a more flexible way, by making proper presumptions on what skills and knowledge are relevant and executing only the necessary computations. Inspired by this, we propose to use task-level mixture-of-expert models, which has a collection of transformer layers (i.e., experts) and a router component that chooses from these experts dynamically and flexibly. We find that these models help improve the average performance gain (ARG) metric by 2.6% when adapting to unseen tasks in the few-shot setting and by 5.6% in the zero-shot generalization setting. Further, we show that the learned routing decisions partly rediscover human categorization of NLP tasks -- certain experts are strongly associated with extractive tasks, some with classification tasks, and some with tasks requiring world knowledge.
Submission history
From: Qinyuan Ye [view email][v1] Wed, 25 May 2022 11:59:05 GMT (6816kb,D)
[v2] Tue, 22 Nov 2022 00:15:25 GMT (6837kb,D)
Link back to: arXiv, form interface, contact.