References & Citations
Computer Science > Computation and Language
Title: Learning Sparse Sharing Architectures for Multiple Tasks
(Submitted on 12 Nov 2019 (v1), last revised 16 Nov 2019 (this version, v2))
Abstract: Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanism depends on the relations among the tasks, which is not easy since it is difficult to understand the underlying shared factors among these tasks. In this paper, we propose a novel parameter sharing mechanism, named \emph{Sparse Sharing}. Given multiple tasks, our approach automatically finds a sparse sharing structure. We start with an over-parameterized base network, from which each task extracts a subnetwork. The subnetworks of multiple tasks are partially overlapped and trained in parallel. We show that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework. We conduct extensive experiments on three sequence labeling tasks. Compared with single-task models and three typical multi-task learning baselines, our proposed approach achieves consistent improvement while requiring fewer parameters.
Submission history
From: Tianxiang Sun [view email][v1] Tue, 12 Nov 2019 17:50:35 GMT (973kb,D)
[v2] Sat, 16 Nov 2019 13:53:10 GMT (1609kb,D)
Link back to: arXiv, form interface, contact.