Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
(Submitted on 2 Dec 2022 (v1), last revised 13 Sep 2023 (this version, v2))
Abstract: We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion. It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the need for shared data. Consequentially, ColD Fusion can give rise to a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based upon. We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was trained on; and (b) is a better starting point for finetuning on unseen datasets. We show that ColD Fusion outperforms RoBERTa and even previous multitask models. Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points on average without any changes to the architecture.
Submission history
From: Shachar Don-Yehiya [view email][v1] Fri, 2 Dec 2022 18:59:04 GMT (8505kb,D)
[v2] Wed, 13 Sep 2023 15:07:01 GMT (12205kb,D)
Link back to: arXiv, form interface, contact.