References & Citations
Computer Science > Computation and Language
Title: FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue
(Submitted on 12 May 2022 (v1), last revised 13 Oct 2022 (this version, v2))
Abstract: Task transfer, transferring knowledge contained in related tasks, holds the promise of reducing the quantity of labeled data required to fine-tune language models. Dialogue understanding encompasses many diverse tasks, yet task transfer has not been thoroughly studied in conversational AI. This work explores conversational task transfer by introducing FETA: a benchmark for few-sample task transfer in open-domain dialogue. FETA contains two underlying sets of conversations upon which there are 10 and 7 tasks annotated, enabling the study of intra-dataset task transfer; task transfer without domain adaptation. We utilize three popular language models and three learning algorithms to analyze the transferability between 132 source-target task pairs and create a baseline for future work. We run experiments in the single- and multi-source settings and report valuable findings, e.g., most performance trends are model-specific, and span extraction and multiple-choice tasks benefit the most from task transfer. In addition to task transfer, FETA can be a valuable resource for future research into the efficiency and generalizability of pre-training datasets and model architectures, as well as for learning settings such as continual and multitask learning.
Submission history
From: Alon Albalak [view email][v1] Thu, 12 May 2022 17:59:00 GMT (4961kb,D)
[v2] Thu, 13 Oct 2022 21:36:38 GMT (4963kb,D)
Link back to: arXiv, form interface, contact.