Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: On the Theory of Transfer Learning: The Importance of Task Diversity
(Submitted on 20 Jun 2020 (v1), last revised 22 Oct 2020 (this version, v2))
Abstract: We provide new statistical guarantees for transfer learning via representation learning--when transfer is achieved by learning a feature representation shared across different tasks. This enables learning on new tasks using far less data than is required to learn them in isolation. Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$. Letting $C(\cdot)$ denote the complexity measure of the function class, we show that for diverse training tasks (1) the sample complexity needed to learn the shared representation across the first $t$ training tasks scales as $C(\mathcal{H}) + t C(\mathcal{F})$, despite no explicit access to a signal from the feature representation and (2) with an accurate estimate of the representation, the sample complexity needed to learn a new task scales only with $C(\mathcal{F})$. Our results depend upon a new general notion of task diversity--applicable to models with general tasks, features, and losses--as well as a novel chain rule for Gaussian complexities. Finally, we exhibit the utility of our general framework in several models of importance in the literature.
Submission history
From: Nilesh Tripuraneni [view email][v1] Sat, 20 Jun 2020 20:33:59 GMT (492kb)
[v2] Thu, 22 Oct 2020 17:19:02 GMT (22478kb)
Link back to: arXiv, form interface, contact.