We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning

Abstract: Meta-learning aims at optimizing the hyperparameters of a model class or training algorithm from the observation of data from a number of related tasks. Following the setting of Baxter [1], the tasks are assumed to belong to the same task environment, which is defined by a distribution over the space of tasks and by per-task data distributions. The statistical properties of the task environment thus dictate the similarity of the tasks. The goal of the meta-learner is to ensure that the hyperparameters obtain a small loss when applied for training of a new task sampled from the task environment. The difference between the resulting average loss, known as meta-population loss, and the corresponding empirical loss measured on the available data from related tasks, known as meta-generalization gap, is a measure of the generalization capability of the meta-learner. In this paper, we present novel information-theoretic bounds on the average absolute value of the meta-generalization gap. Unlike prior work [2], our bounds explicitly capture the impact of task relatedness, the number of tasks, and the number of data samples per task on the meta-generalization gap. Task similarity is gauged via the Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences. We illustrate the proposed bounds on the example of ridge regression with meta-learned bias.
Comments: Accepted to ISIT 2021
Subjects: Machine Learning (cs.LG); Information Theory (cs.IT); Signal Processing (eess.SP)
Cite as: arXiv:2101.08390 [cs.LG]
  (or arXiv:2101.08390v3 [cs.LG] for this version)

Submission history

From: Sharu Theresa Jose [view email]
[v1] Thu, 21 Jan 2021 01:38:16 GMT (143kb,D)
[v2] Mon, 25 Jan 2021 04:55:27 GMT (143kb,D)
[v3] Sat, 8 May 2021 09:23:06 GMT (143kb,D)

Link back to: arXiv, form interface, contact.