We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Fisher Task Distance and Its Application in Neural Architecture Search

Abstract: We formulate an asymmetric (or non-commutative) distance between tasks based on Fisher Information Matrices, called Fisher task distance. This distance represents the complexity of transferring the knowledge from one task to another. We provide a proof of consistency for our distance through theorems and experiments on various classification tasks from MNIST, CIFAR-10, CIFAR-100, ImageNet, and Taskonomy datasets. Next, we construct an online neural architecture search framework using the Fisher task distance, in which we have access to the past learned tasks. By using the Fisher task distance, we can identify the closest learned tasks to the target task, and utilize the knowledge learned from these related tasks for the target task. Here, we show how the proposed distance between a target task and a set of learned tasks can be used to reduce the neural architecture search space for the target task. The complexity reduction in search space for task-specific architectures is achieved by building on the optimized architectures for similar tasks instead of doing a full search and without using this side information. Experimental results for tasks in MNIST, CIFAR-10, CIFAR-100, ImageNet datasets demonstrate the efficacy of the proposed approach and its improvements, in terms of the performance and the number of parameters, over other gradient-based search methods, such as ENAS, DARTS, PC-DARTS.
Comments: Published in IEEE Access, Volume 10, 2022
Subjects: Machine Learning (cs.LG); Image and Video Processing (eess.IV); Machine Learning (stat.ML)
DOI: 10.1109/ACCESS.2022.3171741
Cite as: arXiv:2103.12827 [cs.LG]
  (or arXiv:2103.12827v5 [cs.LG] for this version)

Submission history

From: Cat Le [view email]
[v1] Tue, 23 Mar 2021 20:43:31 GMT (1329kb)
[v2] Thu, 25 Mar 2021 14:13:56 GMT (1329kb)
[v3] Wed, 8 Sep 2021 19:48:59 GMT (2574kb)
[v4] Fri, 21 Jan 2022 22:23:26 GMT (13185kb,D)
[v5] Sat, 30 Apr 2022 04:40:37 GMT (13180kb,D)

Link back to: arXiv, form interface, contact.