Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
(Submitted on 22 Sep 2020 (v1), last revised 6 Sep 2021 (this version, v5))
Abstract: From CNNs to attention mechanisms, encoding inductive biases into neural networks has been a fruitful source of improvement in machine learning. Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations. However, since auxiliary losses are minimized only on training data, they suffer from the same generalization gap as regular task losses. Moreover, by adding a term to the loss function, the model optimizes a different objective than the one we care about. In this work we address both problems: first, we take inspiration from \textit{transductive learning} and note that after receiving an input but before making a prediction, we can fine-tune our networks on any unsupervised loss. We call this process {\em tailoring}, because we customize the model to each input to ensure our prediction satisfies the inductive bias. Second, we formulate {\em meta-tailoring}, a nested optimization similar to that in meta-learning, and train our models to perform well on the task objective after adapting them using an unsupervised loss. The advantages of tailoring and meta-tailoring are discussed theoretically and demonstrated empirically on a diverse set of examples.
Submission history
From: Ferran Alet [view email][v1] Tue, 22 Sep 2020 15:26:24 GMT (7052kb,D)
[v2] Wed, 14 Oct 2020 00:43:24 GMT (8530kb,D)
[v3] Wed, 10 Feb 2021 18:43:40 GMT (5305kb,D)
[v4] Thu, 19 Aug 2021 17:02:21 GMT (6190kb,D)
[v5] Mon, 6 Sep 2021 15:26:52 GMT (6190kb,D)
Link back to: arXiv, form interface, contact.