We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Gradient-based Bi-level Optimization for Deep Learning: A Survey

Abstract: Bi-level optimization, especially the gradient-based category, has been widely used in the deep learning community including hyperparameter optimization and meta knowledge extraction. Bi-level optimization embeds one problem within another and the gradient-based category solves the outer level task by computing the hypergradient, which is much more efficient than classical methods such as the evolutionary algorithm. In this survey, we first give a formal definition of the gradient-based bi-level optimization. Secondly, we illustrate how to formulate a research problem as a bi-level optimization problem, which is of great practical use for beginners. More specifically, there are two formulations: the single-task formulation to optimize hyperparameters such as regularization parameters and the distilled data, and the multi-task formulation to extract meta knowledge such as the model initialization. With a bi-level formulation, we then discuss four bi-level optimization solvers to update the outer variable including explicit gradient update, proxy update, implicit function update, and closed-form update. Last but not least, we conclude the survey by pointing out the great potential of gradient-based bi-level optimization on science problems (AI4Science).
Comments: AI4Science; Bi-level Optimization; Hyperparameter Optimization; Meta Learning; Implicit Function
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
Cite as: arXiv:2207.11719 [cs.LG]
  (or arXiv:2207.11719v3 [cs.LG] for this version)

Submission history

From: Can Chen [view email]
[v1] Sun, 24 Jul 2022 11:23:31 GMT (1475kb,D)
[v2] Thu, 4 Aug 2022 11:22:23 GMT (1477kb,D)
[v3] Tue, 7 Feb 2023 15:15:36 GMT (1484kb,D)

Link back to: arXiv, form interface, contact.