Current browse context:
math.OC
Change to browse by:
References & Citations
Mathematics > Optimization and Control
Title: Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
(Submitted on 23 Sep 2019 (v1), last revised 9 Apr 2020 (this version, v4))
Abstract: Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.
Submission history
From: Edouard Pauwels [view email][v1] Mon, 23 Sep 2019 11:39:16 GMT (39kb,D)
[v2] Mon, 7 Oct 2019 15:00:51 GMT (42kb,D)
[v3] Fri, 27 Mar 2020 15:16:30 GMT (39kb,D)
[v4] Thu, 9 Apr 2020 09:43:26 GMT (39kb,D)
Link back to: arXiv, form interface, contact.