math.OC

# Title: Some convergent results for Backtracking Gradient Descent method on Banach spaces

Abstract: Our main result concerns the following condition:
{\bf Condition C.} Let $X$ be a Banach space. A $C^1$ function $f:X\rightarrow \mathbb{R}$ satisfies Condition C if whenever $\{x_n\}$ weakly converges to $x$ and $\lim _{n\rightarrow\infty}||\nabla f(x_n)||=0$, then $\nabla f(x)=0$.
We assume that there is given a canonical isomorphism between $X$ and its dual $X^*$, for example when $X$ is a Hilbert space.
{\bf Theorem.} Let $X$ be a reflexive, complete Banach space and $f:X\rightarrow \mathbb{R}$ be a $C^2$ function which satisfies Condition C. Moreover, we assume that for every bounded set $S\subset X$, then $\sup _{x\in S}||\nabla ^2f(x)||<\infty$. We choose a random point $x_0\in X$ and construct by the Local Backtracking GD procedure (which depends on $3$ hyper-parameters $\alpha ,\beta ,\delta _0$, see later for details) the sequence $x_{n+1}=x_n-\delta (x_n)\nabla f(x_n)$. Then we have:
1) Every cluster point of $\{x_n\}$, in the {\bf weak} topology, is a critical point of $f$.
2) Either $\lim _{n\rightarrow\infty}f(x_n)=-\infty$ or $\lim _{n\rightarrow\infty}||x_{n+1}-x_n||=0$.
3) Here we work with the weak topology. Let $\mathcal{C}$ be the set of critical points of $f$. Assume that $\mathcal{C}$ has a bounded component $A$. Let $\mathcal{B}$ be the set of cluster points of $\{x_n\}$. If $\mathcal{B}\cap A\not= \emptyset$, then $\mathcal{B}\subset A$ and $\mathcal{B}$ is connected.
4) Assume that $X$ is separable. Then for generic choices of $\alpha ,\beta ,\delta _0$ and the initial point $x_0$, if the sequence $\{x_n\}$ converges - in the {\bf weak} topology, then the limit point cannot be a saddle point.
 Comments: More details and improvements added, including: C^1 convex functions satisfy Condition C, normalized duality mapping, prevalence and shyness of sets in Banach spaces, hereditary Lindelof property of weak topology. Several typos fixed. 10 pages Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Analysis of PDEs (math.AP); Functional Analysis (math.FA); Machine Learning (stat.ML) Cite as: arXiv:2001.05768 [math.OC] (or arXiv:2001.05768v2 [math.OC] for this version)

## Submission history

From: Tuyen Truong [view email]
[v1] Thu, 16 Jan 2020 12:49:42 GMT (9kb)
[v2] Wed, 22 Jan 2020 13:40:10 GMT (12kb)

Link back to: arXiv, form interface, contact.