We gratefully acknowledge support from
the Simons Foundation and member institutions.

Optimization and Control

New submissions

[ total of 28 entries: 1-28 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Fri, 17 Jan 20

[1]  arXiv:2001.05537 [pdf, other]
Title: Accelerated Dual-Averaging Primal-Dual Method for Composite Convex Minimization
Journal-ref: Optimization Methods and Software 2020
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Machine Learning (stat.ML)

Dual averaging-type methods are widely used in industrial machine learning applications due to their ability to promoting solution structure (e.g., sparsity) efficiently. In this paper, we propose a novel accelerated dual-averaging primal-dual algorithm for minimizing a composite convex function. We also derive a stochastic version of the proposed method which solves empirical risk minimization, and its advantages on handling sparse data are demonstrated both theoretically and empirically.

[2]  arXiv:2001.05739 [pdf, other]
Title: Centering ADMM for the Semidefinite Relaxation of the QAP
Subjects: Optimization and Control (math.OC)

We propose a new method for solving the semidefinite (SD) relaxation of the quadratic assignment problem (QAP), called the Centering ADMM. The Centering ADMM is an alternating direction method of multipliers (ADMM) combining the centering steps used in the interior-point method. The first stage of the Centering ADMM updates the iterate so that it approaches the central path by incorporating a barrier function term into the objective function, as in the interior-point method. If the current iterate is sufficiently close to the central path with a sufficiently small value of the barrier parameter, the method switches to the Standard ADMM. To observe the effect of the centering steps, we conducted numerical experiments with SD relaxation problems of instances in the QAPLIB. The results demonstrate that the centering steps are quite efficient for some classes of instances.

[3]  arXiv:2001.05740 [pdf, other]
Title: Lifting to Passivity for $\mathcal{H}_2$-Gain-Scheduling Synthesis with Full Block Scalings
Subjects: Optimization and Control (math.OC)

We focus on the $\mathcal{H}_2$-gain-scheduling synthesis problem for time-varying parametric scheduling blocks with full block scalings. Recently, we have presented a solution of this problem for $D$- and positive real scalings by relying on a convexifying transformation for the controller parameters and by guaranteeing finiteness of the $\mathcal{H}_2$-norm for the closed-loop system with suitable linear fractional plant and controller representations. We extend these methods to full block scalings by designing a triangular scheduling function and by introducing a new lifting technique for gain-scheduled synthesis that enables convexification.

[4]  arXiv:2001.05768 [pdf, ps, other]
Title: Some convergent results for Backtracking Gradient Descent method on Banach spaces
Comments: 8 pages
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Analysis of PDEs (math.AP); Functional Analysis (math.FA); Machine Learning (stat.ML)

Our main result concerns the following condition:
{\bf Condition C.} Let $X$ be a Banach space. A $C^1$ function $f:X\rightarrow \mathbb{R}$ satisfies Condition C if whenever $\{x_n\}$ weakly converges to $x$ and $\lim _{n\rightarrow\infty}||\nabla f(x_n)||=0$, then $\nabla f(x)=0$.
We assume that there is given a canonical isomorphism between $X$ and its dual $X^*$, for example when $X$ is a Hilbert space.
{\bf Theorem.} Let $X$ be a reflexive, complete Banach space and $f:X\rightarrow \mathbb{R}$ be a $C^2$ function which satisfies Condition C. Moreover, we assume that for every bounded set $S\subset X$, then $\sup _{x\in S}||\nabla ^2f(x)||<\infty$. We choose a random point $x_0\in X$ and construct by the Local Backtracking GD procedure (which depends on $3$ hyper-parameters $\alpha ,\beta ,\delta _0$, see later for details) the sequence $x_{n+1}=x_n-\delta (x_n)\nabla f(x_n)$. Then we have:
1) Every cluster point of $\{x_n\}$, in the {\bf weak} topology, is a critical point of $f$.
2) Either $\lim _{n\rightarrow\infty}f(x_n)=-\infty$ or $\lim _{n\rightarrow\infty}||x_{n+1}-x_n||=0$.
3) Here we work with the weak topology. Let $\mathcal{C}$ be the set of critical points of $f$. Assume that $\mathcal{C}$ has a bounded component $A$. Let $\mathcal{B}$ be the set of cluster points of $\{x_n\}$. If $\mathcal{B}\cap A\not= \emptyset$, then $\mathcal{B}\subset A$ and $\mathcal{B}$ is connected.
4) Assume that $f$ has at most countably many saddle points. Then for generic choices of $\alpha ,\beta ,\delta _0$ and the initial point $x_0$, if the sequence $\{x_n\}$ converges - in the {\bf weak} topology, then the limit point cannot be a saddle point.

[5]  arXiv:2001.05795 [pdf, other]
Title: Stable and Robust LQR Design via Scenario Approach
Comments: 14 pages, 3 figures
Subjects: Optimization and Control (math.OC)

Linear Quadratic Regulator (LQR) design is one of the most classical optimal control problems, whose well-known solution is an input sequence expressed as a state-feedback. In this work, finite-horizon and discrete-time LQR is solved under stability constraints and uncertain system dynamics. The resulting feedback controller balances cost value and closed-loop stability. Robustness of the solution is modeled using the scenario approach, without requiring any probabilistic description of the uncertainty in the system matrices. The new methods are tested and compared on the Leslie growth model, where we control population size while minimizing a suitable finite-horizon cost function.

[6]  arXiv:2001.05915 [pdf, ps, other]
Title: Adaptive iterative singular value thresholding algorithm to low-rank matrix recovery
Subjects: Optimization and Control (math.OC)

The problem of recovering a low-rank matrix from the linear constraints, known as affine matrix rank minimization problem, has been attracting extensive attention in recent years. In general, affine matrix rank minimization problem is a NP-hard. In our latest work, a non-convex fraction function is studied to approximate the rank function in affine matrix rank minimization problem and translate the NP-hard affine matrix rank minimization problem into a transformed affine matrix rank minimization problem. A scheme of iterative singular value thresholding algorithm is generated to solve the regularization transformed affine matrix rank minimization problem. However, one of the drawbacks for our iterative singular value thresholding algorithm is that the parameter $a$, which influences the behaviour of non-convex fraction function in the regularization transformed affine matrix rank minimization problem, needs to be determined manually in every simulation. In fact, how to determine the optimal parameter $a$ is not an easy problem. Here instead, in this paper, we will generate an adaptive iterative singular value thresholding algorithm to solve regularization transformed affine matrix rank minimization problem. When doing so, our new algorithm will be intelligent both for the choice of the regularization parameter $\lambda$ and the parameter $a$.

Cross-lists for Fri, 17 Jan 20

[7]  arXiv:2001.05530 (cross-list from math.NA) [pdf, other]
Title: Biorthogonal greedy algorithms in convex optimization
Subjects: Numerical Analysis (math.NA); Optimization and Control (math.OC)

The study of greedy approximation in the context of convex optimization is becoming a promising research direction as greedy algorithms are actively being employed to construct sparse minimizers for convex functions with respect to given sets of elements. In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms for the minimization of convex functions on Banach spaces. Specifically, we define the class of Weak Biorthogonal Greedy Algorithms for convex optimization that contains a wide range of greedy algorithms. We analyze the introduced class of algorithms and establish the properties of convergence, rate of convergence, and numerical stability, which is understood in the sense that the steps of the algorithm are allowed to be performed not precisely but with controlled computational inaccuracies. We show that the following well-known algorithms for convex optimization --- the Weak Chebyshev Greedy Algorithm (co) and the Weak Greedy Algorithm with Free Relaxation (co) --- belong to this class, and introduce a new algorithm --- the Rescaled Weak Relaxed Greedy Algorithm (co). Presented numerical experiments demonstrate the practical performance of the aforementioned greedy algorithms in the setting of convex minimization as compared to optimization with regularization, which is the conventional approach of constructing sparse minimizers.

[8]  arXiv:2001.05781 (cross-list from math.NA) [pdf, ps, other]
Title: Optimal parameter for the SOR-like iteration method for solving the system of absolute value equations
Comments: 15 pages, 5 figures, 6 tables
Subjects: Numerical Analysis (math.NA); Optimization and Control (math.OC)

The absolute value equations (AVE) $Ax - |x| - b = 0$ is of interest of the optimization community. Recently, the SOR-like iteration method has been developed (Ke and Ma [{\em Appl. Math. Comput.}, 311:195--202, 2017]) and shown to be efficient for numerically solving the AVE with $\nu=\|A^{-1}\|_2<1$ (Ke and Ma [{\em Appl. Math. Comput.}, 311:195--202, 2017]; Guo, Wu and Li [{\em Appl. Math. Lett.}, 97:107--113, 2019]). Since the SOR-like iteration method is one-parameter-dependent, it is an important problem to determine the optimal iteration parameter. In this paper, we revisit the convergence conditions of the SOR-like iteration method proposed by Ke and Ma ([{\em Appl. Math. Comput.}, 311:195--202, 2017]). Furthermore, we explore the optimal parameter which minimizes $\|T(\omega)\|_2$ and the approximate optimal parameter which minimizes $\eta=\max\{|1-\omega|,\nu\omega^2\}$. The optimal and approximate optimal parameters are iteration-independent. Numerical results demonstrate that the SOR-like iteration method with the optimal parameter is superior to that with the approximate optimal parameter proposed by Guo, Wu and Li ([{\em Appl. Math. Lett.}, 97:107--113, 2019]).

[9]  arXiv:2001.05820 (cross-list from math.CO) [pdf, other]
Title: Probabilistic values for simplicial complexes
Authors: Ivan Martino
Comments: 23 pages, 1 figure
Subjects: Combinatorics (math.CO); Optimization and Control (math.OC)

In this manuscript, we define and study probabilistic values for cooperative games on simplicial complexes. Inspired by the work of Weber "Probabilistic values for games", we establish the new theory step by step, following the classical axiomatization, i.e. using the linearity axiom, the dummy axiom, etc.
Furthermore, we define Shapley values on simplicial complexes generalizing the classical notion in literature. Remarkably, the traditional axiomatization of Shapley values can be extended to this general setting for a rather interesting class of complexes that generalize the notion of vertex-transitive graphs and vertex-homogeneous simplicial complexes. These combinatorial objects are very popular in the literature because of the study of Evasiveness Conjecture in Complexity Theory.

[10]  arXiv:2001.05992 (cross-list from cs.LG) [pdf, other]
Title: Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
Comments: International Conference on Learning Representations (ICLR) 2020
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

Replacements for Fri, 17 Jan 20

[11]  arXiv:1807.02198 (replaced) [pdf, ps, other]
Title: The Radius of Metric Subregularity
Comments: 20 pages
Journal-ref: Set-Valued and Variational Analysis (2020)
Subjects: Optimization and Control (math.OC)
[12]  arXiv:1808.03864 (replaced) [pdf, ps, other]
Title: Spectral norm of a symmetric tensor and its computation
Comments: 31 pages. arXiv admin note: substantial text overlap with arXiv:1608.01354, to appear in Mathematics of Computation, AMS
Subjects: Optimization and Control (math.OC); Mathematical Physics (math-ph)
[13]  arXiv:1809.09847 (replaced) [pdf, ps, other]
Title: S-SPADE Done Right: Detailed Study of the Sparse Audio Declipper Algorithms
Subjects: Optimization and Control (math.OC); Audio and Speech Processing (eess.AS); Signal Processing (eess.SP)
[14]  arXiv:1812.00734 (replaced) [pdf, other]
Title: Market Integration of HVDC Lines: Internalizing HVDC Losses in Market Clearing
Comments: Submitted to "IEEE Transactions on Power Systems" on December 3, 2018 - Revised on May 28, 2019 - Accepted on July 21, 2019 - Published on January 7, 2020
Journal-ref: IEEE Transactions on Power Systems, vol. 35, no. 1, pp. 451-461, January 2020
Subjects: Optimization and Control (math.OC)
[15]  arXiv:1904.01196 (replaced) [pdf, other]
Title: Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimization
Subjects: Optimization and Control (math.OC)
[16]  arXiv:1904.07028 (replaced) [pdf, other]
Title: Euler's optimal profile problem
Subjects: Optimization and Control (math.OC)
[17]  arXiv:1905.05257 (replaced) [pdf, other]
Title: Oracle-Based Algorithms for Binary Two-Stage Robust Optimization
Subjects: Optimization and Control (math.OC)
[18]  arXiv:1905.05557 (replaced) [pdf, ps, other]
Title: An analytical bound on the fleet size in vehicle routing problems: a dynamic programming approach
Subjects: Optimization and Control (math.OC); Discrete Mathematics (cs.DM)
[19]  arXiv:1907.02989 (replaced) [pdf, ps, other]
Title: An Optimality Gap Test for a Semidefinite Relaxation of a Quadratic Program with Two Quadratic Constraints
Subjects: Optimization and Control (math.OC)
[20]  arXiv:1907.07580 (replaced) [pdf, ps, other]
Title: Feature-driven Improvement of Renewable Energy Forecasting and Trading
Comments: 10 pages, 6 figures
Subjects: Optimization and Control (math.OC); Applications (stat.AP); Machine Learning (stat.ML)
[21]  arXiv:1910.08123 (replaced) [pdf, other]
Title: Optimization and Learning with Information Streams: Time-varying Algorithms and Applications
Comments: Accepted for publication in IEEE Signal Processing Magazine. Limit of 40 references
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Signal Processing (eess.SP)
[22]  arXiv:2001.04341 (replaced) [pdf, other]
Title: Information Newton's flow: second-order optimization method in probability space
Authors: Yifei Wang, Wuchen Li
Comments: 43 pages
Subjects: Optimization and Control (math.OC); Machine Learning (cs.LG); Machine Learning (stat.ML)
[23]  arXiv:2001.04729 (replaced) [pdf, ps, other]
Title: A unified method to decentralized state inference and fault diagnosis/prediction of discrete-event systems
Authors: Kuize Zhang
Comments: 23 pages, 3 figures
Subjects: Optimization and Control (math.OC); Systems and Control (eess.SY)
[24]  arXiv:1901.00279 (replaced) [pdf, other]
Title: Elimination of All Bad Local Minima in Deep Learning
Comments: Accepted to appear in AISTATS 2020
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC); Machine Learning (stat.ML)
[25]  arXiv:1902.04376 (replaced) [pdf, ps, other]
Title: An adaptive stochastic optimization algorithm for resource allocation
Comments: ALT2020, 45 pages, 9 figures
Journal-ref: Proceedings of Machine Learning Research (PMLR), volume 117, 2020
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Optimization and Control (math.OC)
[26]  arXiv:1906.02351 (replaced) [pdf, other]
Title: On the Convergence of SARAH and Beyond
Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
[27]  arXiv:1907.04371 (replaced) [pdf, other]
Title: Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization
Comments: Accepted to appear in AISTATS 2020. Code available at: this https URL
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Optimization and Control (math.OC)
[28]  arXiv:2001.04756 (replaced) [pdf, other]
Title: Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC); Optimization and Control (math.OC); Machine Learning (stat.ML)
[ total of 28 entries: 1-28 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, math, recent, 2001, contact, help  (Access key information)