We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Optimization and Control

Title: Deep Neural Nets with Fixed Bias Configuration

Abstract: For any given neural network architecture a permutation of weights and biases results in the same functional network. This implies that optimization algorithms used to `train' or `learn' the network are faced with a very large number (in the millions even for small networks) of equivalent optimal solutions in the parameter space. To the best of our knowledge, this observation is absent in the literature. In order to narrow down the parameter search space, a novel technique is introduced in order to fix the bias vector configurations to be monotonically increasing. This is achieved by augmenting a typical learning problem with inequality constraints on the bias vectors in each layer. A Moreau-Yosida regularization based algorithm is proposed to handle these inequality constraints and a theoretical convergence of the this algorithm is established. Applications of the proposed approach to standard trigonometric functions and more challenging stiff ordinary differential equations arising in chemically reacting ows clearly illustrate the benefits of the proposed approach.
Subjects: Optimization and Control (math.OC)
Cite as: arXiv:2107.01308 [math.OC]
  (or arXiv:2107.01308v1 [math.OC] for this version)

Submission history

From: Thomas Brown [view email]
[v1] Sat, 3 Jul 2021 00:07:14 GMT (2000kb,D)
[v2] Fri, 18 Feb 2022 20:06:36 GMT (2091kb,D)

Link back to: arXiv, form interface, contact.