We gratefully acknowledge support from
the Simons Foundation and member institutions.

Neural and Evolutionary Computing

New submissions

[ total of 12 entries: 1-12 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Wed, 23 Sep 20

[1]  arXiv:2009.10199 [pdf, other]
Title: Evolutionary Architecture Search for Graph Neural Networks
Comments: 11 pages
Subjects: Neural and Evolutionary Computing (cs.NE)

Automated machine learning (AutoML) has seen a resurgence in interest with the boom of deep learning over the past decade. In particular, Neural Architecture Search (NAS) has seen significant attention throughout the AutoML research community, and has pushed forward the state-of-the-art in a number of neural models to address grid-like data such as texts and images. However, very litter work has been done about Graph Neural Networks (GNN) learning on unstructured network data. Given the huge number of choices and combinations of components such as aggregator and activation function, determining the suitable GNN structure for a specific problem normally necessitates tremendous expert knowledge and laborious trails. In addition, the slight variation of hyper parameters such as learning rate and dropout rate could dramatically hurt the learning capacity of GNN. In this paper, we propose a novel AutoML framework through the evolution of individual models in a large GNN architecture space involving both neural structures and learning parameters. Instead of optimizing only the model structures with fixed parameter settings as existing work, an alternating evolution process is performed between GNN structures and learning parameters to dynamically find the best fit of each other. To the best of our knowledge, this is the first work to introduce and evaluate evolutionary architecture search for GNN models. Experiments and validations demonstrate that evolutionary NAS is capable of matching existing state-of-the-art reinforcement learning approaches for both the semi-supervised transductive and inductive node representation learning and classification.

[2]  arXiv:2009.10460 [pdf, other]
Title: Multi-threaded Memory Efficient Crossover in C++ for Generational Genetic Programming
Authors: W. B. Langdon
Comments: 11 pages, 3 figures
Subjects: Neural and Evolutionary Computing (cs.NE)

C++ code snippets from a multi-core parallel memory-efficient crossover for genetic programming are given. They may be adapted for separate generation evolutionary algorithms where large chromosomes or small RAM require no more than M + (2 times nthreads) simultaneously active individuals.

[3]  arXiv:2009.10520 [pdf, other]
Title: Complex Vehicle Routing with Memory Augmented Neural Networks
Comments: Presented at 3RD IEEE International Conference on Industrial Cyber-Physical Systems, 2020
Subjects: Neural and Evolutionary Computing (cs.NE)

Complex real-life routing challenges can be modeled as variations of well-known combinatorial optimization problems. These routing problems have long been studied and are difficult to solve at scale. The particular setting may also make exact formulation difficult. Deep Learning offers an increasingly attractive alternative to traditional solutions, which mainly revolve around the use of various heuristics. Deep Learning may provide solutions which are less time-consuming and of higher quality at large scales, as it generally does not need to generate solutions in an iterative manner, and Deep Learning models have shown a surprising capacity for solving complex tasks in recent years. Here we consider a particular variation of the Capacitated Vehicle Routing (CVRP) problem and investigate the use of Deep Learning models with explicit memory components. Such memory components may help in gaining insight into the model's decisions as the memory and operations on it can be directly inspected at any time, and may assist in scaling the method to such a size that it becomes viable for industry settings.

[4]  arXiv:2009.10685 [pdf, other]
Title: Tensor Programs III: Neural Matrix Laws
Authors: Greg Yang
Subjects: Neural and Evolutionary Computing (cs.NE); Probability (math.PR)

In a neural network (NN), \emph{weight matrices} linearly transform inputs into \emph{preactivations} that are then transformed nonlinearly into \emph{activations}. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become \emph{independent} from the weights as the NN's widths tend to infinity, in the sense of \emph{asymptotic freeness} in random matrix theory. We call this the \emph{Free Independence Principle (FIP)}, which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of \emph{gradient independence assumption} used for calculating the \emph{Neural Tangent Kernel} of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

Cross-lists for Wed, 23 Sep 20

[5]  arXiv:2009.10214 (cross-list from cs.LG) [pdf, other]
Title: DISPATCH: Design Space Exploration of Cyber-Physical Systems
Comments: 14 pages
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Systems and Control (eess.SY)

Design of Cyber-physical systems (CPSs) is a challenging task that involves searching over a large search space of various CPS configurations and possible values of components composing the system. Hence, there is a need for sample-efficient CPS design space exploration to select the system architecture and component values that meet the target system requirements. We address this challenge by formulating CPS design as a multi-objective optimization problem and propose DISPATCH, a two-step methodology for sample-efficient search over the design space. First, we use a genetic algorithm to search over discrete choices of system component values for architecture search and component selection or only component selection and terminate the algorithm even before meeting the system requirements, thus yielding a coarse design. In the second step, we use an inverse design to search over a continuous space to fine-tune the component values and meet the diverse set of system requirements. We use a neural network as a surrogate function for the inverse design of the system. The neural network, converted into a mixed-integer linear program, is used for active learning to sample component values efficiently in a continuous search space. We illustrate the efficacy of DISPATCH on electrical circuit benchmarks: two-stage and three-stage transimpedence amplifiers. Simulation results show that the proposed methodology improves sample efficiency by 5-14x compared to a prior synthesis method that relies on reinforcement learning. It also synthesizes circuits with the best performance (highest bandwidth/lowest area) compared to designs synthesized using reinforcement learning, Bayesian optimization, or humans.

Replacements for Wed, 23 Sep 20

[6]  arXiv:1903.10410 (replaced) [pdf, ps, other]
Title: A Conceptual Bio-Inspired Framework for the Evolution of Artificial General Intelligence
Comments: 7 pages, 2 figures, accepted to "The 3rd Special Session on Biologically Inspired Parallel and Distributed Computing, Algorithms and Solutions" (BICAS 2020)
Subjects: Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI)
[7]  arXiv:1605.05359 (replaced) [pdf, other]
Title: Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering
Comments: Revised version of ICML 16 Abstraction in Reinforcement Learning workshop paper
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
[8]  arXiv:1605.05365 (replaced) [pdf, other]
Title: Dynamic Frame skip Deep Q Network
Comments: IJCAI 2016 Workshop on Deep Reinforcement Learning: Frontiers and Challenges; 6 pages, 8 figures
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
[9]  arXiv:1702.06054 (replaced) [pdf, other]
Title: Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning
Comments: 24 pages
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
[10]  arXiv:1912.08881 (replaced) [pdf, other]
Title: Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
Comments: 29 pages, 11 figures, 6 tables
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
[11]  arXiv:2008.04245 (replaced) [pdf, other]
Title: TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices
Comments: 9 pages
Subjects: Audio and Speech Processing (eess.AS); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
[12]  arXiv:2009.08841 (replaced) [pdf, other]
Title: On the spatiotemporal behavior in biology-mimicking computing systems
Comments: 33 pages, 6 figures
Subjects: Emerging Technologies (cs.ET); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
[ total of 12 entries: 1-12 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, cs, recent, 2009, contact, help  (Access key information)