We gratefully acknowledge support from
the Simons Foundation and member institutions.

Neural and Evolutionary Computing

New submissions

[ total of 18 entries: 1-18 ]
[ showing up to 2000 entries per page: fewer | more ]

New submissions for Tue, 11 May 21

[1]  arXiv:2105.03649 [pdf, other]
Title: In-Hardware Learning of Multilayer Spiking Neural Networks on a Neuromorphic Processor
Comments: 6 pages, 5 figures, accepted for Design Automation Conference (DAC) 2021
Subjects: Neural and Evolutionary Computing (cs.NE); Distributed, Parallel, and Cluster Computing (cs.DC); Emerging Technologies (cs.ET)

Although widely used in machine learning, backpropagation cannot directly be applied to SNN training and is not feasible on a neuromorphic processor that emulates biological neuron and synapses. This work presents a spike-based backpropagation algorithm with biological plausible local update rules and adapts it to fit the constraint in a neuromorphic hardware. The algorithm is implemented on Intel Loihi chip enabling low power in-hardware supervised online learning of multilayered SNNs for mobile applications. We test this implementation on MNIST, Fashion-MNIST, CIFAR-10 and MSTAR datasets with promising performance and energy-efficiency, and demonstrate a possibility of incremental online learning with the implementation.

[2]  arXiv:2105.03680 [pdf, other]
Title: A Crossover That Matches Diverse Parents Together in Evolutionary Algorithms
Comments: Accepted to GECCO 2021
Subjects: Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI)

Crossover and mutation are the two main operators that lead to new solutions in evolutionary approaches. In this article, a new method of performing the crossover phase is presented. The problem of choice is evolutionary decision tree construction. The method aims at finding such individuals that together complement each other. Hence we say that they are diversely specialized. We propose the way of calculating the so-called complementary fitness. In several empirical experiments, we evaluate the efficacy of the method proposed in four variants and compare it to a fitness-rank-based approach. One variant emerges clearly as the best approach, whereas the remaining ones are below the baseline.

[3]  arXiv:2105.03687 [pdf, other]
Title: Covariance Matrix Adaptation Evolution Strategy Assisted by Principal Component Analysis
Authors: Yangjie Mei
Comments: 13 pages, 4 figures
Subjects: Neural and Evolutionary Computing (cs.NE)

Over the past decades, more and more methods gain a giant development due to the development of technology. Evolutionary Algorithms are widely used as a heuristic method. However, the budget of computation increases exponentially when the dimensions increase. In this paper, we will use the dimensionality reduction method Principal component analysis (PCA) to reduce the dimension during the iteration of Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is a good Evolutionary Algorithm that is presented as the numeric type and useful for different kinds of problems. We assess the performance of our new methods in terms of convergence rate on multi-modal problems from the Black-Box Optimization Benchmarking (BBOB) problem set and we also use the framework COmparing Continuous Optimizers (COCO) to see how the new method going and compare it to the other algorithms.

[4]  arXiv:2105.04097 [pdf]
Title: Examining convolutional feature extraction using Maximum Entropy (ME) and Signal-to-Noise Ratio (SNR) for image classification
Comments: Conference paper, 6 pages, 1 table
Journal-ref: Proceedings of the 46th Annual Conference of the IEEE Industrial Electronics Society (IECON2020). IEEE Computer Society Press, pp.471-476
Subjects: Neural and Evolutionary Computing (cs.NE)

Convolutional Neural Networks (CNNs) specialize in feature extraction rather than function mapping. In doing so they form complex internal hierarchical feature representations, the complexity of which gradually increases with a corresponding increment in neural network depth. In this paper, we examine the feature extraction capabilities of CNNs using Maximum Entropy (ME) and Signal-to-Noise Ratio (SNR) to validate the idea that, CNN models should be tailored for a given task and complexity of the input data. SNR and ME measures are used as they can accurately determine in the input dataset, the relative amount of signal information to the random noise and the maximum amount of information respectively. We use two well known benchmarking datasets, MNIST and CIFAR-10 to examine the information extraction and abstraction capabilities of CNNs. Through our experiments, we examine convolutional feature extraction and abstraction capabilities in CNNs and show that the classification accuracy or performance of CNNs is greatly dependent on the amount, complexity and quality of the signal information present in the input data. Furthermore, we show the effect of information overflow and underflow on CNN classification accuracies. Our hypothesis is that the feature extraction and abstraction capabilities of convolutional layers are limited and therefore, CNN models should be tailored to the input data by using appropriately sized CNNs based on the SNR and ME measures of the input dataset.

[5]  arXiv:2105.04252 [pdf, other]
Title: An Analysis of Phenotypic Diversity in Multi-Solution Optimization
Subjects: Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG)

More and more, optimization methods are used to find diverse solution sets. We compare solution diversity in multi-objective optimization, multimodal optimization, and quality diversity in a simple domain. We show that multiobjective optimization does not always produce much diversity, multimodal optimization produces higher fitness solutions, and quality diversity is not sensitive to genetic neutrality and creates the most diverse set of solutions. An autoencoder is used to discover phenotypic features automatically, producing an even more diverse solution set with quality diversity. Finally, we make recommendations about when to use which approach.

[6]  arXiv:2105.04256 [pdf, other]
Title: Designing Air Flow with Surrogate-assisted Phenotypic Niching
Subjects: Neural and Evolutionary Computing (cs.NE); Computational Engineering, Finance, and Science (cs.CE); Numerical Analysis (math.NA)

In complex, expensive optimization domains we often narrowly focus on finding high performing solutions, instead of expanding our understanding of the domain itself. But what if we could quickly understand the complex behaviors that can emerge in said domains instead? We introduce surrogate-assisted phenotypic niching, a quality diversity algorithm which allows to discover a large, diverse set of behaviors by using computationally expensive phenotypic features. In this work we discover the types of air flow in a 2D fluid dynamics optimization problem. A fast GPU-based fluid dynamics solver is used in conjunction with surrogate models to accurately predict fluid characteristics from the shapes that produce the air flow. We show that these features can be modeled in a data-driven way while sampling to improve performance, rather than explicitly sampling to improve feature models. Our method can reduce the need to run an infeasibly large set of simulations while still being able to design a large diversity of air flows and the shapes that cause them. Discovering diversity of behaviors helps engineers to better understand expensive domains and their solutions.

[7]  arXiv:2105.04311 [pdf]
Title: Overcoming Complexity Catastrophe: An Algorithm for Beneficial Far-Reaching Adaptation under High Complexity
Comments: 10 pages, 5 Figures
Subjects: Neural and Evolutionary Computing (cs.NE); Adaptation and Self-Organizing Systems (nlin.AO)

In his seminal work with NK algorithms, Kauffman noted that fitness outcomes from algorithms navigating an NK landscape show a sharp decline at high complexity arising from pervasive interdependence among problem dimensions. This phenomenon - where complexity effects dominate (Darwinian) adaptation efforts - is called complexity catastrophe. We present an algorithm - incremental change taking turns (ICTT) - that finds distant configurations having fitness superior to that reported in extant research, under high complexity. Thus, complexity catastrophe is not inevitable: a series of incremental changes can lead to excellent outcomes.

Cross-lists for Tue, 11 May 21

[8]  arXiv:2105.03703 (cross-list from cs.LG) [pdf, other]
Title: Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
Comments: ICML 2021
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Probability (math.PR)

Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinite-width limit for a large class of architectures including modern staples such as ResNet and Transformers. However, their analysis does not apply to training. Here, we show the same neural networks (in the so-called NTK parametrization) during training follow a kernel gradient descent dynamics in function space, where the kernel is the infinite-width NTK. This completes the proof of the *architectural universality* of NTK behavior. To achieve this result, we apply the Tensor Programs technique: Write the entire SGD dynamics inside a Tensor Program and analyze it via the Master Theorem. To facilitate this proof, we develop a graphical notation for Tensor Programs.

[9]  arXiv:2105.04045 (cross-list from cs.AI) [pdf, other]
Title: Swarm Differential Privacy for Purpose Driven Data-Information-Knowledge-Wisdom Architecture
Subjects: Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)

Privacy protection has recently attracted the attention of both academics and industries. Society protects individual data privacy through complex legal frameworks. This has become a topic of interest with the increasing applications of data science and artificial intelligence that have created a higher demand to the ubiquitous application of the data. The privacy protection of the broad Data-InformationKnowledge-Wisdom (DIKW) landscape, the next generation of information organization, has not been in the limelight. Next, we will explore DIKW architecture through the applications of popular swarm intelligence and differential privacy. As differential privacy proved to be an effective data privacy approach, we will look at it from a DIKW domain perspective. Swarm Intelligence could effectively optimize and reduce the number of items in DIKW used in differential privacy, this way accelerating both the effectiveness and the efficiency of differential privacy for crossing multiple modals of conceptual DIKW. The proposed approach is proved through the application of personalized data that is based on the open-sourse IRIS dataset. This experiment demonstrates the efficiency of Swarm Intelligence in reducing computing complexity.

[10]  arXiv:2105.04128 (cross-list from cs.CV) [pdf]
Title: Examining and Mitigating Kernel Saturation in Convolutional Neural Networks using Negative Images
Comments: Conference paper, 6 pages, 3 figures, 1 table
Journal-ref: Proceedings of the 46th Annual Conference of the IEEE Industrial Electronics Society (IECON2020). IEEE Computer Society Press, pp.465-470
Subjects: Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)

Neural saturation in Deep Neural Networks (DNNs) has been studied extensively, but remains relatively unexplored in Convolutional Neural Networks (CNNs). Understanding and alleviating the effects of convolutional kernel saturation is critical for enhancing CNN models classification accuracies. In this paper, we analyze the effect of convolutional kernel saturation in CNNs and propose a simple data augmentation technique to mitigate saturation and increase classification accuracy, by supplementing negative images to the training dataset. We hypothesize that greater semantic feature information can be extracted using negative images since they have the same structural information as standard images but differ in their data representations. Varied data representations decrease the probability of kernel saturation and thus increase the effectiveness of kernel weight updates. The two datasets selected to evaluate our hypothesis were CIFAR- 10 and STL-10 as they have similar image classes but differ in image resolutions thus making for a better understanding of the saturation phenomenon. MNIST dataset was used to highlight the ineffectiveness of the technique for linearly separable data. The ResNet CNN architecture was chosen since the skip connections in the network ensure the most important features contributing the most to classification accuracy are retained. Our results show that CNNs are indeed susceptible to convolutional kernel saturation and that supplementing negative images to the training dataset can offer a statistically significant increase in classification accuracies when compared against models trained on the original datasets. Our results present accuracy increases of 6.98% and 3.16% on the STL-10 and CIFAR-10 datasets respectively.

[11]  arXiv:2105.04247 (cross-list from cs.LG) [pdf, other]
Title: Expressivity of Parameterized and Data-driven Representations in Quality Diversity Search
Comments: For code for reproducing experiments, see this https URL
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)

We consider multi-solution optimization and generative models for the generation of diverse artifacts and the discovery of novel solutions. In cases where the domain's factors of variation are unknown or too complex to encode manually, generative models can provide a learned latent space to approximate these factors. When used as a search space, however, the range and diversity of possible outputs are limited to the expressivity and generative capabilities of the learned model. We compare the output diversity of a quality diversity evolutionary search performed in two different search spaces: 1) a predefined parameterized space and 2) the latent space of a variational autoencoder model. We find that the search on an explicit parametric encoding creates more diverse artifact sets than searching the latent space. A learned model is better at interpolating between known data points than at extrapolating or expanding towards unseen examples. We recommend using a generative model's latent space primarily to measure similarity between artifacts rather than for search and generation. Whenever a parametric encoding is obtainable, it should be preferred over a learned representation as it produces a higher diversity of solutions.

[12]  arXiv:2105.04480 (cross-list from stat.ME) [pdf, other]
Title: Is there Anisotropy in Structural Bias?
Subjects: Methodology (stat.ME); Neural and Evolutionary Computing (cs.NE)

Structural Bias (SB) is an important type of algorithmic deficiency within iterative optimisation heuristics. However, methods for detecting structural bias have not yet fully matured, and recent studies have uncovered many interesting questions. One of these is the question of how structural bias can be related to anisotropy. Intuitively, an algorithm that is not isotropic would be considered structurally biased. However, there have been cases where algorithms appear to only show SB in some dimensions. As such, we investigate whether these algorithms actually exhibit anisotropy, and how this impacts the detection of SB. We find that anisotropy is very rare, and even in cases where it is present, there are clear tests for SB which do not rely on any assumptions of isotropy, so we can safely expand the suite of SB tests to encompass these kinds of deficiencies not found by the original tests.
We propose several additional testing procedures for SB detection and aim to motivate further research into the creation of a robust portfolio of tests. This is crucial since no single test will be able to work effectively with all types of SB we identify.

Replacements for Tue, 11 May 21

[13]  arXiv:1910.12478 (replaced) [pdf, other]
Title: Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
Authors: Greg Yang
Comments: Appearing in NeurIPS 2019; 10 pages of main text; 12 figures, 11 programs; 73 pages total
Subjects: Neural and Evolutionary Computing (cs.NE); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (cs.LG); Mathematical Physics (math-ph)
[14]  arXiv:2005.05744 (replaced) [pdf]
Title: Deep Learning: Our Miraculous Year 1990-1991
Comments: 26 pages, 236 references, based on work of 4 Oct 2019
Subjects: Neural and Evolutionary Computing (cs.NE)
[15]  arXiv:2009.10685 (replaced) [pdf, other]
Title: Tensor Programs III: Neural Matrix Laws
Authors: Greg Yang
Subjects: Neural and Evolutionary Computing (cs.NE); Probability (math.PR)
[16]  arXiv:2104.10851 (replaced) [pdf, other]
Title: Continuous Learning and Adaptation with Membrane Potential and Activation Threshold Homeostasis
Comments: 19 pages
Subjects: Neural and Evolutionary Computing (cs.NE); Computer Vision and Pattern Recognition (cs.CV)
[17]  arXiv:1901.07066 (replaced) [src]
Title: On Compression of Unsupervised Neural Nets by Pruning Weak Connections
Comments: This paper needs to be further revised
Subjects: Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
[18]  arXiv:2002.08809 (replaced) [pdf, other]
Title: DDPNOpt: Differential Dynamic Programming Neural Optimizer
Comments: Accepted in International Conference on Learning Representations (ICLR) 2021 as Spotlight
Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC)
[ total of 18 entries: 1-18 ]
[ showing up to 2000 entries per page: fewer | more ]

Disable MathJax (What is MathJax?)

Links to: arXiv, form interface, find, cs, recent, 2105, contact, help  (Access key information)