Neural and Evolutionary Computing
New submissions
[ showing up to 2000 entries per page: fewer  more ]
New submissions for Tue, 11 May 21
 [1] arXiv:2105.03649 [pdf, other]

Title: InHardware Learning of Multilayer Spiking Neural Networks on a Neuromorphic ProcessorComments: 6 pages, 5 figures, accepted for Design Automation Conference (DAC) 2021Subjects: Neural and Evolutionary Computing (cs.NE); Distributed, Parallel, and Cluster Computing (cs.DC); Emerging Technologies (cs.ET)
Although widely used in machine learning, backpropagation cannot directly be applied to SNN training and is not feasible on a neuromorphic processor that emulates biological neuron and synapses. This work presents a spikebased backpropagation algorithm with biological plausible local update rules and adapts it to fit the constraint in a neuromorphic hardware. The algorithm is implemented on Intel Loihi chip enabling low power inhardware supervised online learning of multilayered SNNs for mobile applications. We test this implementation on MNIST, FashionMNIST, CIFAR10 and MSTAR datasets with promising performance and energyefficiency, and demonstrate a possibility of incremental online learning with the implementation.
 [2] arXiv:2105.03680 [pdf, other]

Title: A Crossover That Matches Diverse Parents Together in Evolutionary AlgorithmsAuthors: Maciej ŚwiechowskiComments: Accepted to GECCO 2021Subjects: Neural and Evolutionary Computing (cs.NE); Artificial Intelligence (cs.AI)
Crossover and mutation are the two main operators that lead to new solutions in evolutionary approaches. In this article, a new method of performing the crossover phase is presented. The problem of choice is evolutionary decision tree construction. The method aims at finding such individuals that together complement each other. Hence we say that they are diversely specialized. We propose the way of calculating the socalled complementary fitness. In several empirical experiments, we evaluate the efficacy of the method proposed in four variants and compare it to a fitnessrankbased approach. One variant emerges clearly as the best approach, whereas the remaining ones are below the baseline.
 [3] arXiv:2105.03687 [pdf, other]

Title: Covariance Matrix Adaptation Evolution Strategy Assisted by Principal Component AnalysisAuthors: Yangjie MeiComments: 13 pages, 4 figuresSubjects: Neural and Evolutionary Computing (cs.NE)
Over the past decades, more and more methods gain a giant development due to the development of technology. Evolutionary Algorithms are widely used as a heuristic method. However, the budget of computation increases exponentially when the dimensions increase. In this paper, we will use the dimensionality reduction method Principal component analysis (PCA) to reduce the dimension during the iteration of Covariance Matrix Adaptation Evolution Strategy (CMAES), which is a good Evolutionary Algorithm that is presented as the numeric type and useful for different kinds of problems. We assess the performance of our new methods in terms of convergence rate on multimodal problems from the BlackBox Optimization Benchmarking (BBOB) problem set and we also use the framework COmparing Continuous Optimizers (COCO) to see how the new method going and compare it to the other algorithms.
 [4] arXiv:2105.04097 [pdf]

Title: Examining convolutional feature extraction using Maximum Entropy (ME) and SignaltoNoise Ratio (SNR) for image classificationComments: Conference paper, 6 pages, 1 tableJournalref: Proceedings of the 46th Annual Conference of the IEEE Industrial Electronics Society (IECON2020). IEEE Computer Society Press, pp.471476Subjects: Neural and Evolutionary Computing (cs.NE)
Convolutional Neural Networks (CNNs) specialize in feature extraction rather than function mapping. In doing so they form complex internal hierarchical feature representations, the complexity of which gradually increases with a corresponding increment in neural network depth. In this paper, we examine the feature extraction capabilities of CNNs using Maximum Entropy (ME) and SignaltoNoise Ratio (SNR) to validate the idea that, CNN models should be tailored for a given task and complexity of the input data. SNR and ME measures are used as they can accurately determine in the input dataset, the relative amount of signal information to the random noise and the maximum amount of information respectively. We use two well known benchmarking datasets, MNIST and CIFAR10 to examine the information extraction and abstraction capabilities of CNNs. Through our experiments, we examine convolutional feature extraction and abstraction capabilities in CNNs and show that the classification accuracy or performance of CNNs is greatly dependent on the amount, complexity and quality of the signal information present in the input data. Furthermore, we show the effect of information overflow and underflow on CNN classification accuracies. Our hypothesis is that the feature extraction and abstraction capabilities of convolutional layers are limited and therefore, CNN models should be tailored to the input data by using appropriately sized CNNs based on the SNR and ME measures of the input dataset.
 [5] arXiv:2105.04252 [pdf, other]

Title: An Analysis of Phenotypic Diversity in MultiSolution OptimizationSubjects: Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG)
More and more, optimization methods are used to find diverse solution sets. We compare solution diversity in multiobjective optimization, multimodal optimization, and quality diversity in a simple domain. We show that multiobjective optimization does not always produce much diversity, multimodal optimization produces higher fitness solutions, and quality diversity is not sensitive to genetic neutrality and creates the most diverse set of solutions. An autoencoder is used to discover phenotypic features automatically, producing an even more diverse solution set with quality diversity. Finally, we make recommendations about when to use which approach.
 [6] arXiv:2105.04256 [pdf, other]

Title: Designing Air Flow with Surrogateassisted Phenotypic NichingSubjects: Neural and Evolutionary Computing (cs.NE); Computational Engineering, Finance, and Science (cs.CE); Numerical Analysis (math.NA)
In complex, expensive optimization domains we often narrowly focus on finding high performing solutions, instead of expanding our understanding of the domain itself. But what if we could quickly understand the complex behaviors that can emerge in said domains instead? We introduce surrogateassisted phenotypic niching, a quality diversity algorithm which allows to discover a large, diverse set of behaviors by using computationally expensive phenotypic features. In this work we discover the types of air flow in a 2D fluid dynamics optimization problem. A fast GPUbased fluid dynamics solver is used in conjunction with surrogate models to accurately predict fluid characteristics from the shapes that produce the air flow. We show that these features can be modeled in a datadriven way while sampling to improve performance, rather than explicitly sampling to improve feature models. Our method can reduce the need to run an infeasibly large set of simulations while still being able to design a large diversity of air flows and the shapes that cause them. Discovering diversity of behaviors helps engineers to better understand expensive domains and their solutions.
 [7] arXiv:2105.04311 [pdf]

Title: Overcoming Complexity Catastrophe: An Algorithm for Beneficial FarReaching Adaptation under High ComplexityComments: 10 pages, 5 FiguresSubjects: Neural and Evolutionary Computing (cs.NE); Adaptation and SelfOrganizing Systems (nlin.AO)
In his seminal work with NK algorithms, Kauffman noted that fitness outcomes from algorithms navigating an NK landscape show a sharp decline at high complexity arising from pervasive interdependence among problem dimensions. This phenomenon  where complexity effects dominate (Darwinian) adaptation efforts  is called complexity catastrophe. We present an algorithm  incremental change taking turns (ICTT)  that finds distant configurations having fitness superior to that reported in extant research, under high complexity. Thus, complexity catastrophe is not inevitable: a series of incremental changes can lead to excellent outcomes.
Crosslists for Tue, 11 May 21
 [8] arXiv:2105.03703 (crosslist from cs.LG) [pdf, other]

Title: Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training DynamicsComments: ICML 2021Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Probability (math.PR)
Yang (2020a) recently showed that the Neural Tangent Kernel (NTK) at initialization has an infinitewidth limit for a large class of architectures including modern staples such as ResNet and Transformers. However, their analysis does not apply to training. Here, we show the same neural networks (in the socalled NTK parametrization) during training follow a kernel gradient descent dynamics in function space, where the kernel is the infinitewidth NTK. This completes the proof of the *architectural universality* of NTK behavior. To achieve this result, we apply the Tensor Programs technique: Write the entire SGD dynamics inside a Tensor Program and analyze it via the Master Theorem. To facilitate this proof, we develop a graphical notation for Tensor Programs.
 [9] arXiv:2105.04045 (crosslist from cs.AI) [pdf, other]

Title: Swarm Differential Privacy for Purpose Driven DataInformationKnowledgeWisdom ArchitectureAuthors: Yingbo Li, Yucong Duan, Zakaria Maama, Haoyang Che, AnamariaBeatrice Spulber, Stelios FuentesSubjects: Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
Privacy protection has recently attracted the attention of both academics and industries. Society protects individual data privacy through complex legal frameworks. This has become a topic of interest with the increasing applications of data science and artificial intelligence that have created a higher demand to the ubiquitous application of the data. The privacy protection of the broad DataInformationKnowledgeWisdom (DIKW) landscape, the next generation of information organization, has not been in the limelight. Next, we will explore DIKW architecture through the applications of popular swarm intelligence and differential privacy. As differential privacy proved to be an effective data privacy approach, we will look at it from a DIKW domain perspective. Swarm Intelligence could effectively optimize and reduce the number of items in DIKW used in differential privacy, this way accelerating both the effectiveness and the efficiency of differential privacy for crossing multiple modals of conceptual DIKW. The proposed approach is proved through the application of personalized data that is based on the opensourse IRIS dataset. This experiment demonstrates the efficiency of Swarm Intelligence in reducing computing complexity.
 [10] arXiv:2105.04128 (crosslist from cs.CV) [pdf]

Title: Examining and Mitigating Kernel Saturation in Convolutional Neural Networks using Negative ImagesComments: Conference paper, 6 pages, 3 figures, 1 tableJournalref: Proceedings of the 46th Annual Conference of the IEEE Industrial Electronics Society (IECON2020). IEEE Computer Society Press, pp.465470Subjects: Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
Neural saturation in Deep Neural Networks (DNNs) has been studied extensively, but remains relatively unexplored in Convolutional Neural Networks (CNNs). Understanding and alleviating the effects of convolutional kernel saturation is critical for enhancing CNN models classification accuracies. In this paper, we analyze the effect of convolutional kernel saturation in CNNs and propose a simple data augmentation technique to mitigate saturation and increase classification accuracy, by supplementing negative images to the training dataset. We hypothesize that greater semantic feature information can be extracted using negative images since they have the same structural information as standard images but differ in their data representations. Varied data representations decrease the probability of kernel saturation and thus increase the effectiveness of kernel weight updates. The two datasets selected to evaluate our hypothesis were CIFAR 10 and STL10 as they have similar image classes but differ in image resolutions thus making for a better understanding of the saturation phenomenon. MNIST dataset was used to highlight the ineffectiveness of the technique for linearly separable data. The ResNet CNN architecture was chosen since the skip connections in the network ensure the most important features contributing the most to classification accuracy are retained. Our results show that CNNs are indeed susceptible to convolutional kernel saturation and that supplementing negative images to the training dataset can offer a statistically significant increase in classification accuracies when compared against models trained on the original datasets. Our results present accuracy increases of 6.98% and 3.16% on the STL10 and CIFAR10 datasets respectively.
 [11] arXiv:2105.04247 (crosslist from cs.LG) [pdf, other]

Title: Expressivity of Parameterized and Datadriven Representations in Quality Diversity SearchComments: For code for reproducing experiments, see this https URLSubjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
We consider multisolution optimization and generative models for the generation of diverse artifacts and the discovery of novel solutions. In cases where the domain's factors of variation are unknown or too complex to encode manually, generative models can provide a learned latent space to approximate these factors. When used as a search space, however, the range and diversity of possible outputs are limited to the expressivity and generative capabilities of the learned model. We compare the output diversity of a quality diversity evolutionary search performed in two different search spaces: 1) a predefined parameterized space and 2) the latent space of a variational autoencoder model. We find that the search on an explicit parametric encoding creates more diverse artifact sets than searching the latent space. A learned model is better at interpolating between known data points than at extrapolating or expanding towards unseen examples. We recommend using a generative model's latent space primarily to measure similarity between artifacts rather than for search and generation. Whenever a parametric encoding is obtainable, it should be preferred over a learned representation as it produces a higher diversity of solutions.
 [12] arXiv:2105.04480 (crosslist from stat.ME) [pdf, other]

Title: Is there Anisotropy in Structural Bias?Subjects: Methodology (stat.ME); Neural and Evolutionary Computing (cs.NE)
Structural Bias (SB) is an important type of algorithmic deficiency within iterative optimisation heuristics. However, methods for detecting structural bias have not yet fully matured, and recent studies have uncovered many interesting questions. One of these is the question of how structural bias can be related to anisotropy. Intuitively, an algorithm that is not isotropic would be considered structurally biased. However, there have been cases where algorithms appear to only show SB in some dimensions. As such, we investigate whether these algorithms actually exhibit anisotropy, and how this impacts the detection of SB. We find that anisotropy is very rare, and even in cases where it is present, there are clear tests for SB which do not rely on any assumptions of isotropy, so we can safely expand the suite of SB tests to encompass these kinds of deficiencies not found by the original tests.
We propose several additional testing procedures for SB detection and aim to motivate further research into the creation of a robust portfolio of tests. This is crucial since no single test will be able to work effectively with all types of SB we identify.
Replacements for Tue, 11 May 21
 [13] arXiv:1910.12478 (replaced) [pdf, other]

Title: Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian ProcessesAuthors: Greg YangComments: Appearing in NeurIPS 2019; 10 pages of main text; 12 figures, 11 programs; 73 pages totalSubjects: Neural and Evolutionary Computing (cs.NE); Disordered Systems and Neural Networks (condmat.disnn); Machine Learning (cs.LG); Mathematical Physics (mathph)
 [14] arXiv:2005.05744 (replaced) [pdf]

Title: Deep Learning: Our Miraculous Year 19901991Authors: Juergen SchmidhuberComments: 26 pages, 236 references, based on work of 4 Oct 2019Subjects: Neural and Evolutionary Computing (cs.NE)
 [15] arXiv:2009.10685 (replaced) [pdf, other]

Title: Tensor Programs III: Neural Matrix LawsAuthors: Greg YangSubjects: Neural and Evolutionary Computing (cs.NE); Probability (math.PR)
 [16] arXiv:2104.10851 (replaced) [pdf, other]

Title: Continuous Learning and Adaptation with Membrane Potential and Activation Threshold HomeostasisAuthors: Alexander HadjiivanovComments: 19 pagesSubjects: Neural and Evolutionary Computing (cs.NE); Computer Vision and Pattern Recognition (cs.CV)
 [17] arXiv:1901.07066 (replaced) [src]

Title: On Compression of Unsupervised Neural Nets by Pruning Weak ConnectionsComments: This paper needs to be further revisedSubjects: Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
 [18] arXiv:2002.08809 (replaced) [pdf, other]

Title: DDPNOpt: Differential Dynamic Programming Neural OptimizerComments: Accepted in International Conference on Learning Representations (ICLR) 2021 as SpotlightSubjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Optimization and Control (math.OC)
[ showing up to 2000 entries per page: fewer  more ]
Disable MathJax (What is MathJax?)
Links to: arXiv, form interface, find, cs, recent, 2105, contact, help (Access key information)