Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
(Submitted on 4 Nov 2021)
Abstract: Modern generative adversarial networks (GANs) predominantly use piecewise linear activation functions in discriminators (or critics), including ReLU and LeakyReLU. Such models learn piecewise linear mappings, where each piece handles a subset of the input space, and the gradients per subset are piecewise constant. Under such a class of discriminator (or critic) functions, we present Gradient Normalization (GraN), a novel input-dependent normalization method, which guarantees a piecewise K-Lipschitz constraint in the input space. In contrast to spectral normalization, GraN does not constrain processing at the individual network layers, and, unlike gradient penalties, strictly enforces a piecewise Lipschitz constraint almost everywhere. Empirically, we demonstrate improved image generation performance across multiple datasets (incl. CIFAR-10/100, STL-10, LSUN bedrooms, and CelebA), GAN loss functions, and metrics. Further, we analyze altering the often untuned Lipschitz constant K in several standard GANs, not only attaining significant performance gains, but also finding connections between K and training dynamics, particularly in low-gradient loss plateaus, with the common Adam optimizer.
Submission history
From: Vineeth S. Bhaskara [view email][v1] Thu, 4 Nov 2021 21:13:02 GMT (5793kb,D)
Link back to: arXiv, form interface, contact.