Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness
(Submitted on 25 May 2019 (v1), last revised 20 Feb 2020 (this version, v3))
Abstract: Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.
Submission history
From: Tianyu Pang [view email][v1] Sat, 25 May 2019 16:11:14 GMT (2179kb,D)
[v2] Wed, 25 Sep 2019 14:35:29 GMT (2279kb,D)
[v3] Thu, 20 Feb 2020 08:50:47 GMT (2291kb,D)
Link back to: arXiv, form interface, contact.