Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Distinction Maximization Loss: Efficiently Improving Classification Accuracy, Uncertainty Estimation, and Out-of-Distribution Detection Simply Replacing the Loss and Calibrating
(Submitted on 12 May 2022 (this version), latest version 5 Aug 2022 (v5))
Abstract: Building robust deterministic deep neural networks is still a challenge. On the one hand, some approaches improve out-of-distribution detection at the cost of reducing classification accuracy in some situations. On the other hand, some methods simultaneously increase classification accuracy, out-of-distribution detection, and uncertainty estimation, but reduce inference efficiency, in addition to training the same model many times to tune hyperparameters. In this paper, we propose training deterministic deep neural networks using our DisMax loss, which works as a drop-in replacement for the commonly used SoftMax loss (i.e., the combination of the linear output layer, the SoftMax activation, and the cross-entropy loss). Starting from the IsoMax+ loss, we created novel logits that are based on the distance to all prototypes rather than just the one associated with the correct class. We also propose a novel way to augment images to construct what we call fractional probability regularization. Moreover, we propose a new score to perform out-of-distribution detection and a fast way to calibrate the network after training. Our experiments show that DisMax usually outperforms all current approaches simultaneously in classification accuracy, uncertainty estimation, inference efficiency, and out-of-distribution detection, avoiding hyperparameter tuning and repetitive model training. The code to replace the SoftMax loss with the DisMax loss and reproduce the results in this paper is available at this https URL
Submission history
From: David Macêdo [view email][v1] Thu, 12 May 2022 04:37:35 GMT (2039kb,D)
[v2] Thu, 19 May 2022 12:04:27 GMT (2039kb,D)
[v3] Thu, 28 Jul 2022 12:53:20 GMT (2044kb,D)
[v4] Mon, 1 Aug 2022 05:04:23 GMT (2044kb,D)
[v5] Fri, 5 Aug 2022 18:26:34 GMT (2043kb,D)
Link back to: arXiv, form interface, contact.