We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:


References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Neural and Evolutionary Computing

Title: Alternate Loss Functions Can Improve the Performance of Artificial Neural Networks

Abstract: All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of new loss functions that are more liberal or strict compared to the popular Cross-entropy loss in penalizing incorrect outputs. Eight new loss functions are proposed and a comparison of performance with different loss functions is presented. The new loss functions presented in this paper are shown to outperform Cross-entropy loss on computer vision and NLP benchmarks.
Subjects: Neural and Evolutionary Computing (cs.NE)
MSC classes: 68
ACM classes: I.2
Cite as: arXiv:2303.09935 [cs.NE]
  (or arXiv:2303.09935v1 [cs.NE] for this version)

Submission history

From: Mathew Mithra Noel [view email]
[v1] Fri, 17 Mar 2023 12:52:06 GMT (309kb,D)

Link back to: arXiv, form interface, contact.