We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error

Abstract: Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose a novel framework with two contributions: introducing a differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and a meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that we achieve competitive performance with state-of-the-art calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work in this important challenge.
Comments: Shorter version was presented at ICML 2021 Workshop on Uncertainty and Robustness in Deep Learning
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:2106.09613 [cs.LG]
  (or arXiv:2106.09613v2 [cs.LG] for this version)

Submission history

From: Ondrej Bohdal [view email]
[v1] Thu, 17 Jun 2021 15:47:50 GMT (1592kb,D)
[v2] Fri, 15 Jul 2022 16:38:06 GMT (4091kb,D)

Link back to: arXiv, form interface, contact.