References & Citations
Computer Science > Artificial Intelligence
Title: A critical analysis of metrics used for measuring progress in artificial intelligence
(Submitted on 6 Aug 2020 (v1), last revised 8 Nov 2021 (this version, v2))
Abstract: Comparing model performances on benchmark datasets is an integral part of measuring and driving progress in artificial intelligence. A model's performance on a benchmark dataset is commonly assessed based on a single or a small set of performance metrics. While this enables quick comparisons, it may entail the risk of inadequately reflecting model performance if the metric does not sufficiently cover all performance characteristics. It is unknown to what extent this might impact benchmarking efforts.
To address this question, we analysed the current landscape of performance metrics based on data covering 3867 machine learning model performance results from the open repository 'Papers with Code'. Our results suggest that the large majority of metrics currently used have properties that may result in an inadequate reflection of a models' performance. While alternative metrics that address problematic properties have been proposed, they are currently rarely used.
Furthermore, we describe ambiguities in reported metrics, which may lead to difficulties in interpreting and comparing model performances.
Submission history
From: Matthias Samwald [view email][v1] Thu, 6 Aug 2020 11:14:37 GMT (624kb)
[v2] Mon, 8 Nov 2021 14:38:58 GMT (503kb)
Link back to: arXiv, form interface, contact.