Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction
(Submitted on 21 Apr 2020 (v1), last revised 10 Jul 2020 (this version, v3))
Abstract: We develop conformal prediction methods for constructing valid predictive confidence sets in multiclass and multilabel problems without assumptions on the data generating distribution. A challenge here is that typical conformal prediction methods---which give marginal validity (coverage) guarantees---provide uneven coverage, in that they address easy examples at the expense of essentially ignoring difficult examples. By leveraging ideas from quantile regression, we build methods that always guarantee correct coverage but additionally provide (asymptotically optimal) conditional coverage for both multiclass and multilabel prediction problems. To address the potential challenge of exponentially large confidence sets in multilabel prediction, we build tree-structured classifiers that efficiently account for interactions between labels. Our methods can be bolted on top of any classification model---neural network, random forest, boosted tree---to guarantee its validity. We also provide an empirical evaluation, simultaneously providing new validation methods, that suggests the more robust coverage of our confidence sets.
Submission history
From: Suyash Gupta [view email][v1] Tue, 21 Apr 2020 17:45:38 GMT (1251kb,D)
[v2] Fri, 24 Apr 2020 22:53:23 GMT (233kb,D)
[v3] Fri, 10 Jul 2020 18:22:12 GMT (1417kb,D)
Link back to: arXiv, form interface, contact.