References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification
(Submitted on 21 Nov 2022 (v1), last revised 25 Apr 2023 (this version, v2))
Abstract: Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into human-readable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches.
Submission history
From: Yue Yang [view email][v1] Mon, 21 Nov 2022 03:05:02 GMT (35166kb,D)
[v2] Tue, 25 Apr 2023 22:06:42 GMT (17649kb,D)
Link back to: arXiv, form interface, contact.