References & Citations
Computer Science > Information Theory
Title: An Achievable and Analytic Solution to Information Bottleneck for Gaussian Mixtures
(Submitted on 7 Feb 2023 (v1), last revised 31 May 2023 (this version, v2))
Abstract: The information bottleneck (IB) approach, initially introduced by Tishby et al. to assess the "compression--relevance" tradeoff for a remote source coding problem in communications, gains popularity recently in its application to modern machine learning (ML). Despite its seemingly simple form, the solution to IB problem remains largely unknown, and can only be assessed numerically even in the simple setting of Gaussian mixture model that is of fundamental significance in ML. In this paper, by combining ideas of hard quantization and soft nonlinear transformation, we derive closed-form achievable bounds for the IB problem under the above setting. The derived bounds establish surprisingly close behavior to the (numerically) optimal IB solution obtained by Blahut--Arimoto (BA) algorithm, on both synthetic and real-world (so non-Gaussian mixture) datasets, suggesting possibly wider applicability of our results.
Submission history
From: Yi Song [view email][v1] Tue, 7 Feb 2023 15:56:10 GMT (625kb,D)
[v2] Wed, 31 May 2023 13:58:02 GMT (848kb,D)
Link back to: arXiv, form interface, contact.