Current browse context:
q-bio.NC
Change to browse by:
References & Citations
Quantitative Biology > Neurons and Cognition
Title: The deterministic information bottleneck
(Submitted on 1 Apr 2016 (this version), latest version 19 Dec 2016 (v2))
Abstract: Compression fundamentally involves a decision about what is relevant and what is not. The information bottleneck method (IB) by Tishby, Pereira, and Bialek formalized this notion as an information-theoretic optimization problem and proposed an optimal tradeoff between throwing away as many bits as possible, and selectively keeping those that are most important. Here, we introduce an alternative formulation, the deterministic information bottleneck (DIB), that we argue better captures this notion of compression. As suggested by its name, the solution to the DIB problem is a deterministic encoder, as opposed to the stochastic encoder that is optimal under the IB. After deriving the method, we then compare the IB and DIB on synthetic data, showing that the IB and DIB perform similarly in terms of the IB cost function, but that the DIB significantly outperforms the IB in terms of the DIB cost function. Moreover, the DIB offered a 1-2 order of magnitude speedup over the IB in our experiments. Our derivation of the DIB also suggests a method for continuously interpolating between the soft clustering of the IB and the hard clustering of the DIB.
Submission history
From: David Schwab [view email][v1] Fri, 1 Apr 2016 14:48:31 GMT (95kb)
[v2] Mon, 19 Dec 2016 05:26:11 GMT (42kb,D)
Link back to: arXiv, form interface, contact.