We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IT

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Theory

Title: Estimating the Unique Information of Continuous Variables in Recurrent Networks

Abstract: The integration and transfer of information from multiple sources to multiple targets is a core motive of neural systems. The emerging field of partial information decomposition (PID) provides a novel information-theoretic lens into these mechanisms by identifying synergistic, redundant, and unique contributions to the mutual information between one and several variables. While many works have studied aspects of PID for Gaussian and discrete distributions, the case of general continuous distributions is still uncharted territory. In this work we present a method for estimating the unique information in continuous distributions, for the case of one versus two variables. Our method solves the associated optimization problem over the space of distributions with fixed bivariate marginals by combining copula decompositions and techniques developed to optimize variational autoencoders. We obtain excellent agreement with known analytic results for Gaussians, and illustrate the power of our new approach in several brain-inspired neural models. Our method is capable of recovering the effective connectivity of a chaotic network of rate neurons, and uncovers a complex trade-off between redundancy, synergy and unique information in recurrent networks trained to solve a generalized XOR task.
Subjects: Information Theory (cs.IT); Computation (stat.CO)
Cite as: arXiv:2102.00218 [cs.IT]
  (or arXiv:2102.00218v4 [cs.IT] for this version)

Submission history

From: Ari Pakman [view email]
[v1] Sat, 30 Jan 2021 12:34:42 GMT (120kb,D)
[v2] Wed, 3 Feb 2021 19:41:17 GMT (120kb,D)
[v3] Tue, 22 Jun 2021 09:34:39 GMT (3286kb,D)
[v4] Wed, 23 Jun 2021 07:33:21 GMT (3286kb,D)
[v5] Wed, 27 Oct 2021 02:16:04 GMT (8195kb,D)

Link back to: arXiv, form interface, contact.