We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IT

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Theory

Title: Collaborative Representation Learning

Abstract: This paper investigates an information-theoretic approach to the problem of collaborative representation learning: how to extract salient features of statistical relationships in order to build cooperatively meaningful representations of some relevant content. Modeling the structure of data and its hidden representations by independently identically distributed samples, our goal is to study fundamental limits of the so-called Two-way Collaborative Representation Learning (TW-CRL) and the Collaborative Distributed Representation Learning (CDRL) problems. The TW-CRL problem consists of two distant encoders that separately observe marginal (dependent) components $X_1$ and $X_2$ and can cooperate through multiple exchanges of limited information with the aim of learning hidden representations $(Y_1,Y_2)$, which can be arbitrarily dependent on $(X_1,X_2)$. On the other hand, in CDRL there are two cooperating encoders and the learner of the hidden representation $Y$ is a third node which can listen the exchanges between the two encoders. The relevance (figure-of-merit) of such learned representations is measured in terms of a normalized (per-sample) multi-letter mutual information metric. Inner and outer bounds to the complexity-relevance region of these problems are derived from which optimality is characterized for several cases of interest. Our resulting complexity-relevance regions are finally evaluated for binary symmetric and Gaussian statistical models showing how to identify comparatively random features that represent complexity-constrained statistics for the inference of the hidden representations.
Comments: Submitted to IEEE Transactions on Information Theory
Subjects: Information Theory (cs.IT); Machine Learning (stat.ML)
Cite as: arXiv:1604.01433 [cs.IT]
  (or arXiv:1604.01433v1 [cs.IT] for this version)

Submission history

From: Leonardo Rey Vega [view email]
[v1] Tue, 5 Apr 2016 21:37:05 GMT (310kb,D)
[v2] Fri, 17 Nov 2017 13:23:10 GMT (313kb,D)

Link back to: arXiv, form interface, contact.