Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Learning from Multiview Correlations in Open-Domain Videos
(Submitted on 21 Nov 2018 (v1), last revised 1 Mar 2019 (this version, v2))
Abstract: An increasing number of datasets contain multiple views, such as video, sound and automatic captions. A basic challenge in representation learning is how to leverage multiple views to learn better representations. This is further complicated by the existence of a latent alignment between views, such as between speech and its transcription, and by the multitude of choices for the learning objective. We explore an advanced, correlation-based representation learning method on a 4-way parallel, multimodal dataset, and assess the quality of the learned representations on retrieval-based tasks. We show that the proposed approach produces rich representations that capture most of the information shared across views. Our best models for speech and textual modalities achieve retrieval rates from 70.7% to 96.9% on open-domain, user-generated instructional videos. This shows it is possible to learn reliable representations across disparate, unaligned and noisy modalities, and encourages using the proposed approach on larger datasets.
Submission history
From: Nils Holzenberger [view email][v1] Wed, 21 Nov 2018 19:57:11 GMT (375kb,D)
[v2] Fri, 1 Mar 2019 18:21:28 GMT (286kb,D)
Link back to: arXiv, form interface, contact.