Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Multimodal Generative Models for Scalable Weakly-Supervised Learning
(Submitted on 14 Feb 2018 (v1), last revised 12 Nov 2018 (this version, v3))
Abstract: Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.
Submission history
From: Mike Wu [view email][v1] Wed, 14 Feb 2018 22:08:05 GMT (2387kb,D)
[v2] Fri, 18 May 2018 18:14:29 GMT (2832kb,D)
[v3] Mon, 12 Nov 2018 16:39:10 GMT (2865kb,D)
Link back to: arXiv, form interface, contact.