We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Ancillary-file links:

Ancillary files (details):

Current browse context:


Change to browse by:

References & Citations

DBLP - CS Bibliography


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis

Abstract: Multimodal sentiment analysis aims to recognize people's attitudes from multiple communication channels such as verbal content (i.e., text), voice, and facial expressions. It has become a vibrant and important research topic in natural language processing. Much research focuses on modeling the complex intra- and inter-modal interactions between different communication channels. However, current multimodal models with strong performance are often deep-learning-based techniques and work like black boxes. It is not clear how models utilize multimodal information for sentiment predictions. Despite recent advances in techniques for enhancing the explainability of machine learning models, they often target unimodal scenarios (e.g., images, sentences), and little research has been done on explaining multimodal models. In this paper, we present an interactive visual analytics system, M2Lens, to visualize and explain multimodal models for sentiment analysis. M2Lens provides explanations on intra- and inter-modal interactions at the global, subset, and local levels. Specifically, it summarizes the influence of three typical interaction types (i.e., dominance, complement, and conflict) on the model predictions. Moreover, M2Lens identifies frequent and influential multimodal features and supports the multi-faceted exploration of model behaviors from language, acoustic, and visual modalities. Through two case studies and expert interviews, we demonstrate our system can help users gain deep insights into the multimodal models for sentiment analysis.
Comments: 11 pages, 7 figures. This paper is accepted by IEEE VIS, 2021. To appear in IEEE Transactions on Visualization and Computer Graphics (TVCG)
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Multimedia (cs.MM)
ACM classes: I.2.7; H.1.2; H.4.2; J.4
Cite as: arXiv:2107.08264 [cs.LG]
  (or arXiv:2107.08264v4 [cs.LG] for this version)

Submission history

From: Xingbo Wang [view email]
[v1] Sat, 17 Jul 2021 15:54:27 GMT (17090kb,AD)
[v2] Tue, 20 Jul 2021 02:20:19 GMT (17090kb,AD)
[v3] Sun, 1 Aug 2021 04:25:16 GMT (17091kb,AD)
[v4] Sun, 29 Aug 2021 09:08:48 GMT (17091kb,AD)

Link back to: arXiv, form interface, contact.