We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Electrical Engineering and Systems Science > Signal Processing

Title: Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies

Abstract: Humans are emotional creatures. Multiple modalities are often involved when we express emotions, whether we do so explicitly (e.g., facial expression, speech) or implicitly (e.g., text, image). Enabling machines to have emotional intelligence, i.e., recognizing, interpreting, processing, and simulating emotions, is becoming increasingly important. In this tutorial, we discuss several key aspects of multi-modal emotion recognition (MER). We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks, followed by the description of main challenges in MER. Furthermore, we present some representative approaches on representation learning of each affective modality, feature fusion of different affective modalities, classifier optimization for MER, and domain adaptation for MER. Finally, we outline several real-world applications and discuss some future directions.
Comments: Accepted by IEEE Signal Processing Magazine (SPM)
Subjects: Signal Processing (eess.SP); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)
DOI: 10.1109/MSP.2021.3106895
Cite as: arXiv:2108.10152 [eess.SP]
  (or arXiv:2108.10152v1 [eess.SP] for this version)

Submission history

From: Sicheng Zhao [view email]
[v1] Wed, 18 Aug 2021 21:55:20 GMT (3019kb,D)

Link back to: arXiv, form interface, contact.