We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

Abstract: Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. speech during a phone call. While previous work proposed crossmodal emotion embeddings to improve monomodal recognition performance, despite its importance, an explicit representation of gaze was not included. We propose a new approach to emotion recognition that incorporates an explicit representation of gaze in a crossmodal emotion embedding framework. We show that our method outperforms the previous state of the art for both audio-only and video-only emotion classification on the popular One-Minute Gradual Emotion Recognition dataset. Furthermore, we report extensive ablation experiments and provide detailed insights into the performance of different state-of-the-art gaze representations and integration strategies. Our results not only underline the importance of gaze for emotion recognition but also demonstrate a practical and highly effective approach to leveraging gaze information for this task.
Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
DOI: 10.1145/3530879
Cite as: arXiv:2205.00129 [cs.LG]
  (or arXiv:2205.00129v1 [cs.LG] for this version)

Submission history

From: Ahmed Abdou [view email]
[v1] Sat, 30 Apr 2022 02:05:39 GMT (3443kb,D)

Link back to: arXiv, form interface, contact.