Current browse context:
cs.HC
Change to browse by:
References & Citations
Computer Science > Human-Computer Interaction
Title: Multimodal analysis of the predictability of hand-gesture properties
(Submitted on 12 Aug 2021 (v1), last revised 14 Jan 2022 (this version, v3))
Abstract: Embodied conversational agents benefit from being able to accompany their speech with gestures. Although many data-driven approaches to gesture generation have been proposed in recent years, it is still unclear whether such systems can consistently generate gestures that convey meaning. We investigate which gesture properties (phase, category, and semantics) can be predicted from speech text and/or audio using contemporary deep learning. In extensive experiments, we show that gesture properties related to gesture meaning (semantics and category) are predictable from text features (time-aligned FastText embeddings) alone, but not from prosodic audio features, while rhythm-related gesture properties (phase) on the other hand can be predicted from audio features better than from text. These results are encouraging as they indicate that it is possible to equip an embodied agent with content-wise meaningful co-speech gestures using a machine-learning model.
Submission history
From: Taras Kucherenko [view email][v1] Thu, 12 Aug 2021 14:16:00 GMT (4856kb,D)
[v2] Wed, 13 Oct 2021 08:46:37 GMT (5372kb,D)
[v3] Fri, 14 Jan 2022 10:34:25 GMT (5808kb,D)
Link back to: arXiv, form interface, contact.