We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: A non-hierarchical attention network with modality dropout for textual response generation in multimodal dialogue systems

Abstract: Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, we propose a non-hierarchical attention network with modality dropout, which abandons the HRED framework and utilizes attention modules to encode each utterance and model the context representation. To evaluate our proposed model, we conduct comprehensive experiments on a public multimodal dialogue dataset. Automatic and human evaluation demonstrate that our proposed model outperforms the existing methods and achieves state-of-the-art performance.
Comments: Submitted to ICASSP2022 (currently under review)
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2110.09702 [cs.CL]
  (or arXiv:2110.09702v2 [cs.CL] for this version)

Submission history

From: Rongyi Sun [view email]
[v1] Tue, 19 Oct 2021 03:08:16 GMT (343kb,D)
[v2] Wed, 20 Oct 2021 03:55:34 GMT (343kb,D)

Link back to: arXiv, form interface, contact.