We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Representation Learning for Conversational Data using Discourse Mutual Information Maximization

Abstract: Although many pretrained models exist for text or images, there have been relatively fewer attempts to train representations specifically for dialog understanding. Prior works usually relied on finetuned representations based on generic text representation models like BERT or GPT-2. But such language modeling pretraining objectives do not take the structural information of conversational text into consideration. Although generative dialog models can learn structural features too, we argue that the structure-unaware word-by-word generation is not suitable for effective conversation modeling. We empirically demonstrate that such representations do not perform consistently across various dialog understanding tasks. Hence, we propose a structure-aware Mutual Information based loss-function DMI (Discourse Mutual Information) for training dialog-representation models, that additionally captures the inherent uncertainty in response prediction. Extensive evaluation on nine diverse dialog modeling tasks shows that our proposed DMI-based models outperform strong baselines by significant margins.
Comments: Preprint, 15 pages, To appear in NAACL 2022 (Main)
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2112.05787 [cs.CL]
  (or arXiv:2112.05787v2 [cs.CL] for this version)

Submission history

From: Pawan Goyal [view email]
[v1] Sat, 4 Dec 2021 13:17:07 GMT (4086kb,D)
[v2] Tue, 3 May 2022 14:21:56 GMT (7719kb,D)

Link back to: arXiv, form interface, contact.