We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Jurassic is (almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue

Authors: Lena Reed, Cecilia Li, Angela Ramirez, Liren Wu, Marilyn Walker (Natural Language and Dialogue Systems Lab, University of California, Santa Cruz)
Abstract: One challenge with open-domain dialogue systems is the need to produce truthful, high-quality responses on any topic. We aim to improve the quality and coverage of Athena, an Alexa Prize dialogue system. We experiment with few-shot prompt-based learning, comparing GPT-Neo to Jurassic-1, for the movies, music, TV, sports, and video game domains, both within and cross-domain, with different prompt set sizes (2, 3, 10), formats, and meaning representations consisting of either sets of WikiData KG triples, or dialogue acts. Our evaluation uses BLEURT and human metrics, and shows that with 10-shot prompting, Athena-Jurassic's performance is significantly better for coherence and semantic accuracy. Experiments with 2-shot cross-domain prompts results in a huge performance drop for Athena-GPT-Neo, whose semantic accuracy falls to 0.41, and whose untrue hallucination rate increases to 12%. Experiments with dialogue acts for video games show that with 10-shot prompting, both models learn to control dialogue acts, but Athena-Jurassic has significantly higher coherence, and only 4% untrue hallucinations. Our results suggest that Athena-Jurassic produces high enough quality outputs to be useful in live systems with real users. To our knowledge, these are the first results demonstrating that few-shot semantic prompt-based learning can create NLGs that generalize to new domains, and produce high-quality, semantically-controlled, conversational responses directly from meaning representations.
Comments: Final Conference Proceedings version
Subjects: Computation and Language (cs.CL)
Journal reference: The 12th International Workshop on Spoken Dialog System Technology, IWSDS 2021
Cite as: arXiv:2110.08094 [cs.CL]
  (or arXiv:2110.08094v2 [cs.CL] for this version)

Submission history

From: Marilyn Walker [view email]
[v1] Fri, 15 Oct 2021 13:42:25 GMT (2041kb,D)
[v2] Wed, 10 Nov 2021 13:36:14 GMT (1971kb,D)

Link back to: arXiv, form interface, contact.