We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Shades of BLEU, Flavours of Success: The Case of MultiWOZ

Abstract: The MultiWOZ dataset (Budzianowski et al.,2018) is frequently used for benchmarking context-to-response abilities of task-oriented dialogue systems. In this work, we identify inconsistencies in data preprocessing and reporting of three corpus-based metrics used on this dataset, i.e., BLEU score and Inform & Success rates. We point out a few problems of the MultiWOZ benchmark such as unsatisfactory preprocessing, insufficient or under-specified evaluation metrics, or rigid database. We re-evaluate 7 end-to-end and 6 policy optimization models in as-fair-as-possible setups, and we show that their reported scores cannot be directly compared. To facilitate comparison of future systems, we release our stand-alone standardized evaluation scripts. We also give basic recommendations for corpus-based benchmarking in future works.
Comments: Accepted to GEM Workshop at ACL 2021; for the source files, see this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2106.05555 [cs.CL]
  (or arXiv:2106.05555v1 [cs.CL] for this version)

Submission history

From: Tomáš Nekvinda [view email]
[v1] Thu, 10 Jun 2021 07:33:53 GMT (63kb,D)

Link back to: arXiv, form interface, contact.