We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation

Abstract: Evaluating automatically generated text is generally hard due to the inherently subjective nature of many aspects of the output quality. This difficulty is compounded in automatic consultation note generation by differing opinions between medical experts both about which patient statements should be included in generated notes and about their respective importance in arriving at a diagnosis. Previous real-world evaluations of note-generation systems saw substantial disagreement between expert evaluators. In this paper we propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. We observed good levels of inter-annotator agreement in a first evaluation study using the protocol; further, using Consultation Checklists produced in the study as reference for automatic metrics such as ROUGE or BERTScore improves their correlation with human judgements compared to using the original human note.
Comments: Accepted for publication at EMNLP 2022
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2211.09455 [cs.CL]
  (or arXiv:2211.09455v1 [cs.CL] for this version)

Submission history

From: Francesco Moramarco [view email]
[v1] Thu, 17 Nov 2022 10:54:28 GMT (1419kb,D)

Link back to: arXiv, form interface, contact.