References & Citations
Computer Science > Computation and Language
Title: Pointwise Paraphrase Appraisal is Potentially Problematic
(Submitted on 25 May 2020 (v1), last revised 5 Jun 2020 (this version, v2))
Abstract: The prevailing approach for training and evaluating paraphrase identification models is constructed as a binary classification problem: the model is given a pair of sentences, and is judged by how accurately it classifies pairs as either paraphrases or non-paraphrases. This pointwise-based evaluation method does not match well the objective of most real world applications, so the goal of our work is to understand how models which perform well under pointwise evaluation may fail in practice and find better methods for evaluating paraphrase identification models. As a first step towards that goal, we show that although the standard way of fine-tuning BERT for paraphrase identification by pairing two sentences as one sequence results in a model with state-of-the-art performance, that model may perform poorly on simple tasks like identifying pairs with two identical sentences. Moreover, we show that these models may even predict a pair of randomly-selected sentences with higher paraphrase score than a pair of identical ones.
Submission history
From: Hannah Chen [view email][v1] Mon, 25 May 2020 09:27:31 GMT (144kb,D)
[v2] Fri, 5 Jun 2020 03:18:28 GMT (144kb,D)
Link back to: arXiv, form interface, contact.