Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!
(Submitted on 22 Sep 2020 (v1), last revised 9 Aug 2021 (this version, v3))
Abstract: Despite efforts to distinguish three different evaluation setups (Bekoulis et al., 2018), numerous end-to-end Relation Extraction (RE) articles present unreliable performance comparison to previous work. In this paper, we first identify several patterns of invalid comparisons in published papers and describe them to avoid their propagation. We then propose a small empirical study to quantify the impact of the most common mistake and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. We also seize this opportunity to study the unexplored ablations of two recent developments: the use of language model pretraining (specifically BERT) and span-level NER. This meta-analysis emphasizes the need for rigor in the report of both the evaluation setting and the datasets statistics and we call for unifying the evaluation setting in end-to-end RE.
Submission history
From: Bruno Taillé [view email][v1] Tue, 22 Sep 2020 16:59:15 GMT (7355kb,D)
[v2] Mon, 26 Oct 2020 16:43:35 GMT (7208kb,D)
[v3] Mon, 9 Aug 2021 12:43:27 GMT (7280kb,D)
Link back to: arXiv, form interface, contact.