References & Citations
Computer Science > Computation and Language
Title: Segmenting Natural Language Sentences via Lexical Unit Analysis
(Submitted on 10 Dec 2020 (v1), last revised 16 Apr 2021 (this version, v3))
Abstract: In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.
Submission history
From: Yangming Li [view email][v1] Thu, 10 Dec 2020 02:31:52 GMT (266kb,D)
[v2] Mon, 14 Dec 2020 02:59:12 GMT (266kb,D)
[v3] Fri, 16 Apr 2021 08:30:03 GMT (223kb)
Link back to: arXiv, form interface, contact.