We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IR

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Retrieval

Title: Impact of Stop Sets on Stopping Active Learning for Text Classification

Abstract: Active learning is an increasingly important branch of machine learning and a powerful technique for natural language processing. The main advantage of active learning is its potential to reduce the amount of labeled data needed to learn high-performing models. A vital aspect of an effective active learning algorithm is the determination of when to stop obtaining additional labeled data. Several leading state-of-the-art stopping methods use a stop set to help make this decision. However, there has been relatively less attention given to the choice of stop set than to the stopping algorithms that are applied on the stop set. Different choices of stop sets can lead to significant differences in stopping method performance. We investigate the impact of different stop set choices on different stopping methods. This paper shows the choice of the stop set can have a significant impact on the performance of stopping methods and the impact is different for stability-based methods from that on confidence-based methods. Furthermore, the unbiased representative stop sets suggested by original authors of methods work better than the systematically biased stop sets used in recently published work, and stopping methods based on stabilizing predictions have stronger performance than confidence-based stopping methods when unbiased representative stop sets are used. We provide the largest quantity of experimental results on the impact of stop sets to date. The findings are important for helping to illuminate the impact of this important aspect of stopping methods that has been under-considered in recently published work and that can have a large practical impact on the performance of stopping methods for important semantic computing applications such as technology assisted review and text classification more broadly.
Comments: 8 pages, 3 tables, 1 figure; published in Proceedings of the IEEE 16th International Conference on Semantic Computing (ICSC), pages 25-32, January 2022. IEEE
Subjects: Information Retrieval (cs.IR); Computation and Language (cs.CL); Machine Learning (cs.LG)
ACM classes: H.3.3; I.2.6; I.2.7; I.5.4
Journal reference: In Proceedings of the 2022 IEEE 16th International Conference on Semantic Computing (ICSC), pages 25-32, January 2022. IEEE
DOI: 10.1109/ICSC52841.2022.00012
Cite as: arXiv:2201.05460 [cs.IR]
  (or arXiv:2201.05460v2 [cs.IR] for this version)

Submission history

From: Michael Bloodgood [view email]
[v1] Sat, 8 Jan 2022 18:59:15 GMT (281kb,D)
[v2] Sat, 2 Apr 2022 05:19:00 GMT (281kb,D)

Link back to: arXiv, form interface, contact.