We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: Calibration for Stratified Classification Models

Authors: Chandler Zuo
Abstract: In classification problems, sampling bias between training data and testing data is critical to the ranking performance of classification scores. Such bias can be both unintentionally introduced by data collection and intentionally introduced by the algorithm, such as under-sampling or weighting techniques applied to imbalanced data. When such sampling bias exists, using the raw classification score to rank observations in the testing data can lead to suboptimal results. In this paper, I investigate the optimal calibration strategy in general settings, and develop a practical solution for one specific sampling bias case, where the sampling bias is introduced by stratified sampling. The optimal solution is developed by analytically solving the problem of optimizing the ROC curve. For practical data, I propose a ranking algorithm for general classification models with stratified data. Numerical experiments demonstrate that the proposed algorithm effectively addresses the stratified sampling bias issue. Interestingly, the proposed method shows its potential applicability in two other machine learning areas: unsupervised learning and model ensembling, which can be future research topics.
Comments: 14 pages, 12 figures
Subjects: Methodology (stat.ME); Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:1711.00064 [stat.ME]
  (or arXiv:1711.00064v1 [stat.ME] for this version)

Submission history

From: Chandler Zuo [view email]
[v1] Tue, 31 Oct 2017 19:23:57 GMT (2383kb,D)

Link back to: arXiv, form interface, contact.