We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Minimum discrepancy principle strategy for choosing $k$ in $k$-NN regression

Abstract: We present a novel data-driven strategy to choose the hyperparameter $k$ in the $k$-NN regression estimator. We treat the problem of choosing the hyperparameter as an iterative procedure (over $k$) and propose using an easily implemented in practice strategy based on the idea of early stopping and the minimum discrepancy principle. This model selection strategy is proven to be minimax-optimal, under the fixed-design assumption on covariates, over some smoothness function classes, for instance, the Lipschitz functions class on a bounded domain. The novel method often improves statistical performance on artificial and real-world data sets in comparison to other model selection strategies, such as the Hold-out method and 5-fold cross-validation. The novelty of the strategy comes from reducing the computational time of the model selection procedure while preserving the statistical (minimax) optimality of the resulting estimator. More precisely, given a sample of size $n$, assuming that the nearest neighbors are already precomputed, if one should choose $k$ among $\left\{ 1, \ldots, n \right\}$, the strategy reduces the computational time of the generalized cross-validation or Akaike's AIC criteria from $\mathcal{O}\left( n^3 \right)$ to $\mathcal{O}\left( n^2 (n - k) \right)$, where $k$ is the proposed (minimum discrepancy principle) value of the nearest neighbors. Code for the simulations is provided at this https URL
Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Statistics Theory (math.ST)
Cite as: arXiv:2008.08718 [stat.ML]
  (or arXiv:2008.08718v4 [stat.ML] for this version)

Submission history

From: Yaroslav Averyanov [view email]
[v1] Thu, 20 Aug 2020 00:13:19 GMT (93kb,D)
[v2] Wed, 20 Jan 2021 15:11:35 GMT (178kb)
[v3] Fri, 26 Feb 2021 11:53:33 GMT (212kb)
[v4] Wed, 5 May 2021 11:33:16 GMT (68kb)

Link back to: arXiv, form interface, contact.