We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

math.OC

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: A Generalized Weighted Optimization Method for Computational Learning and Inversion

Abstract: The generalization capacity of various machine learning models exhibits different phenomena in the under- and over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of different data points in the loss function. Here, we characterize the impact of the weighting scheme on the generalization error of the learning method, where we derive explicit generalization errors for the random Fourier feature model in both the under- and over-parameterized regimes. For more general feature maps, error bounds are provided based on the singular values of the feature matrix. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model.
Comments: 25 pages, 2 figures; Published as a conference paper at ICLR 2022. In International Conference on Learning Representations (2022)
Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC)
Cite as: arXiv:2201.09223 [cs.LG]
  (or arXiv:2201.09223v3 [cs.LG] for this version)

Submission history

From: Yunan Yang [view email]
[v1] Sun, 23 Jan 2022 10:35:34 GMT (773kb,D)
[v2] Fri, 25 Feb 2022 15:48:34 GMT (771kb,D)
[v3] Fri, 11 Mar 2022 16:31:55 GMT (774kb,D)

Link back to: arXiv, form interface, contact.