We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.LG

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Machine Learning

Title: Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayes

Abstract: We present new excess risk bounds for general unbounded loss functions including log loss and squared loss, where the distribution of the losses may be heavy-tailed. The bounds hold for general estimators, but they are optimized when applied to $\eta$-generalized Bayesian, MDL, and empirical risk minimization estimators. In the case of log loss, the bounds imply convergence rates for generalized Bayesian inference under misspecification in terms of a generalization of the Hellinger metric as long as the learning rate $\eta$ is set correctly. For general loss functions, our bounds rely on two separate conditions: the $v$-GRIP (generalized reversed information projection) conditions, which control the lower tail of the excess loss; and the newly introduced witness condition, which controls the upper tail. The parameter $v$ in the $v$-GRIP conditions determines the achievable rate and is akin to the exponent in the Tsybakov margin condition and the Bernstein condition for bounded losses, which the $v$-GRIP conditions generalize; favorable $v$ in combination with small model complexity leads to $\tilde{O}(1/n)$ rates. The witness condition allows us to connect the excess risk to an "annealed" version thereof, by which we generalize several previous results connecting Hellinger and R\'enyi divergence to KL divergence.
Comments: accepted to JMLR pending minor final modifications
Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Cite as: arXiv:1605.00252 [cs.LG]
  (or arXiv:1605.00252v4 [cs.LG] for this version)

Submission history

From: Nishant Mehta [view email]
[v1] Sun, 1 May 2016 13:35:15 GMT (73kb,D)
[v2] Thu, 7 Dec 2017 07:39:14 GMT (110kb,D)
[v3] Wed, 5 Sep 2018 16:26:22 GMT (116kb,D)
[v4] Tue, 5 Nov 2019 18:36:02 GMT (115kb,D)

Link back to: arXiv, form interface, contact.