### Current browse context:

math.ST

### Change to browse by:

### References & Citations

# Mathematics > Statistics Theory

# Title: On lower bounds for the bias-variance trade-off

(Submitted on 30 May 2020 (v1), last revised 20 Nov 2021 (this version, v3))

Abstract: It is a common phenomenon that for high-dimensional and nonparametric statistical models, rate-optimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the trade-off between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the bias-variance trade-off is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the Kullback-Leibler or chi-square-divergence. Some of these inequalities rely on a new concept of information matrices. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the high-dimensional linear regression model. For these specific statistical applications, different types of bias-variance trade-offs occur that vary considerably in their strength. For the trade-off between integrated squared bias and integrated variance in the Gaussian white noise model, we combine the general strategy for lower bounds with a reduction technique. This allows us to link the original problem to the bias-variance trade-off for estimators with additional symmetry properties in a simpler statistical model. In the Gaussian sequence model, different phase transitions of the bias-variance trade-off occur. Although there is a non-trivial interplay between bias and variance, the rate of the squared bias and the variance do not have to be balanced in order to achieve the minimax estimation rate.

## Submission history

From: Johannes Schmidt-Hieber [view email]**[v1]**Sat, 30 May 2020 14:07:43 GMT (50kb)

**[v2]**Thu, 4 Nov 2021 15:34:48 GMT (69kb)

**[v3]**Sat, 20 Nov 2021 21:09:48 GMT (70kb)

Link back to: arXiv, form interface, contact.