We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Methodology

Title: Valid sequential inference on probability forecast performance

Abstract: Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to p-values for hypothesis testing, and they can easily be transformed into conservative p-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation taking into account the available data at any time and still draw statistically valid inference, which is generally not true for classical p-value based tests. In a case study on postprocessing of precipitation forecasts, state-of-the-art forecasts dominance tests and e-values lead to the same conclusions.
Subjects: Methodology (stat.ME); Statistics Theory (math.ST)
Cite as: arXiv:2103.08402 [stat.ME]
  (or arXiv:2103.08402v2 [stat.ME] for this version)

Submission history

From: Alexander Henzi [view email]
[v1] Mon, 15 Mar 2021 14:18:03 GMT (459kb,D)
[v2] Tue, 16 Mar 2021 09:22:24 GMT (459kb,D)

Link back to: arXiv, form interface, contact.