We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ME

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo

Statistics > Methodology

Title: The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results

Abstract: Statistical insignificance does not suggest the absence of effect, yet scientists must often use null results as evidence of negligible (near-zero) effect size to falsify scientific hypotheses. Doing so must assess a result's null strength, defined as the evidence for a negligible effect size. Such an assessment would differentiate high null strength results that suggest a negligible effect size from low null strength results that suggest a broad range of potential effect sizes. We propose the most difference in means ({\delta}M) as a two-sample statistic that can both quantify null strength and perform a hypothesis test for determining negligible effect size. Scientists will likely disagree on the value of the threshold for negligible effect. To facilitate consensus when interpreting results, our statistic allows scientists to conclude that a result has negligible effect size using different thresholds with no recalculation of the statistic required. This is accomplished with a post-hoc hypothesis test for negligibility. To assist with selecting a threshold, {\delta}M can also compare null strength between related results. Both {\delta}M and the relative form of {\delta}M outperform other candidate statistics in identifying results with higher null strength. We compile results from broadly related experiments and use the relative {\delta}M to compare null strength across different treatments, measurement methods, and experiment models. Reporting the relative {\delta}M may provide a technical solution to the file drawer problem by encouraging the publication of null and near-zero results and allowing scientists with different perspectives to reach consensus for which results have negligible effect.
Subjects: Methodology (stat.ME); Quantitative Methods (q-bio.QM)
Cite as: arXiv:2201.01239 [stat.ME]
  (or arXiv:2201.01239v3 [stat.ME] for this version)

Submission history

From: Bruce Corliss [view email]
[v1] Tue, 4 Jan 2022 16:46:44 GMT (6425kb)
[v2] Thu, 6 Jan 2022 04:54:42 GMT (6425kb)
[v3] Fri, 1 Apr 2022 17:41:04 GMT (6464kb)
[v4] Tue, 24 May 2022 23:39:03 GMT (6185kb)

Link back to: arXiv, form interface, contact.