Current browse context:
stat.ME
Change to browse by:
References & Citations
Statistics > Methodology
Title: The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results
(Submitted on 4 Jan 2022 (v1), last revised 24 May 2022 (this version, v4))
Abstract: Statistical insignificance does not suggest the absence of effect, yet scientists must often use null results as evidence of negligible (near-zero) effect size to falsify scientific hypotheses. Doing so must assess a result's null strength, defined as the evidence for a negligible effect size. Such an assessment would differentiate strong null results that suggest a negligible effect size from weak null results that suggest a broad range of potential effect sizes. We propose the most difference in means ($\delta_M$) as a two-sample statistic that can both quantify null strength and perform a hypothesis test for negligible effect size. To facilitate consensus when interpreting results, our statistic allows scientists to conclude that a result has negligible effect size using different thresholds with no recalculation required. To assist with selecting a threshold, $\delta_M$ can also compare null strength between related results. Both $\delta_M$ and the relative form of $\delta_M$ outperform other candidate statistics in comparing null strength. We compile broadly related results and use the relative $\delta_M$ to compare null strength across different treatments, measurement methods, and experiment models. Reporting the relative $\delta_M$ may provide a technical solution to the file drawer problem by encouraging the publication of null and near-zero results.
Submission history
From: Bruce Corliss [view email][v1] Tue, 4 Jan 2022 16:46:44 GMT (6425kb)
[v2] Thu, 6 Jan 2022 04:54:42 GMT (6425kb)
[v3] Fri, 1 Apr 2022 17:41:04 GMT (6464kb)
[v4] Tue, 24 May 2022 23:39:03 GMT (6185kb)
Link back to: arXiv, form interface, contact.