Current browse context:
stat.ME
Change to browse by:
References & Citations
Statistics > Methodology
Title: The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results
(Submitted on 4 Jan 2022 (v1), revised 1 Apr 2022 (this version, v3), latest version 24 May 2022 (v4))
Abstract: Statistical insignificance does not suggest the absence of effect, yet scientists must often use null results as evidence of negligible (near-zero) effect size to falsify scientific hypotheses. Doing so must assess a result's null strength, defined as the evidence for a negligible effect size. Such an assessment would differentiate high null strength results that suggest a negligible effect size from low null strength results that suggest a broad range of potential effect sizes. We propose the most difference in means ({\delta}M) as a two-sample statistic that can both quantify null strength and perform a hypothesis test for determining negligible effect size. Scientists will likely disagree on the value of the threshold for negligible effect. To facilitate consensus when interpreting results, our statistic allows scientists to conclude that a result has negligible effect size using different thresholds with no recalculation of the statistic required. This is accomplished with a post-hoc hypothesis test for negligibility. To assist with selecting a threshold, {\delta}M can also compare null strength between related results. Both {\delta}M and the relative form of {\delta}M outperform other candidate statistics in identifying results with higher null strength. We compile results from broadly related experiments and use the relative {\delta}M to compare null strength across different treatments, measurement methods, and experiment models. Reporting the relative {\delta}M may provide a technical solution to the file drawer problem by encouraging the publication of null and near-zero results and allowing scientists with different perspectives to reach consensus for which results have negligible effect.
Submission history
From: Bruce Corliss [view email][v1] Tue, 4 Jan 2022 16:46:44 GMT (6425kb)
[v2] Thu, 6 Jan 2022 04:54:42 GMT (6425kb)
[v3] Fri, 1 Apr 2022 17:41:04 GMT (6464kb)
[v4] Tue, 24 May 2022 23:39:03 GMT (6185kb)
Link back to: arXiv, form interface, contact.