Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing

Abstract

Null Hypothesis Significance Testing (NHST) remains the cornerstone of hypothesis testing in management research, yet it faces enduring critiques regarding its scientific limitations. This paper addresses these critiques and proposes strategies to mitigate them while preserving NHST's utility. We focus on four key issues: (1) the binary nature of NHST, (2) its limited practical relevance, (3) its role in fostering publication bias and questionable research practices, and (4) widespread misinterpretations of statistical significance. Arguing that alternative statistical frameworks cannot fully resolve these challenges, we advocate for a nuanced application of NHST emphasizing practical significance. Our recommendations include setting a minimum effect size of interest, adjusting alpha levels based on sample size, and promoting the publication of null results through preregistration and transparent reporting. To substantiate our approach, we analyze 45 articles from the Strategic Management Journal following an editorial shift criticizing standard NHST practices, finding that a preference for binary results persists. By adopting our proposed strategies, researchers can derive more actionable conclusions, advance cumulative scientific knowledge, and more effectively navigate the inherent uncertainties of empirical research.