Advertisement

‘No Effect’ Conclusions in Studies Reporting Nonsignificant Results Are Potentially Incorrect

Published:September 08, 2021DOI:https://doi.org/10.1016/j.arthro.2021.08.036

      Purpose

      To examine the spectrum of effect sizes in line with “no effect” claims in clinical studies published in high-impact orthopaedic journals.

      Methods

      Eight orthopaedic journals with the highest impact factors for 2018 were included in this study. The titles and abstracts of all journal articles published in these journals in 2019 were screened for no-effect statements. The effect sizes for effects claimed to be no effect were calculated. The asymmetry of 95% confidence interval (CI) tails of effect estimated in relation to 0 was examined by calculating the ratio between the tail lengths.

      Results

      The analysis included 255 articles reporting coefficients sufficiently. The median Cohen’s d value was 0.15 (interquartile range [IQR] 0.06 to 0.25) in the studies comparing means and the median ϕ value in the studies comparing frequency distributions was 0.06 (IQR 0.04 to 0.15). In studies reporting odds ratios or hazard ratios, the median estimate value was 1.16 (IQR 1.09 to 1.34). The median asymmetry ratio for all studies was 1.9 (IQR 1.3 to 4.0). Asymmetry ratio values exceeded 5 in 55 studies (22%).

      Conclusions

      No-effect statements are used for a wide variety of observed effects. Despite the conclusion of no effect, observed effects advocated toward actual effects. In addition, many cases in which the asymmetry of CIs related to the 0-effect estimate showed a high tendency of effect direction.

      Clinical Relevance

      Rather than emphasizing dichotomized interpretation of statistical inference, reporting observed coefficients with elaboration of related uncertainty and compatibility with meaningful effect sizes in the specific context is encouraged.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Arthroscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Chavalarias D.
        • Wallach J.D.
        • Li A.H.T.
        • et al.
        Evolution of reporting P values in the biomedical literature, 1990-2015.
        JAMA. 2016; 315: 1141-1148
        • Fanelli D.
        Negative results are disappearing from most disciplines and countries.
        Scientometrics. 2012; 90: 891-904
        • Stang A.
        • Deckert M.
        • Poole C.
        • et al.
        Statistical inference in abstracts of major medical and epidemiology journals 1975–2014: A systematic review.
        Eur J Epidemiol. 2017; 32: 21-29
        • Greenland S.
        • Senn S.J.
        • Rothman K.J.
        • et al.
        Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations.
        Eur J Epidemiol. 2016; 31: 337-350
        • Greenland S.
        Null misinterpretation in statistical testing and its impact on health risk assessment.
        Prev Med. 2011; 53: 225-228
        • Alderson P.
        • Chalmers I.
        Survey of claims of no effect in abstracts of Cochrane reviews.
        BMJ. 2003; 326: 475
        • Altman D.G.
        • Bland J.M.
        Statistics notes: Absence of evidence is not evidence of absence.
        BMJ. 1995; 311: 485
        • Gates S.
        • Ealing E.
        Reporting and interpretation of results from clinical trials that did not claim a treatment difference: Survey of four general medical journals.
        BMJ Open. 2019; 9e024785
        • Smith P.R.M.
        • Ware L.
        • Adams C.
        • et al.
        Claims of ‘no difference’or ‘no effect’ in Cochrane and other systematic reviews.
        BMJ Evid Based Med. 2021; 26: 118-120
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        Academic Press, Cambridge, MA2013
        • da Costa B.R.
        • Reichenbach S.
        • Keller N.
        • et al.
        Effectiveness of non-steroidal anti-inflammatory drugs for the treatment of pain in knee and hip osteoarthritis: A network meta-analysis.
        Lancet. 2017; 390: e21-e33
        • Fransen M.
        • McConnell S.
        • Hernandez-Molina G.
        • et al.
        Exercise for osteoarthritis of the hip.
        Cochrane Database Syst Rev. 2003; : CD004286
        • Lamberink H.J.
        • Otte W.M.
        • Sinke M.R.
        • et al.
        Statistical power of clinical trials increased while effect size remained stable: An empirical analysis of 136,212 clinical trials between 1975 and 2014.
        J Clin Epidemiol. 2018; 102: 123-128
        • Domb B.G.
        • Sabetian P.W.
        When no difference does not mean there is no difference: The blight of the type II error.
        Arthroscopy. 2021; 37: 1353-1356
        • Huth E.J.
        • King K.
        • Lock S.
        • et al.
        Uniform requirements for manuscripts submitted to biomedical journals.
        Ann Intern Med. 1988; 108: 258-265
        • Moher D.
        • Hopewell S.
        • Schulz K.F.
        • et al.
        CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.
        BMJ. 2010; 340: c869
        • Raittio L.
        • Reito A.
        Assessing variability and uncertainty in orthopedic randomized controlled trials.
        Acta Orthop. 2020; 91: 479-484
        • Vavken P.
        • Heinrich K.M.
        • Koppelhuber C.
        • et al.
        The use of confidence intervals in reporting orthopaedic research findings.
        Clin Orthop Relat Res. 2009; 467: 3334
        • Wasserstein R.L.
        • Lazar N.A.
        The ASA statement on p-values: Context, process, and purpose.
        Taylor & Francis, Milton Park, UK2016
        • Wagenmakers E.-J.
        A practical solution to the pervasive problems of p values.
        Psychonom Bull Rev. 2007; 14: 779-804
        • Edwards W.
        • Lindman H.
        • Savage L.J.
        Bayesian statistical inference for psychological research.
        Psychol Rev. 1963; 70: 193
        • Berger J.O.
        • Sellke T.
        Testing a point null hypothesis: The irreconcilability of p values and evidence.
        J Am Stat Assoc. 1987; 82: 112-122
        • Sellke T.
        • Bayarri M.
        • Berger J.O.
        Calibration of ρ values for testing precise null hypotheses.
        Am Stat. 2001; 55: 62-71
        • Goodman S.N.
        Introduction to Bayesian methods I: Measuring the strength of evidence.
        Sage Publications, Thousand Oaks, CA2005
        • King M.T.
        A point of minimal important difference (MID): A critique of terminology and methods.
        Expert Rev Pharmacoecon Outcomes Res. 2011; 11: 171-184
        • Devji T.
        • Guyatt G.H.
        • Lytvyn L.
        • et al.
        Application of minimal important differences in degenerative knee disease outcomes: A systematic review and case study to inform BMJ Rapid Recommendations.
        BMJ Open. 2017; 7e015587
        • Young P.J.
        • Nickson C.P.
        • Perner A.
        When should clinicians act on non–statistically significant results from clinical trials?.
        JAMA. 2020; 323: 2256-2257