Nearly One-Third of Published Systematic Reviews and Meta-analyses Yield Inconclusive Conclusions: A Systematic Review



      To perform a systematic review that determines the percentage of published orthopedic surgery and sports medicine systematic reviews and meta-analyses that have a conclusive conclusion.


      A systematic review was performed using PRISMA guidelines. Six high-quality orthopedics journals were chosen for analysis over a 10-year eligibility period. Systematic reviews and meta-analyses published in these journals were included in the investigation. Narrative, scoping, and umbrella reviews were excluded. A systematic review or meta-analysis was defined as having an inconclusive conclusion if the conclusion in the manuscript body or abstract was stated directly as inconclusive, indeterminate, unknown, or having a lack of evidence (or no evidence). A conclusive conclusion stated a direct answer to the study’s primary and/or accessory outcomes. Due to the categorical nature of the data, comparisons were made using χ2 test and logistic regression.


      There were 1,108 systematic reviews/meta-analyses analyzed (30.9 ± 70.3 studies analyzed per review). More reviews (69.9%) were published with conclusive conclusions rather than without (30.1%). More reviews were surgical (73%) rather than nonsurgical. The United States and North America published the most reviews by country and continent, respectively. There were statistically significant differences between countries (highest proportion with China) and continents (highest proportion with Asia) based on the number of conclusive conclusions in published reviews, respectively. There were no significant differences in the proportion of conclusive conclusion reviews between the 6 analyzed journals. Australia published the largest proportion on nonsurgical reviews. The British Journal of Sports Medicine published a significantly higher proportion of nonsurgical reviews than the other 5 journals. There was no temporal relationship with the proportion of conclusive conclusion reviews.


      This systematic review observed that only 70% of orthopedic systematic reviews and meta-analyses published in 6 high-quality orthopedic journals over a 10-year eligibility period had conclusive conclusions.

      Level of Evidence

      Level IV, systematic review and/or meta-analysis of studies with Levels I to IV.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Arthroscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Harris J.D.
        • Brand J.C.
        • Cote M.P.
        • Dhawan A.
        Research pearls: the significance of statistics and perils of pooling. Part 3: Pearls and pitfalls of meta-analyses and systematic reviews.
        Arthroscopy. 2017; 33: 1594-1602
        • Ioannidis J.P.
        The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
        Milbank Q. 2016; 94: 485-514
        • Anderson M.J.
        • Browning III, W.M.
        • Urband C.E.
        • Kluczynski M.A.
        • Bisson L.J.
        A systematic summary of systematic reviews on the topic of the anterior cruciate ligament.
        Orthop J Sports Med. 2016; 4 (2325967116634074)
        • Webster K.E.
        • Hewett T.E.
        Meta-analysis of meta-analyses of anterior cruciate ligament injury reduction training programs.
        J Orthop Res. 2018; 36: 2696-2708
        • Collins J.A.
        • Ward J.P.
        • Youm T.
        Is prophylactic surgery for femoroacetabular impingement indicated? A systematic review.
        Am J Sports Med. 2014; 42: 3009-3015
        • Garner P.
        • Hopewell S.
        • Chandler J.
        • et al.
        When and how to update systematic reviews: Consensus and checklist.
        BMJ. 2016; 354: i3507
        • Wright J.G.
        • Swiontkowski M.F.
        • Tolo V.T.
        Meta-analyses and systematic reviews: New guidelines for JBJS.
        J Bone Joint Surg Am. 2012; 94: 1537
        • Juhl C.B.
        • Lund H.
        Do we really need another systematic review?.
        Br J Sports Med. 2018; 52: 1408-1409
        • Lund H.
        • Juhl C.
        • Christensen R.
        Systematic reviews and research waste.
        Lancet. 2016; 387: 123-124
        • Chalmers I.
        • Glasziou P.
        Systematic reviews and research waste.
        Lancet. 2016; 387: 122-123
        • Nelson R.L.
        Systematic reviews and research waste.
        Lancet. 2016; 387: 124
        • Roberts I.
        • Ker K.
        How systematic reviews cause research waste.
        Lancet. 2015; 386: 1536
        • Dijkman B.G.
        • Abouali J.A.
        • Kooistra B.W.
        • et al.
        Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity?.
        J Bone Joint Surg Am. 2010; 92: 48-57
        • Manta A.
        • Opingari E.
        • Saleh A.H.
        • et al.
        A systematic review of meta-analyses in orthopaedic surgery between 2000 and 2016.
        Bone Joint J. 2018; 100: 1270-1274
        • Gagnier J.J.
        • Kellam P.J.
        Reporting and methodological quality of systematic reviews in the orthopaedic literature.
        J Bone Joint Surg Am. 2013; 95: e771-e777
      1. PROSPERO—International Prospective Register of Systematic Reviews. 2018.
        Date accessed: October 4, 2018
      2. OCEBM Levels of Evidence (Oxford Centre for Evidence-Based Medicine). University of Oxford, 2016
        • Shea B.J.
        • Hamel C.
        • Wells G.A.
        • et al.
        AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
        J Clin Epidemiol. 2009; 62: 1013-1020
        • Kung J.
        • Chiappelli F.
        • Cajulis O.O.
        • et al.
        From systematic reviews to clinical recommendations for evidence-based health care: Validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for grading of clinical relevance.
        Open Dent J. 2010; 4: 84-91
        • Shea B.J.
        • Reeves B.C.
        • Wells G.
        • et al.
        AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.
        BMJ. 2017; 358: j4008
        • Moher D.
        • Cook D.J.
        • Eastwood S.
        • Olkin I.
        • Rennie D.
        • Stroup D.F.
        Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Quality of Reporting of Meta-analyses.
        Lancet. 1999; 354: 1896-1900
        • Liberati A.
        • Altman D.G.
        • Tetzlaff J.
        • et al.
        The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.
        J Clin Epidemiol. 2009; 62: e1-e34
        • Harris J.D.
        • Quatman C.E.
        • Manring M.M.
        • Siston R.A.
        • Flanigan D.C.
        How to write a systematic review.
        Am J Sports Med. 2014; 42: 2761-2768
        • Harris J.D.
        • Cvetanovich G.
        • Erickson B.J.
        • et al.
        Current status of evidence-based sports medicine.
        Arthroscopy. 2014; 30: 362-371
        • Weir A.
        • Rabia S.
        • Ardern C.
        Trusting systematic reviews and meta-analyses: All that glitters is not gold!.
        Br J Sports Med. 2016; 50: 1100-1101
        • DiSilvestro K.J.
        • Tjoumakaris F.P.
        • Maltenfort M.G.
        • Spindler K.P.
        • Freedman K.B.
        Systematic reviews in sports medicine.
        Am J Sports Med. 2016; 44: 533-538