Advertisement

Reviews Pooling Heterogeneous, Low-Evidence, High-Bias Data Result in Incorrect Conclusions: But Heterogeneity is an Opportunity to Explore

      Abstract

      Systematic Review submissions to our journal commonly pool heterogeneous studies of low levels of evidence and a high risk of bias. Pooling, or quantitative synthesis, of such study data regularly results in incorrect conclusions. We reject these submissions without peer-review (desk rejection), and typically invite authors to submit a new, subjective synthesis without pooling and to report ranges of the results of included studies rather than pooled values. Generally, quantitative synthesis, or meta-analysis, should restrict included studies to randomized controlled trials. However, systematic review with exploration of heterogeneity can result in valuable information toward determining strengths and deficiencies of current literature, and thus guide future research.
      Systematic Review submissions to our journal commonly pool heterogeneous studies of low levels of evidence and a high risk of bias. Pooling, or quantitative synthesis of such study data regularly results in incorrect conclusions.
      • Ioannidis J.P.
      The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
      • Cote M.P.
      • Apostolakos J.M.
      • Voss A.
      • DiVenere J.
      • Arciero R.A.
      • Mazzocca A.D.
      A systematic review of meta-analyses published in Arthroscopy: The Journal of Arthroscopic and Related Surgery.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      • Cote M.P.
      Editorial commentary: Leveraging the literature: What can a half-century’s worth of studies tell us about recurrent instability following primary shoulder dislocation?.
      We reject these submissions without peer-review (desk rejection),
      • Friedman P.K.
      Desk reject.
      and typically invite authors to submit a new, subjective synthesis without pooling and to report ranges of the results of included studies rather than pooled mean values. Generally, quantitative synthesis, or meta-analysis, should restrict included studies to randomized controlled trials.
      However, systematic review with exploration of heterogeneity can result in valuable information.
      • Lubowitz J.H.
      • Brand J.C.
      • Rossi M.J.
      Where do arthroscopy editors stand on systematic reviews of the literature?.
      • Dhawan A.
      • Brand J.C.
      • Provencher M.T.
      • Rossi M.J.
      • Lubowitz J.H.
      Research pearls: The significance of statistics and perils of pooling.
      • Lubowitz J.H.
      • Brand J.C.
      • Provencher M.T.
      • Rossi M.J.
      Systematic reviews keep arthroscopy up to date.
      Shifting the analysis away from summary estimates to explore sources of heterogeneity can result in identification of the clinical and methodologic factors with which the heterogeneity is associated. Then, instead of the trite and low-value conclusion “future research is required,” a conclusion can pinpoint deficiencies in the literature to inform and direct future investigation.

      A Hypothetical Example (Based on a Real Submission)

      Consider a review of 10 case series (Level IV evidence) reporting reoperation rates after knee arthroscopy. Data reported in the included studies is illustrated in Table 1, and a forest plot illustrating reoperation rates including 95% confidence intervals by study is illustrated in Figure 1.
      Table 1Number of Reoperations, Reoperation Rate, and Sample Size Reported in 10 Studies on Knee Arthroscopy
      Number of ReoperationsReoperation RateSample Size
      Study 112%45
      Study 24146%90
      Study 358%60
      Study 43244%72
      Study 59553%180
      Study 621%165
      Study 744%112
      Study 81219%62
      Study 95343%122
      Study 10511%47
      Total25026%955
      NOTE. Total reoperation rate of 26% represents weighted mean value of reoperation rates of the 10 studies as a result of improper (see text) pooling.
      Figure thumbnail gr1
      Fig 1Forest plot displaying effect size (reoperation rates represented by squares, proportional in size to sample size of the study) and confidence intervals (horizontal lines) of reoperation by study. (ES, Effect Size; CI, Confidence Interval; I-squared, percentage of variation due to heterogeneity rather than chance.)
      Hypothetical misinterpretation based on improper pooling is as follows: the rate of reoperation after knee arthroscopy is 26% (effect size = 0.26). Readers will note that the 26% weighted mean value or summary estimate of reoperation rates is labeled as improper in Table 1 and thus not included in Figure 1.
      But why do we deem this to be a misinterpretation based on “improper” pooling? The studies are wildly heterogeneous. Figure 1 shows that the I-squared statistic (how much of the difference in reoperation rates among the 10 studies is attributable to heterogeneity, rather than chance) is 98%. The large degree of statistical heterogeneity in a relatively small group of studies indicates underlying differences among the studies. Heterogeneity represents an opportunity to explore.
      Table 2 illustrates additional data regarding the included studies. Four included patients with chondral defects. Stratifying these (Fig 2) reveals reoperation rates among studies excluding chondral defects ranges from 1% to 11% (I-squared, 53%) versus rates ranging from 19% to 53% (I-squared, 86%) among studies including chondral defects. Accounting for chondral defects begins to improve the consistency of the outcomes. Methodologic factors may also be examined. Stratifying by type of study (prospective versus retrospective, Fig 3) reveals reoperation rates ranging from 43% to 53% (I-squared, only 7%) in prospective studies versus 1% to 19% (I-squared, 75%) in retrospective analyses. (Perhaps patients requiring reoperation after knee arthroscopy less commonly return to the same surgeon resulting in loss to research follow-up).
      Table 2Inclusion (Yes) or Exclusion (No) of Patients with Chondral Defects, and Type of Study (Prospective Versus Retrospective) Reported in 10 Studies on Knee Arthroscopy
      Chondral DefectType of Study
      Study 1NoRetrospective
      Study 2YesProspective
      Study 3NoRetrospective
      Study 4YesProspective
      Study 5YesProspective
      Study 6NoRetrospective
      Study 7NoRetrospective
      Study 8YesRetrospective
      Study 9YesProspective
      Study 10NoRetrospective
      Figure thumbnail gr2
      Fig 2Forest plot displaying effect size (reoperation rates represented by squares, proportional in size to sample size of the study) and confidence intervals (horizontal lines) of reoperation by study and by chondral defects. (ES, Effect Size; CI, Confidence Interval; I-squared, percentage of variation due to heterogeneity rather than chance.)
      Figure thumbnail gr3
      Fig 3Forest plot displaying effect size (reoperation rates represented by squares proportional in size to sample size of the study) and confidence intervals (horizontal lines) of reoperation by study and by type of study (ES, Effect Size; CI, Confidence Interval; I-squared, percentage of variation due to heterogeneity rather than chance.)
      Considering both chondral defects and type of study provides additional clarity (Fig 4). All 4 prospective studies include patients with chondral defects, but only 1 of 6 retrospective studies includes patients with chondral defects (another explanation for the higher reoperation rates in the hypothetical prospective studies). Visual inspection of the forest plot in Figure 4 indicates that Study 8 (the sole retrospective study including chondral defects) is primarily responsible for the greater heterogeneity among retrospective studies, because the Study 8 reoperation rate of 19% deviates substantially from the rates reported in the other retrospective studies.
      Figure thumbnail gr4
      Fig 4Forest plot displaying effect size (reoperation rates represented by squares, proportional in size to sample size of the study) and confidence intervals (horizontal lines) of reoperation by study and by chondral defects and by type of study. (ES, Effect Size; CI, Confidence Interval; I-squared, percentage of variation due to heterogeneity rather than chance.)

      High-value Conclusions

      As above, pooling low-evidence, high-risk-of-bias data from heterogeneous studies results in misinterpretation. We assert that the conclusion, “Knee arthroscopy has a 26% reoperation rate” is misconstrued, and the conclusion, “Future research is required” is pedestrian.
      Exploration of heterogeneity leads to more meaningful, high-value conclusions: The (hypothetical) literature on knee arthroscopy is limited to Level IV evidence. The reported rates of reoperation are highly inconsistent and range from 1% to 53%. Accounting for inclusion of patients with chondral defects at the time of surgery, and prospective study design improves the consistency of reported reoperation rates. Future retrospective case series are not recommended; this type of study contributes to inconsistency in the literature. Future study should be prospective and should include explicit eligibility criteria with regard to the presence or absence of chondral defects.

      Heterogeneity is an Opportunity to Explore

      We hope that someday, arthroscopic and related surgical literature will be replete with high-quality, low risk of bias, homogeneous studies, and pooling study data on most any topic will well-inform patient care. Until then, quantitative synthesis of heterogeneous studies with a low level of evidence and a high risk of bias must be avoided in favor of subjective synthesis combined with an exploratory embrace of heterogeneity to determine strengths and deficiencies of current literature and unambiguously guide future research.

      References

        • Ioannidis J.P.
        The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses.
        Milbank Q. 2016; 94: 485-514
        • Cote M.P.
        • Apostolakos J.M.
        • Voss A.
        • DiVenere J.
        • Arciero R.A.
        • Mazzocca A.D.
        A systematic review of meta-analyses published in Arthroscopy: The Journal of Arthroscopic and Related Surgery.
        Arthroscopy. 2016; 32: 528-537
        • Harris J.D.
        • Brand J.C.
        • Cote M.P.
        • Faucett S.C.
        • Dhawan A.
        Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
        Arthroscopy. 2017; 33: 1102-1112
        • Cote M.P.
        Editorial commentary: Leveraging the literature: What can a half-century’s worth of studies tell us about recurrent instability following primary shoulder dislocation?.
        Arthroscopy. 2017; 33: 1882-1883
        • Friedman P.K.
        Desk reject.
        (Available at)
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Where do arthroscopy editors stand on systematic reviews of the literature?.
        Arthroscopy. 2017; 33: 1753-1754
        • Dhawan A.
        • Brand J.C.
        • Provencher M.T.
        • Rossi M.J.
        • Lubowitz J.H.
        Research pearls: The significance of statistics and perils of pooling.
        Arthroscopy. 2017; 33: 1099-1101
        • Lubowitz J.H.
        • Brand J.C.
        • Provencher M.T.
        • Rossi M.J.
        Systematic reviews keep arthroscopy up to date.
        Arthroscopy. 2016; 32: 237