Advertisement

Can’t See the Right Forest Plot for the Wrong Trees!

Published:September 23, 2020DOI:https://doi.org/10.1016/j.arthro.2020.07.046
      We have with great interest studied the meta-analysis and forest plot in Dwyer et al.
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      which is supposed to illustrate the effect of operative versus nonoperative treatment of femoroacetabular impingement syndrome. The meta-analysis, based on three randomized controlled trials,
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      • Griffin D.R.
      • Dickenson E.J.
      • Wall P.D.H.
      • et al.
      Hip arthroscopy versus best conservative care for the treatment of femoroacetabular impingement syndrome (UK FASHIoN): A multicentre randomised controlled trial.
      • Mansell N.S.
      • Rhon D.I.
      • Meyer J.
      • Slevin J.M.
      • Marchant B.G.
      Arthroscopic surgery or physical therapy for patients with femoroacetabular impingement syndrome: A randomized controlled trial with 2-year follow-up.
      shows a between-group difference on the International Hip Outcome Tool-33 (iHOT-33; 0 points [worst] to 100 points [best]), favoring operative treatment (mean difference: 3.46 points; 95% confidence interval [CI] 0.07 to 6.86).
      We congratulate the authors on the first meta-analysis for this very important topic, but unfortunately the forest plot presented by Dwyer et al.
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      is flawed owing to incorrect use of the iHOT-33 value derived from Palmer et al.,
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      published in 2019 in the British Medical Journal (BMJ). Dwyer et al.
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      have used the between-group difference of 2 points on the iHOT-33 reported by Palmer et al.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      The iHOT-33 score is normally measured on a visual analog scale from 0 to 100 mm, where each millimeter represents 1 point.
      • Mohtadi N.G.
      • Griffin D.R.
      • Pedersen M.E.
      • et al.
      The development and validation of a self-administered quality-of-life outcome measure for young, active patients with symptomatic hip disease: The International Hip Outcome Tool (iHOT-33).
      When Palmer et al.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      was published, we were surprised to see that a 2-point between-group difference could yield statistical significance with only 180 subjects. We contacted the authors at BMJ Rapid Response with our concern (see https://www.bmj.com/content/364/bmj.l185/rr-4), who informed us that they had used a centimeter scale (0 to 10 cm) instead of the usual millimeter scale (0 to 100 mm) (see https://www.bmj.com/content/364/bmj.l185/rr-5). Unfortunately, this is not clear in the BMJ paper.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      Furthermore, Palmer et al.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      informed us in their Rapid Response that Dwyer et al.
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      have used these incorrect values in their meta-analysis. This means that Dwyer et al.
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      unknowingly have combined different scales (0 to 10 cm and 0 to 100 mm) in their meta-analysis, mixing absolute millimeter points from 2 studies,
      • Griffin D.R.
      • Dickenson E.J.
      • Wall P.D.H.
      • et al.
      Hip arthroscopy versus best conservative care for the treatment of femoroacetabular impingement syndrome (UK FASHIoN): A multicentre randomised controlled trial.
      ,
      • Mansell N.S.
      • Rhon D.I.
      • Meyer J.
      • Slevin J.M.
      • Marchant B.G.
      Arthroscopic surgery or physical therapy for patients with femoroacetabular impingement syndrome: A randomized controlled trial with 2-year follow-up.
      with absolute centimeter points from Palmer et al.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      The correct iHOT-33 value to include in the meta-analysis from Palmer et al.
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      should have been 20.4 points (95% CI 13.2 to 27.6), not 2 points (95% CI 1.2 to 2.8) (https://www.bmj.com/content/364/bmj.l185/rr-5). This has, based on what seems to be incorrect reporting using the iHOT-33 scaling by Palmer et al. in BMJ,
      • Palmer A.J.R.
      • Ayyar Gupta V.
      • Fernquest S.
      • et al.
      Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
      led to (1) an incorrect data point in the meta-analysis
      • Dwyer T.
      • Whelan D.
      • Shah P.S.
      • Ajrawat P.
      • Hoit G.
      • Chahal J.
      Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
      and (2) an effect on the accompanying editorial commentary.
      • Kemp J.L.
      A commentary on a meta-analysis of short-term outcomes.
      We have conducted a revised meta-analysis including the correct data and scale in millimeters (Fig 1). For transparency, we present both a fixed- and random-effects model, but we prefer to use the random-effects model, since the 3 studies in the meta-analysis 1) concern very different populations (military versus general population), (2) use different eligibility criteria, and (3) lack a transparent description of the nonoperative treatment and postoperative rehabilitation.
      • Deeks J.J.
      • Higgins J.P.T.
      • Altman D.G.
      Chapter 10: Analysing data and undertaking meta-analyses.
      Figure thumbnail gr1
      Fig 1Revised meta-analyses and forest plots on the effect of operative versus nonoperative treatment of femoroacetabular impingement syndrome using a random-effects restricted maximum likelihood model (upper forest plot) and a fixed-effects inverse-variance model (lower forest plot). Positive International Hip Outcome Tool-33 (iHOT-33) scores denotes an effect in favor of operative treatment.
      Our revised analysis (random-effects model) shows a between-group difference of 11.19 points (95% CI 1.47 to 20.92) favoring operative treatment (Fig 1). We urgently encourage Arthroscopy to submit the relevant Errata for both meta-analysis and editorial commentary, alongside our letter to the editor, for transparency and to avoid any possible misunderstandings.

      Supplementary Data

      References

        • Dwyer T.
        • Whelan D.
        • Shah P.S.
        • Ajrawat P.
        • Hoit G.
        • Chahal J.
        Operative versus nonoperative treatment of femoroacetabular impingement syndrome: A meta-analysis of short-term outcomes.
        Arthroscopy. 2020; 36: 263-273
        • Palmer A.J.R.
        • Ayyar Gupta V.
        • Fernquest S.
        • et al.
        Arthroscopic hip surgery compared with physiotherapy and activity modification for the treatment of symptomatic femoroacetabular impingement: Multicentre randomised controlled trial.
        BMJ. 2019; 364: 1185
        • Griffin D.R.
        • Dickenson E.J.
        • Wall P.D.H.
        • et al.
        Hip arthroscopy versus best conservative care for the treatment of femoroacetabular impingement syndrome (UK FASHIoN): A multicentre randomised controlled trial.
        Lancet. 2018; 391: 2225-2235
        • Mansell N.S.
        • Rhon D.I.
        • Meyer J.
        • Slevin J.M.
        • Marchant B.G.
        Arthroscopic surgery or physical therapy for patients with femoroacetabular impingement syndrome: A randomized controlled trial with 2-year follow-up.
        Am J Sports Med. 2018; 46: 1306-1314
        • Mohtadi N.G.
        • Griffin D.R.
        • Pedersen M.E.
        • et al.
        The development and validation of a self-administered quality-of-life outcome measure for young, active patients with symptomatic hip disease: The International Hip Outcome Tool (iHOT-33).
        Arthroscopy. 2012; 28: 595-605
        • Kemp J.L.
        A commentary on a meta-analysis of short-term outcomes.
        Arthroscopy. 2020; 36: 274-276
        • Deeks J.J.
        • Higgins J.P.T.
        • Altman D.G.
        Chapter 10: Analysing data and undertaking meta-analyses.
        in: Higgins J.P.T. Thomas J. Chandler J. Cochrane Handbook for Systematic Reviews of Interventions, version 6.0 (updated July 2019). London, Cochrane2019