Advertisement

Understanding Network Meta-analysis (NMA) Conclusions Requires Scrutiny of Methods and Results: Introduction to NMA and the Geometry of Evidence

      Abstract

      Synthesis of medical literature to determine the best treatment for a given problem is challenging, particularly when multiple options exist. Network meta-analysis (NMA) allows the comparison of different treatment approaches in a single, systematic review including treatments that have never been compared head-to-head. A key to understanding NMA is to focus on the network geometry showing the number of included studies and their relationships: different treatment options are illustrated as nodes. Lines between nodes represent direct comparisons. For nodes not directly compared, indirect effects may be determined by use of the property of transitivity. Limitations of NMA include heterogeneity, where variability among included studies biases pairwise comparisons, and consistency, if direct and indirect comparisons between treatments do not agree. In the end, NMA allows numeric ranking of the estimated effects of each treatment from most to least effective. A disadvantage of NMA ranking methods is that readers may focus overly on what treatment ranks best and focus insufficiently on the methods and results that determine the rankings. The reliability of the rankings requires consideration of the geometry and strength of the network, including evaluation of heterogeneity, consistency, and transitivity. The conclusion of an NMA requires scrutiny of the methods and results.
      Comparative clinical trials comprise a small portion of the medical literature compared with case series.
      • Harris J.D.
      • Cvetanovich G.
      • Erickson B.J.
      • et al.
      Current status of evidence-based sports medicine.
      However, in areas of clinical controversy in which numerous comparative trials exist, network meta-analysis (NMA) methods allow us to reach wider conclusions by allowing us to compare different treatment approaches in a single review.
      • Cote M.P.
      Editorial commentary: Network geometry of nonoperative management of patellar tendinopathy—Can the shape of the evidence inform practice?.
      Remarkably, as described herein, NMA methods allow comparison of treatments “that have never been compared head-to-head.”
      • Wang Y.
      • Lei G.
      • Zeng C.
      • et al.
      Comparative risk-benefit profiles of individual devices for graft fixation in anterior cruciate ligament reconstruction: A systematic review and network meta-analysis.
      An excellent example can be found in the current issue, where Forsythe, Lavoie-Gagne, Patel, Lu, Ritz, Chahla, Okoroha, Allen, and Nwachukwu synthesize Level I evidence regarding the “Efficacy of Arthroscopic Surgery in the Management of Adhesive Capsulitis: A Systematic Review and Network Meta-analysis of Randomized Controlled Trials.”
      • Forsythe B.
      • Lavoie-Gagne O.
      • Patel B.H.
      • et al.
      Efficacy of arthroscopic surgery in the management of adhesive capsulitis: A systematic review and network meta-analysis of randomized controlled trials.

      To Understand NMA, Focus on the Geometry

      An excellent start to understanding an NMA is to focus on the graphically illustrated geometry of the “network” of included studies and their relationships. As can be seen in a simple example, illustrated in Figure 1, different treatment options (A, B, and C) are represented as “nodes.” Lines between nodes represent “direct” comparisons (i.e., studies in which different treatments were directly compared). The thickness of the lines represents the number of included clinical trials comparing the different treatments; this is also indicated numerically (4 and 9). A thicker line indicates that more trials (9) compared treatment B with treatment C, compared with the thinner line, where only 4 different trials compared treatment A with C.
      Figure thumbnail gr1
      Fig 1NMA network geometry. Treatment options (A, B, and C) are represented as “nodes.” Lines between nodes represent “direct” comparisons (i.e., studies in which different treatments are directly compared). The thickness of the lines represents the number of included clinical trials comparing the different treatments; this is also indicated numerically (4 and 9). A thicker line indicates that more trials (9) compared treatment B with C, as compared to the thinner line where only 4 different trials compared treatment A with C. Note that A and B are not directly compared.
      The greater number of internodal connections, the stronger or more well-connected the network. A strong network includes ample connections between nodes. For example, the network indicated in Figure 1 would be stronger or more well-connected if there were 1 or more direct connections between A and B, which is not the case. Similarly, a strong network includes more comparisons between treatments, or thicker lines. Here, a sense of the potential of an NMA to deliver valid results can be gleaned. Well-connected networks indicate a diversity of direct treatment comparisons, resulting in greater potential validity. Less well-connected networks indicate a relatively lower number of head-to-head comparisons (i.e., more “indirect” comparisons) resulting in lower potential validity. Quite simply, in an ideal setting, an NMA geometry would show well-connected nodes held together by strong thick lines. As a network becomes less connected, the analysis becomes more reliant on indirect comparisons and a more cautious interpretation of the results is required.
      • Mills E.J.
      • Thorlund K.
      • Ioannidis J.P.
      Demystifying trial networks and network meta-analysis.
      ,
      • Rouse B.
      • Chaimani A.
      • Tianjing L.
      Network meta-analysis: An introduction for clinicians.

      To Interpret NMA, Consider Heterogeneity and Consistency

      All systematic reviews of the literature (of which meta-analyses and NMA are subsets) result in articles being included or excluded according to “systematic” search methods. Once the included articles are identified, a vital step is to determine the heterogeneity (as opposed to agreement) among studies that are compared head-to-head. Typically, and unfortunately, even studies meeting strict inclusion criteria may be quite heterogeneous due to diversity in (just to mention a few examples of areas of variability) patient demographics, treatments, outcome measures, or time to follow-up. A clear example of heterogeneity of treatment is if a treatment were defined as “nonoperative” but if nonoperative treatment was immobilization in some studies and physical therapy in others. Both treatments are “nonoperative,” but obviously, the treatments are different or heterogeneous.

      Heterogeneity

      In NMA, heterogeneity applies specifically to pairwise comparisons that are directly compared (i.e., node to node such as A with C, or B with C, in Fig 1). Statistical heterogeneity can be quantified and the extent of clinical heterogeneity requires scrutiny (by authors, editors, and readers) when study data are pooled.
      • Cote M.P.
      • Apostolakos J.M.
      • Voss A.
      • DiVenere J.
      • Arciero R.A.
      • Mazzocca A.D.
      A systematic review of meta-analyses published in Arthroscopy: The Journal of Arthroscopic and Related Surgery.
      ,
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 3: Pearls and pitfalls of meta-analyses and systematic reviews.
      Heterogeneity in an NMA introduces bias, may limit validity of the results and conclusions, and represents a primary, potential limitation of NMA.
      • Zamborsky R.
      • Danisovic L.
      Surgical techniques for knee cartilage repair: An updated large-scale systematic review and network meta-analysis of randomized controlled trials.
      • Zhao Z.
      • Ma J.-X.
      • Ma Y.-X.
      Different intra-articular injections as therapy for hip osteoarthritis: A systematic review and network meta-analysis.
      • Chen P.-C.
      • Wu K.-T.
      • Chou W.-Y.
      • et al.
      Comparative effectiveness of different nonsurgical treatments for patellar tendinopathy: A systematic review and network meta-analysis.

      Consistency

      The term consistency is also used to describe agreement, or lack thereof, among treatment comparisons. However, when it comes to NMA, consistency, like heterogeneity, is defined quite specifically.
      • Salanti G.
      Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: Many names, many benefits, many concerns for the next generation evidence synthesis tool.
      Consistency in an NMA refers to agreement between direct and indirect comparisons, noting that determination of indirect effects is dependent on the property of transitivity (reviewed below). Like heterogeneity, consistency can be quantified and requires scrutiny (by authors, editors, and readers) when study data are pooled.
      The NMA network geometry illustrated in Figure 2 aids in understanding consistency. Again, treatment options (A, B, and C) are represented as “nodes,” lines between nodes represent “direct” comparisons, and the thickness of the lines represents the number of included clinical trials directly comparing the different treatments; this is also indicated numerically (2, 4, and 9). Next, for example, note that A and B can be compared directly (by synthesizing the 2 head-to-head trials) or indirectly (by comparing how treatment A compares with treatment C vs how treatment B compares with the shared node C). It follows that the indirect effect should be similar to the direct effect, and the extent to which the direct and indirect effect estimates agree describes the degree of “consistency” in a treatment network.
      Figure thumbnail gr2
      Fig 2NMA network geometry. Treatment options (A, B, and C) are represented as “nodes.” Lines between nodes represent “direct” comparisons (i.e., studies in which different treatments are directly compared). The thickness of the lines represents the number of included clinical trials comparing the different treatments; this is also indicated numerically (2, 4, and 9). A thicker line indicates that more trials (9) compared treatment B with C, as compared to the thinner lines where only 2 (A with B) or 4 (A with C) trials were compared. Note that A versus B, for example, can be compared directly (by synthesizing the 2 head-to-head trials) and indirectly (by comparing how treatment A compares with the shared node C, versus how treatment B compares with the shared node C). It follows that the indirect effect should be similar to the direct effect. The extent to which the direct and indirect effect estimates agree describes the degree of “consistency” in a treatment network.

      The Hallmark of NMA is Transitivity

      Recall the transitive property of equality: if A = C and B = C, then A = B.
      • Simmons B.
      Transitive property of equality. Mathworks.
      In NMA, and as best illustrated in Figure 1, if 2 treatments (A and B) have not been directly compared, but if both have been compared to a third (or control) group (C), then by relying on the assumption of transitivity, the 2 treatments can be indirectly compared. And as mentioned previously, a hallmark of NMA methods is the ability to derive estimates for treatment comparisons that have not been compared head-to-head. In addition, for an indirect comparison to hold, the common comparator should be consistent across trials.
      The assumption of transitivity is reliant on uniformity among treatments, and among baseline risks and effect modifiers.
      • Salanti G.
      Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: Many names, many benefits, many concerns for the next generation evidence synthesis tool.
      For example, studies reporting identical treatments may show different results because of different inclusion and exclusion criteria, resulting in different expected outcomes among diverse groups of patients (selection bias), or because of varying surgical expertise among the authors of the different studies (performance bias). As nodes become less transitive, NMA results and conclusions become biased and less reliable.

      The Conclusion of an NMA Requires Careful Scrutiny

      In the end, NMA methods allow quantitative synthesis of different studies investigating different treatments, and, ultimately, determination and reporting of a numeric ranking of the estimated effects of each treatment from most to least effective. The obvious goal is to determine which treatment has the greatest probability of ranking number one. Readers may be tempted to skip right to the conclusion to find out “the winner,” but rankings require consideration of the strength of the network including heterogeneity and consistency.
      If readers were to take home a single point from this editorial, we recommend that: A disadvantage of NMA ranking methods is that readers may focus overly on what treatment ranked best and focus insufficiently on the methods and results that determine the rankings.

      NMA on the Management of Adhesive Capsulitis

      In this issue’s NMA on the management of adhesive capsulitis by Forsythe et al.,
      • Forsythe B.
      • Lavoie-Gagne O.
      • Patel B.H.
      • et al.
      Efficacy of arthroscopic surgery in the management of adhesive capsulitis: A systematic review and network meta-analysis of randomized controlled trials.
      a variety of treatments are included. The networks for range of motion and pain are well connected in many areas, and physical therapy and injection are the 2 most represented treatments. Several other treatments, however, are connected by thin lines, indicating a smaller number of trials contributing to these other treatments’ representation in the included data. As treatments supported by thin lines are under-represented, conclusions regarding these other treatments are less reliable, mandating a close examination of the effect of these treatments in the results section of the report.
      The assumption of transitivity is well-addressed. The authors provide detailed information on the similarity of the treatments and patients that constitute the shared nodes in support of a transitive network and, here, more reliable conclusions. In addition, the network harbors a number of closed loops, indicating treatments that are informed by both direct and indirect evidence. Moreover, direct and indirect effects are similar, indicating relative consistency among the comparisons. Thus, the network presented shows significant strength.

      Summary

      NMA allows comparison of different treatment approaches in a single, systematic review including treatments that have never been compared head-to-head. NMA allows numeric ranking of the effectiveness of treatments, and it can be tempting to skip right to the conclusion to find out of the winner. However, NMAs provide information well beyond which treatment ranks the best. The reliability of the rankings requires consideration of the geometry and strength of the network, including evaluation of heterogeneity, consistency, and transitivity. The conclusion of an NMA requires scholarly scrutiny and nuanced consideration of the methods and results.

      References

        • Harris J.D.
        • Cvetanovich G.
        • Erickson B.J.
        • et al.
        Current status of evidence-based sports medicine.
        Arthroscopy. 2014; 30: 362-371
        • Cote M.P.
        Editorial commentary: Network geometry of nonoperative management of patellar tendinopathy—Can the shape of the evidence inform practice?.
        Arthroscopy. 2019; 35: 3132-3134
        • Wang Y.
        • Lei G.
        • Zeng C.
        • et al.
        Comparative risk-benefit profiles of individual devices for graft fixation in anterior cruciate ligament reconstruction: A systematic review and network meta-analysis.
        Arthroscopy. 2020; 36: 1953-1972
        • Forsythe B.
        • Lavoie-Gagne O.
        • Patel B.H.
        • et al.
        Efficacy of arthroscopic surgery in the management of adhesive capsulitis: A systematic review and network meta-analysis of randomized controlled trials.
        Arthroscopy. 2021; 37: 2281-2297
        • Mills E.J.
        • Thorlund K.
        • Ioannidis J.P.
        Demystifying trial networks and network meta-analysis.
        BMJ. 2013; 346: f2914
        • Rouse B.
        • Chaimani A.
        • Tianjing L.
        Network meta-analysis: An introduction for clinicians.
        Intern Emerg Med. 2017; 12: 103-111
        • Cote M.P.
        • Apostolakos J.M.
        • Voss A.
        • DiVenere J.
        • Arciero R.A.
        • Mazzocca A.D.
        A systematic review of meta-analyses published in Arthroscopy: The Journal of Arthroscopic and Related Surgery.
        Arthroscopy. 2016; 32: 528-537
        • Harris J.D.
        • Brand J.C.
        • Cote M.P.
        • Dhawan A.
        Research pearls: The significance of statistics and perils of pooling. Part 3: Pearls and pitfalls of meta-analyses and systematic reviews.
        Arthroscopy. 2017; 33: 1594-1602
        • Zamborsky R.
        • Danisovic L.
        Surgical techniques for knee cartilage repair: An updated large-scale systematic review and network meta-analysis of randomized controlled trials.
        Arthroscopy. 2020; 36: 845-858
        • Zhao Z.
        • Ma J.-X.
        • Ma Y.-X.
        Different intra-articular injections as therapy for hip osteoarthritis: A systematic review and network meta-analysis.
        Arthroscopy. 2020; 36: 1452-1464
        • Chen P.-C.
        • Wu K.-T.
        • Chou W.-Y.
        • et al.
        Comparative effectiveness of different nonsurgical treatments for patellar tendinopathy: A systematic review and network meta-analysis.
        Arthroscopy. 2019; 35: 3117-3131
        • Salanti G.
        Indirect and mixed-treatment comparison, network, or multiple-treatments meta-analysis: Many names, many benefits, many concerns for the next generation evidence synthesis tool.
        Res Synth Methods. 2012; 3: 80-97
        • Simmons B.
        Transitive property of equality. Mathworks.