Advertisement

Sports Performance and Injury Research: Methodologic Limitations and Recommendations for Future Improvements

Published:October 09, 2020DOI:https://doi.org/10.1016/j.arthro.2020.08.038

      Abstract

      Evidence-based research has resulted in incredible advances in sports medicine and is an important component of minimizing injury risk. Such research is similarly important when applied to care delivery to athletes after injury. For research into injury reduction and treatment outcomes to be most impactful, however, the methods must be of sufficient rigor to generate high-quality evidence. Two recent trends in sports injury research have led to specific concerns about evidence quality: 1) use of athletic performance metrics as an injury or treatment outcome and 2) use of publicly available data for injury or treatment research.

      Use of Athletic Performance Metrics as an Injury or Treatment Outcome

      The impact of injury on an athlete’s ability to return to competition has been studied in professional, collegiate, and other elite athletes across a number of injuries and sports, including the National Collegiate Athletic Association (NCAA), National Basketball Association (NBA), National Football League (NFL), Major League Baseball (MLB), Major League Soccer (MLS), and National Hockey League (NHL).
      • Mohtadi N.G.
      • Chan D.S.
      Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
      • Teramoto M.
      • Cross C.L.
      • Cushman D.M.
      • Maak T.G.
      • Petron D.J.
      • Willick S.E.
      Game injuries in relation to game schedules in the National Basketball Association.
      • Erickson B.J.
      • Gupta A.K.
      • Harris J.D.
      • et al.
      Rate of return to pitching and performance after Tommy John surgery in Major League Baseball pitchers.
      • Lai C.C.H.
      • Ardern C.L.
      • Feller J.A.
      • Webster K.E.
      Eighty-three per cent of elite athletes return to preinjury sport after anterior cruciate ligament reconstruction: A systematic review with meta-analysis of return to sport rates, graft rupture rates and performance outcomes.
      Increasing focus on the impact of injury on player performance has resulted in multiple evaluations including analyses of player morphometrics, training and game participation, team schedules, and specific injuries, among others.
      • Mohtadi N.G.
      • Chan D.S.
      Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
      ,
      • Teramoto M.
      • Cross C.L.
      • Cushman D.M.
      • Maak T.G.
      • Petron D.J.
      • Willick S.E.
      Game injuries in relation to game schedules in the National Basketball Association.
      ,
      • Begly J.P.
      • Buckley P.S.
      • Utsunomiya H.
      • Briggs K.K.
      • Philippon M.J.
      Femoroacetabular impingement in professional basketball players: Return to play, career length, and performance after hip arthroscopy.
      • Begly J.P.
      • Guss M.
      • Ramme A.J.
      • Karia R.
      • Meislin R.J.
      Return to play and performance after Jones fracture in National Basketball Association athletes.
      • Teramoto M.
      • Cross C.L.
      • Rieger R.H.
      • Maak T.G.
      • Willick S.E.
      Predictive validity of National Basketball Association draft combine on future performance.
      • Yeh P.C.
      • Starkey C.
      • Lombardo S.
      • Vitti G.
      • Kharrazi F.D.
      Epidemiology of isolated meniscal injury and its effect on performance in athletes from the National Basketball Association.
      These data may be used to influence decisions on tailoring workouts, evaluating player loads, rehabilitating players after injury, and deciding timing and circumstances related to safe return to sport. However, little is known about the reliability of the use of performance data to assess postinjury outcomes and the confounding variables that may influence these findings.
      For example, Mohtadi and Chan
      • Mohtadi N.G.
      • Chan D.S.
      Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
      recently conducted a 15-article systematic review of sport-specific performance after anterior cruciate ligament (ACL) reconstruction, reporting that ACL injury and subsequent reconstruction may result in a decrease in performance. However, the authors evaluated the validity and risks of bias of each study using the Quality in Prognosis Studies (QUIPS) tool, which demonstrated multiple high-risk categories, specifically: study participation, prognostic factor measurement, study confounding measurement, statistical analysis and reporting, and overall risk of bias. Selection bias and survivor effect were specifically highlighted, as patients who may not return to sport may be missed in the study cohort. In fact, opposite conclusions were drawn between studies, including ACL injury resulting in a significant decline in performance in one study, compared with no difference from controls in another. Given these biases, the authors concluded that results from these types of studies should be interpreted cautiously.
      Optimizing player performance and mitigating injury are important to success in sport, and thus analysis of these issues is similarly important to athletes and the medical staff alike. Also of importance, however, is the accuracy of the data that are analyzed and the results and conclusions that are drawn from the data analysis. Previous evaluation of research methodology has identified 6 important factors that influence the accuracy and trustworthiness of research studies: (1) small study size; (2) small effect size; (3) greater number and lesser selection of trusted relationships; (4) increased design flexibility, definitions, outcomes, and analytical modes; (5) increased financial interest or other scientific prejudice; and (6) hotter topics in the scientific field.
      • Ioannidis J.P.
      Why most published research findings are false.
      All 6 of these factors apply to sport performance and injury research, thus highlighting the importance of optimizing data fidelity and carefully evaluating research methodologies and potential limitations in this research area.
      • Mohtadi N.G.
      • Chan D.S.
      Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
      ,
      • Teramoto M.
      • Cross C.L.
      • Cushman D.M.
      • Maak T.G.
      • Petron D.J.
      • Willick S.E.
      Game injuries in relation to game schedules in the National Basketball Association.
      ,
      • Begly J.P.
      • Buckley P.S.
      • Utsunomiya H.
      • Briggs K.K.
      • Philippon M.J.
      Femoroacetabular impingement in professional basketball players: Return to play, career length, and performance after hip arthroscopy.
      • Begly J.P.
      • Guss M.
      • Ramme A.J.
      • Karia R.
      • Meislin R.J.
      Return to play and performance after Jones fracture in National Basketball Association athletes.
      • Teramoto M.
      • Cross C.L.
      • Rieger R.H.
      • Maak T.G.
      • Willick S.E.
      Predictive validity of National Basketball Association draft combine on future performance.
      • Yeh P.C.
      • Starkey C.
      • Lombardo S.
      • Vitti G.
      • Kharrazi F.D.
      Epidemiology of isolated meniscal injury and its effect on performance in athletes from the National Basketball Association.
      ,
      • Mack C.D.
      • Herzog M.M.
      • DiFiori J.P.
      • Meisel P.L.
      • Dreyer N.A.
      A second look at NBA game schedules: Response to Teramoto et al.
      First, the elite nature of collegiate, professional, and other elite sports by definition typically results in a relatively small number of athletes with diverse injury types. Second, effect size of a specific training regimen or injury on a single performance metric such as a player efficiency rating in basketball (a measure of an individual player’s impact in terms of points per minute or possession) may be extremely small in isolation and difficult to accurately isolate from the multitude of training strategies that are often used in combination. Moreover, these types of metrics are confounded by the variability of team composition, match-ups, competition, and coaching decisions. Third, the relatively large number of metrics and their relationship to specific player performance and efficiency variables is significant. Fourth, design flexibility and complex analytics have been used in this field, such as cross-referencing potentially unrelated datasets (or data obtained without quality assurance, such as publicly available data) with single-surgeon cohort data to assess postoperative sports performance.
      • Begly J.P.
      • Buckley P.S.
      • Utsunomiya H.
      • Briggs K.K.
      • Philippon M.J.
      Femoroacetabular impingement in professional basketball players: Return to play, career length, and performance after hip arthroscopy.
      Fifth, there are clear financial implications for these metrics, especially in the setting of professional sports, where multimillion-dollar contracts may hang in the balance. Moreover, there may well be a lack of impartiality depending on the relationships and positions of those gathering the data.
      Lastly, the high-profile nature of collegiate, professional, and elite sports, including media coverage, sports betting, and fan support, places research into sports performance and injury in the crosshairs of “hot topic” research, resulting in an increased number of research teams and subsequent publications. Given these factors, there is a relatively higher risk that the conclusions of research in this area will be erroneous owing to the Proteus phenomenon, in which alternating extreme research claims and refutations frequently surrounds areas of high scientific and public interest, or hot topics.
      • Ioannidis J.P.
      Why most published research findings are false.
      For this reason, and in particular for those studies that use athletic performance metrics to evaluate injury or treatment outcomes, it is important for researchers, medical personnel, journal reviewers, and journal editors to critically evaluate the data quality and research methodologies for studies in this field to ensure that all limitations that may result in misleading findings are minimized and clearly described.

      Use of Publicly Available Sports Data for Injury and Treatment Research

      Related to the influx of studies evaluating performance outcomes after injury or treatment, multiple studies in the existing literature use publicly available health and athletic metrics to study sports injuries and treatment outcomes.
      • Mohtadi N.G.
      • Chan D.S.
      Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
      ,
      • Teramoto M.
      • Cross C.L.
      • Cushman D.M.
      • Maak T.G.
      • Petron D.J.
      • Willick S.E.
      Game injuries in relation to game schedules in the National Basketball Association.
      ,
      • Begly J.P.
      • Buckley P.S.
      • Utsunomiya H.
      • Briggs K.K.
      • Philippon M.J.
      Femoroacetabular impingement in professional basketball players: Return to play, career length, and performance after hip arthroscopy.
      • Begly J.P.
      • Guss M.
      • Ramme A.J.
      • Karia R.
      • Meislin R.J.
      Return to play and performance after Jones fracture in National Basketball Association athletes.
      • Teramoto M.
      • Cross C.L.
      • Rieger R.H.
      • Maak T.G.
      • Willick S.E.
      Predictive validity of National Basketball Association draft combine on future performance.
      • Yeh P.C.
      • Starkey C.
      • Lombardo S.
      • Vitti G.
      • Kharrazi F.D.
      Epidemiology of isolated meniscal injury and its effect on performance in athletes from the National Basketball Association.
      These studies frequently discuss limitations such as absence of injury specifics, inability to evaluate associated injuries that are frequently not included in public reports, unmeasured confounding variables, absence of data regarding surgical technique and intraoperative findings, rehabilitation protocols, and other medical history. Furthermore, player performance is affected by an array of nonmedical factors, including coaching decisions, team personnel changes, trade decisions, and in-season timing. It is clear that the quality of research results and conclusions are only as good as the quality of the data being analyzed. Careful construction and quality review of data sources is important in both clinical and sports settings.
      • Mack C.D.
      • Meisel P.
      • Herzog M.M.
      • et al.
      The establishment and refinement of the National Basketball Association Player Injury and Illness Database.
      Although the above study design limitations may be described in the limitations section of an article, in some cases, incomplete data or significant confounding variables may represent fatal methodologic flaws. In these cases, discussing these flaws in the limitations section may not be sufficient to mitigate the effect on the results. In these circumstances, careful reconsideration of the study design and improvement of the data source should be considered before publication. Peer review is crucial in highlighting these issues, and in some cases, publication may be declined to ensure that these inaccurate or incomplete results are not added to the scientific literature.
      The use of publicly available sources for sports injury and treatment research is of concern because of the potential for biased conclusions. In particular, there is significant potential for selection bias when identifying and reporting more-severe injuries. Less-severe injuries may be underreported or deemphasized in the public sphere, thereby overstating results, including time lost from sport participation or proportion of injuries requiring surgery. This selection bias could inadvertently lead to the conclusion that an injury has a greater impact on performance and return to sport owing to the lack of inclusion of less severe injuries of the same type. Data may be particularly susceptible to changes in quality over long time horizons, with limited ability for researchers to understand changes in data collection or reporting over time.
      More broadly, publicly available data undergo insufficient standardization, data cleaning, quality management, or assessment for data integrity and completeness that would otherwise typically accompany a curated injury surveillance and outcomes database.
      • Mack C.D.
      • Herzog M.M.
      • DiFiori J.P.
      • Meisel P.L.
      • Dreyer N.A.
      A second look at NBA game schedules: Response to Teramoto et al.
      Public reports also often lack sufficient detail related to confounding variables that may affect outcomes and conclusions. For example, details pertaining to concomitant injuries are often not provided, and details related to the timing and specificity of treatment may be limited. Moreover, accurate denominators, such as athlete exposure, including total number of games or minutes played before and after injury, are frequently missing. These factors may significantly affect any conclusions related to the timing of return to play, risk of reinjury, and the performance of the athlete once activated.

      Recommendations

      Ultimately, mitigating injury and optimizing player performance are 2 important aims of the science of sport. To achieve these goals, evidence-based, data-driven practices should be implemented whenever practicable. For this reason, the National Basketball Association (NBA), like many other professional and elite sport organizations, has created and supported research in the form of grants, collaborative committees, and open dialogue aimed at developing recommended practice guidelines on data acquisition, research study design, statistical analysis, and data utilization that exceed the current standards in sport research. The NBA Research Committee, in collaboration with the National Basketball Players Association (NBPA) and NBA league office, have also identified specific recommended practices for research methodology to guide future research in sport:
      • 1.
        Sample size. Although retrospective descriptive studies are difficult to power, sample sizes should be large enough to allow meaningful conclusions and comprehensive injury descriptions relevant to the specific research question.
      • 2.
        Injuries/reinjuries. The use of injuries or reinjuries as an outcome is subject to multiple confounding variables that are difficult to evaluate. For example, when considering risk factors for a lower-extremity injury, the player’s history of prior lower-extremity injury is relevant and one of several confounding variables that impact this outcome. This type of analysis should be conducted when preinjury data exist and are verified as accurate. Publicly available data such as “prior history of foot fracture” is insufficient, as the specifics of these injuries may not be available, and thus analysis and conclusions derived from analysis of foot fracture data may be false.
      • 3.
        Return to play. Return-to-play statistics, including time out before return and minutes played upon return, among others, are vulnerable to many factors that are not related to injury. Some of these confounding factors include the timing of injury in the season (late-season injuries will miss fewer games) and coaching decisions that are team or personnel related and not necessarily injury related.
      • 4.
        Games and performance metrics. Use of game and performance metrics/statistics (such as minutes per game, rebounds, or player efficiency rating) as an outcome measure may be impacted by multiple confounding variables that may not be related to the health of the player or efficacy of a treatment modality (e.g., coach’s decision, team record).
      • 5.
        Rehabilitation, treatments, and surgeries. The inclusion of data regarding surgical and nonsurgical treatment and rehabilitation metrics of interest can be subject to variations in protocols among teams (or treatment sites) that may significantly affect any outcome comparisons. Research regarding surgical or nonsurgical treatments and rehabilitation metrics should optimize variable homogeneity and minimize broad analyses. For example, “hip injuries and return to sport” as the specifics of the hip injury and treatment significantly affects any other data that may be evaluated. Studies in which limitations include “data regarding concomitant injuries were not available” should be avoided, as such concomitant injuries may be insurmountable confounding variables. Examples include ACL reconstruction and impact on return to play or sports performance without information regarding other ligament, articular cartilage, or meniscal injuries, all of which have been previously shown to significantly affect outcomes.
      It is our hope that this letter will raise awareness that some of the current research assessing the effects of injury on return to sport and sport performance are potentially misleading owing to the many confounders described above. The purpose of this letter is to identify a growing concern regarding studies that make use of athletic performance metrics as an injury or treatment outcome or publicly available data for injury or treatment research, not to critique those investigators who have previously published studies using these methodologies (which includes some of the authors of this letter). We propose that implementation of the practices listed, which we plan to follow and believe other investigators and journals should as well, will lead to higher-quality evidence for optimizing player performance and preventing and mitigating injuries.
      The impetus for this editorial is our shared belief that the research community focused on sports medicine and injury bears the responsibility to consider these research methodologies and the potential negative downstream effect of flawed data and conclusions. Specifically, published data that inaccurately suggest a negative predictive effect of an injury on sports performance may have a significant impact on an athlete’s pursuit of their sport due to a sporting club’s fear of an investment of time, effort, or finances into the previously injured athlete. Although this downstream, unintended effect of research data may occur regardless of our efforts, at the very least it is our responsibility to minimize published research based on flawed data or methodology.
      Ultimately, this will allow athletes to train and compete at the highest levels while mitigating injury risk in an accurate, evidence-based fashion and minimizing unintended downstream effects based on flawed data that may negatively affect the athlete. Moreover, these suggested research standards will ideally assist in promoting a continued trust between athletes, their medical and sports performance staff, and the research community that is fundamental to optimal care delivery and safe sport participation.

      Supplementary Data

      References

        • Mohtadi N.G.
        • Chan D.S.
        Return to sport-specific performance after primary anterior cruciate ligament reconstruction: A systematic review.
        Am J Sports Med. 2018; 46: 3307-3316
        • Teramoto M.
        • Cross C.L.
        • Cushman D.M.
        • Maak T.G.
        • Petron D.J.
        • Willick S.E.
        Game injuries in relation to game schedules in the National Basketball Association.
        J Sci Med Sport. 2017; 20: 230-235
        • Erickson B.J.
        • Gupta A.K.
        • Harris J.D.
        • et al.
        Rate of return to pitching and performance after Tommy John surgery in Major League Baseball pitchers.
        Am J Sports Med. 2014; 42: 536-543
        • Lai C.C.H.
        • Ardern C.L.
        • Feller J.A.
        • Webster K.E.
        Eighty-three per cent of elite athletes return to preinjury sport after anterior cruciate ligament reconstruction: A systematic review with meta-analysis of return to sport rates, graft rupture rates and performance outcomes.
        Br J Sports Med. 2018; 52: 128-138
        • Begly J.P.
        • Buckley P.S.
        • Utsunomiya H.
        • Briggs K.K.
        • Philippon M.J.
        Femoroacetabular impingement in professional basketball players: Return to play, career length, and performance after hip arthroscopy.
        Am J Sports Med. 2018; 46: 3090-3096
        • Begly J.P.
        • Guss M.
        • Ramme A.J.
        • Karia R.
        • Meislin R.J.
        Return to play and performance after Jones fracture in National Basketball Association athletes.
        Sports Health. 2016; 8: 342-346
        • Teramoto M.
        • Cross C.L.
        • Rieger R.H.
        • Maak T.G.
        • Willick S.E.
        Predictive validity of National Basketball Association draft combine on future performance.
        J Strength Cond Res. 2018; 32: 396-408
        • Yeh P.C.
        • Starkey C.
        • Lombardo S.
        • Vitti G.
        • Kharrazi F.D.
        Epidemiology of isolated meniscal injury and its effect on performance in athletes from the National Basketball Association.
        Am J Sports Med. 2012; 40: 589-594
        • Ioannidis J.P.
        Why most published research findings are false.
        PLoS Med. 2005; 2: e124
        • Mack C.D.
        • Herzog M.M.
        • DiFiori J.P.
        • Meisel P.L.
        • Dreyer N.A.
        A second look at NBA game schedules: Response to Teramoto et al.
        J Sci Med Sport. 2018; 21: 228-229
        • Mack C.D.
        • Meisel P.
        • Herzog M.M.
        • et al.
        The establishment and refinement of the National Basketball Association Player Injury and Illness Database.
        J Athl Train. 2019; 54: 466-471