Advertisement

Expert Panels: Can They Be Trusted?

      To the Editor:
      Van Meer et al.
      • van Meer B.L.
      • Meuffels D.E.
      • Vissers M.M.
      • et al.
      Knee Injury and Osteoarthritis Outcome Score or International Knee Documentation Committee Subjective Knee Form: Which questionnaire is most useful to monitor patients with an anterior cruciate ligament rupture in the short term?.
      recently compared the Knee Injury and Osteoarthritis Outcome Score (KOOS)
      • Roos E.M.
      • Roos H.P.
      • Lohmander L.S.
      • et al.
      Knee Injury and Osteoarthritis Outcome Score (KOOS)—Development of a self-administered outcome measure.
      with the International Knee Documentation Committee (IKDC) Subjective Knee Form
      • Irrgang J.J.
      • Anderson A.F.
      • Boland A.L.
      • et al.
      Development and validation of the International Knee Documentation Committee Subjective Knee Form.
      to investigate which of the instruments is most useful after anterior cruciate ligament reconstruction (ACLR). Following the COSMIN (COnsensus-based Standards for the selection of health status Measurement INstruments) checklist, an expert panel reached consensus on a priori hypotheses regarding responsiveness and construct validity. This letter raises concerns about the use of expert panels in general and gives examples of bias introduced by the expert panel in this particular study.
      van Meer et al. state: “Without specific hypotheses there is a risk of bias, because it is tempting to formulate explanations for the low and high correlation coefficients retrospectively instead of concluding that the questionnaire may not be valid. Conversely, the choice of magnitude of the hypotheses is arbitrary.” If the expert panel had considered available literature when establishing a priori hypotheses, their results would have been different. Using an expert panel is thus not necessarily associated with less bias compared with relying on available literature. A minimum requirement for an expert panel must be awareness of existing literature when formulating a priori hypotheses.
      Especially evident for KOOS, correlations hypothesized are not in line with existing literature. In contrast, the majority of the correlations actually found for both KOOS and IKDC are in line with existing literature. One example relates to effect size (ES). For the IKDC, van Meer et al. hypothesized an ES >0.8, found an ES of 1.36, and the hypothesis was confirmed, well in line with existing literature (ES, 0.76 to 2.11).
      • Collins N.J.
      • Misra D.
      • Felson D.T.
      • et al.
      Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS).
      For KOOS, however, it seems that the predefined hypotheses were arbitrary. For the subscales Pain, activities of daily living (ADL), and quality of life, ESs less than 0.4 were hypothesized. The ESs found (0.58 to 1.51) are, however, in line with existing literature (ACLR, 0.84 to 1.65).
      • Roos E.M.
      • Roos H.P.
      • Lohmander L.S.
      • et al.
      Knee Injury and Osteoarthritis Outcome Score (KOOS)—Development of a self-administered outcome measure.
      • Collins N.J.
      • Misra D.
      • Felson D.T.
      • et al.
      Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS).
      Expert clinicians and patients evaluated content validity. This approach may introduce further bias because the perspectives of clinicians and patients differ and current standards consider the patient to be the expert. The importance of focusing on the patient's perspective is underscored by the findings in the van Meer study. Although the authors concluded that the KOOS subscales Pain and ADL were irrelevant, they reported ES > 0.50. A moderate improvement in Pain and ADL at 1 year after ACLR is not irrelevant.
      Van Meer et al. state: “A disadvantage of the IKDC subjective is the use of one total score, which means it is impossible to see in which domain the patients have improved. According to our results, the IKDC subjective is more responsive to changes over time than is the KOOS.” This statement is incorrect. The ES for the different KOOS subscales ranged from 0.55 to 1.51, reflecting the difference in improvement for the 5 domains at 1 year after ACLR, whereas the ES for the IKDC subjective was 1.36.
      Finally, in Table 2 the correlation found between the IKDC and Lysholm score (0.47) is wrongly interpreted as being higher than the hypothesized correlation of 0.6. Overall, 75% of the hypothesized and the found correlations are not given in the tables and can thus neither be compared with the previous literature nor checked for correctness.

      References

        • van Meer B.L.
        • Meuffels D.E.
        • Vissers M.M.
        • et al.
        Knee Injury and Osteoarthritis Outcome Score or International Knee Documentation Committee Subjective Knee Form: Which questionnaire is most useful to monitor patients with an anterior cruciate ligament rupture in the short term?.
        Arthroscopy. 2013; 29: 701-715
        • Roos E.M.
        • Roos H.P.
        • Lohmander L.S.
        • et al.
        Knee Injury and Osteoarthritis Outcome Score (KOOS)—Development of a self-administered outcome measure.
        J Orthop Sports Phys Ther. 1998; 28: 88-96
        • Irrgang J.J.
        • Anderson A.F.
        • Boland A.L.
        • et al.
        Development and validation of the International Knee Documentation Committee Subjective Knee Form.
        Am J Sports Med. 2001; 29: 600-613
        • Collins N.J.
        • Misra D.
        • Felson D.T.
        • et al.
        Measures of knee function: International Knee Documentation Committee (IKDC) Subjective Knee Evaluation Form, Knee Injury and Osteoarthritis Outcome Score (KOOS), Knee Injury and Osteoarthritis Outcome Score Physical Function Short Form (KOOS-PS), Knee Outcome Survey Activities of Daily Living Scale (KOS-ADL), Lysholm Knee Scoring Scale, Oxford Knee Score (OKS), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), Activity Rating Scale (ARS), and Tegner Activity Score (TAS).
        Arthritis Care Res (Hoboken). 2011; 63: S208-S228

      Linked Article