Advertisement

Our Measure of Medical Research Should Be Appreciable Benefit to the Patient

      Abstract

      The clinical relevance of research is much more important than statistical significance. Patient-reported outcome measures (PROM) are the strongest determinants of satisfaction as the result of an intervention or treatment. Outcomes can be measured in terms of the minimal clinically important difference (MCID) detectable by a patient, bearing in mind that “detectable” includes worsening as well as improvement. Patient-acceptable symptomatic state (PASS) and substantial clinical benefit (SCB) ultimately correlate with whether patients are happy or willing to undergo an intervention again. These measures should not be reported in terms of group means, where outliers could distort the score. These measures should be reported in terms of the proportion of individual patients whose scores cross the meaningful thresholds of satisfaction.
      Some perform research most of the time; most of our readers have performed research at some time. Regardless, and without doubt, most readers are interested in benefit to patients all of the time. When it comes to medical research, it's the patient that matters … even when it comes to biostatistics.

      Tools for Authors

      In 2017 and 2018, Arthroscopy journal's team of Editors and Associate Editors directed great energy toward preparation and publication of “Tools to Improve Scientific Research”
      • Rossi M.
      • Brand J.
      • Lubowitz J.
      Tools to improve scientific research.
      including Research Pearls,
      • Dhawan A.
      • Brand J.C.
      • Provencher M.T.
      • Rossi M.J.
      • Lubowitz J.H.
      Research pearls: The significance of statistics and perils of pooling.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      • Hohmann E.
      • Wetzler M.J.
      • D’Agostino Jr., R.B.
      Research pearls: The significance of statistics and perils of pooling. Part 2: Predictive modeling.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 3: Pearls and pitfalls of meta-analyses and systematic reviews.
      • Kartus J.
      • Cote M.
      Invention versus gold standard: A hands-on research pearl on study design and statistical concerns.
      • Hohmann E.
      • Cote M.
      • Brand J.
      Research pearls: Expert consensus based evidence using the Delphi method.
      • Hohmann E.
      • Feldman M.
      • Hunt T.J.
      • Cote M.
      • Brand J.C.
      Research pearls: How do we establish the level of evidence?.
      Checklists and Templates (accessible at www.arthroscopyjournal.org under a dropdown menu), and an Editorial
      • Cote M.P.
      • Lubowitz J.H.
      • Rossi M.J.
      • Brand J.C.
      Reviews pooling heterogeneous, low-evidence, high-bias data result in incorrect conclusions: But heterogeneity is an opportunity to explore.
      intended to stamp out improper pooling of data otherwise resulting in incorrect conclusions in systematic reviews of low level of evidence, high risk of bias, heterogeneous literature.
      For authors seeking successful acceptance of their research submissions under our journal peer-review process, meticulous review of our Instructions for Authors, Checklists and Templates, Editorials, and Pearls is essential.

      Tools for Readers

      Tools for authors, in combination with peer review, facilitate improving, rather than merely refereeing, the articles we accept. The aim is to ensure the reliability of conclusions of the studies we publish. In addition, we have today identified an opportunity to help readers better discern the real “significance” of our content: benefit to patients. Our take-home message is a “statistically significant difference may not be clinically relevant.”
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.

      What Does This Mean?

      If we compare groups of patients enough, we will eventually detect a statistically significant difference (i.e., P < .05) in the result of an intervention.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      In addition, “small changes in the numbers of outcomes,” often “less than the number of patients lost to follow-up,” could change the results of a clinical trial from statistically significant to nonsignificant, an epidemiologic concept known as “fragility.”
      • Walsh M.
      • Srinathan S.
      • McAuley D.
      • et al.
      The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index.
      Most of all, patients may not even be able to detect differences calculated to be of statistical significance. Thus, other measures must determine whether an intervention results in a benefit to patients.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      Historically, arthroscopic and related outcome reporting focused on surgeon-recorded measures (like range of motion), and we termed these measures “objective,” suggesting they were unbiased. On the contrary, patient-reported outcome measures (PROM) were termed “subjective,” connotating bias, idiosyncrasy, or diminished seriousness. How wrong we were. Today we acknowledge that PROM are the sturdiest measures of patient satisfaction. The logical next step is to consider whether patients are clinically satisfied with their treatment. To put it bluntly, patients don't care about their P value, which “may or may not reflect a clinically meaningful change.”
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.

      How Do We Determine Benefit to the Patient?

      To start, outcomes could be measured in terms of the minimal clinically important difference (MCID) detectable by a patient. However, while MCID and related measures evaluate changes “detectable” by a patient (including either improvement or worsening), more important measures of clinical relevance could be thresholds for determining patient satisfaction. Readers could be on the lookout for reporting of patient-acceptable symptomatic state (PASS) and substantial clinical benefit (SCB). PASS and SCB ultimately correlate with whether patients are “satisfied” or “happy” or “willing to undergo the intervention again.”
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      Finally, readers should be aware that these measures (MCID, PASS, SCB) should not be reported in terms of group means or average values because a few outliers could distort the score for an entire group. When it comes to satisfaction, each patient is either satisfied or not. Thus, SCB, for example, should be reported in terms of the proportion of patients within the group who either did or did not experience a substantial clinical benefit. Again, these measures should not be reported in terms of the mean scores for a group but should be reported in terms of the proportion of individual patients whose scores crossed the meaningful threshold.
      • Harris J.D.
      • Brand J.C.
      • Cote M.P.
      • Faucett S.C.
      • Dhawan A.
      Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
      In summary, equipped with PROM tools such as MCID, PASS, and SCB reported in terms of the proportion of individuals meeting measurable thresholds of satisfaction, readers could interpret the relevance of research in terms of clinical significance (appreciable benefit to the patient) as opposed to statistical significance (based on P values).

      References

        • Rossi M.
        • Brand J.
        • Lubowitz J.
        Tools to improve scientific research.
        Arthroscopy. 2018; 34: 113-125
        • Dhawan A.
        • Brand J.C.
        • Provencher M.T.
        • Rossi M.J.
        • Lubowitz J.H.
        Research pearls: The significance of statistics and perils of pooling.
        Arthroscopy. 2017; 33: 1099-1101
        • Harris J.D.
        • Brand J.C.
        • Cote M.P.
        • Faucett S.C.
        • Dhawan A.
        Research pearls: The significance of statistics and perils of pooling. Part 1: Clinical versus statistical significance.
        Arthroscopy. 2017; 33: 1102-1112
        • Hohmann E.
        • Wetzler M.J.
        • D’Agostino Jr., R.B.
        Research pearls: The significance of statistics and perils of pooling. Part 2: Predictive modeling.
        Arthroscopy. 2017; 33: 1423-1432
        • Harris J.D.
        • Brand J.C.
        • Cote M.P.
        • Dhawan A.
        Research pearls: The significance of statistics and perils of pooling. Part 3: Pearls and pitfalls of meta-analyses and systematic reviews.
        Arthroscopy. 2017; 33: 1594-1602
        • Kartus J.
        • Cote M.
        Invention versus gold standard: A hands-on research pearl on study design and statistical concerns.
        Arthroscopy. 2018; 34: 3266-3270
        • Hohmann E.
        • Cote M.
        • Brand J.
        Research pearls: Expert consensus based evidence using the Delphi method.
        Arthroscopy. 2018; 34: 3278-3282
        • Hohmann E.
        • Feldman M.
        • Hunt T.J.
        • Cote M.
        • Brand J.C.
        Research pearls: How do we establish the level of evidence?.
        Arthroscopy. 2018; 34: 3271-3277
        • Cote M.P.
        • Lubowitz J.H.
        • Rossi M.J.
        • Brand J.C.
        Reviews pooling heterogeneous, low-evidence, high-bias data result in incorrect conclusions: But heterogeneity is an opportunity to explore.
        Arthroscopy. 2018; 34: 3126-3128
        • Walsh M.
        • Srinathan S.
        • McAuley D.
        • et al.
        The statistical significance of randomized controlled trial results is frequently fragile: A case for a Fragility Index.
        J Clin Epidemiol. 2014; 67: 622-628