Advertisement

Authors Dichotomize Medical Research Findings as Significant Versus Not Significant, Creating a False Sense of Certainty, and Report Outcomes on Patients Whose Results Have Been Previously Reported Without Proper Disclosure

      Abstract

      Statistical significance dichotomizes research findings into significant versus not significant, creating a false sense of certainty. It is insufficient to mindlessly report results as significant versus not significant without providing a quantitative estimate of the uncertainty of the data. Authors could provide a confidence interval, draw a P value function graph, or run a Bayesian analysis. Authors could calculate and report a Surprise or S value. Most importantly, authors could thoughtfully consider how the uncertainty within their research data informs the results of their study. And, clinical databases allow researchers to test multiple hypotheses. This could result in reporting outcomes on the same patient or patients in more than 1 study. Such “double-dipping” is not a dilemma in and of itself, but a problem occurs if multiple reporting of outcomes on the same patient or patients is not disclosed in the methods of a study. Absent clarifying disclosure of multiple reporting, a single patient might then be counted twice in future systematic reviews or meta-analyses, resulting in a biased and incorrect review of the literature. Authors using databases to report clinical outcomes must absolutely and explicitly clarify in their methods if the results of 1 or more patients included in their study have been reported in previous publications.
      Quiz:
      Section I: The Uncertainty Challenge
      • 1.
        In April 2021, we published:
      • “Statistical significance dichotomizes research findings into significant versus not significant creating a false sense of certainty.”
        • Cote M.P.
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Misinterpretation of P values and statistical power creates a false sense of certainty: Statistical significance, lack of significance, and the uncertainty challenge.
      • ___True ___False
      • 2.
        In April 2021, we published:
      • “It is insufficient to mindlessly report results as significant versus not significant without providing a quantitative estimate of the uncertainty of the data.”
        • Cote M.P.
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Misinterpretation of P values and statistical power creates a false sense of certainty: Statistical significance, lack of significance, and the uncertainty challenge.
      • ___True ___False
      • 3.
        In April 2021, we published:
      • “Authors could provide a confidence interval, draw a P-value function graph, or run a Bayesian analysis. Authors could calculate and report an S value.”
        • Cote M.P.
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Misinterpretation of P values and statistical power creates a false sense of certainty: Statistical significance, lack of significance, and the uncertainty challenge.
      • ___True ___False
      • 4.
        In April 2021, we published:
      • “More importantly: interpret. Consider the values …and interpret how these may inform the results of the study.”
        • Cote M.P.
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Misinterpretation of P values and statistical power creates a false sense of certainty: Statistical significance, lack of significance, and the uncertainty challenge.
      • ___True ___False
      • 5.
        Nevertheless, in 2022, authors continue to report results as “statistically significant” versus “not statistically significant” without thoughtfully considering how the uncertainty within their research data informs the results of their study.
      • ___True ___False
      • 6.
        In 2022, authors submitting original scientific articles to Arthroscopy will actually have to check a box prior to submission, answering the Yes/No question:
      • “Have you reported results as significant versus not significant without thoughtfully providing a quantitative estimate of the uncertainty of the data?”
      • ___True ___False
      • 7.
        In 2022, authors checking the box described above as “Yes” will be instructed not to submit their paper until they revise their submission by interpreting how the uncertainty within their research data informs the results of their study.
      • ___True ___False
      Section II: Proper Disclosure When Reporting Outcomes on Patients Included in Previous Publications
      • 8.
        In March 2020, we published:
      • “Clinical databases allow researchers to test multiple hypotheses. This could result in including outcomes on the same patient or patients in more than 1 study.”
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
      • ___True ___False
      • 9.
        In March 2020, we published:
      • “Such ‘double-dipping’ is not a dilemma in and of itself, but a problem occurs if multiple reporting of outcomes on the same patient or patients is not disclosed in the methods of a study.”
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
      • ___True ___False
      • 10.
        In March 2020, we published:
      • “Absent clarifying disclosure of multiple reporting, a single patient might then be counted twice in future systematic reviews or meta-analyses, resulting in biased and incorrect review of the literature.”
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
      • ___True ___False
      • 11.
        In March 2020, we published:
      • “(A)uthors using databases to report clinical outcomes must absolutely and explicitly clarify in their methods if the results of 1 or more patients included in their study have been reported in previous publications.”
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
      • ___True ___False
      • 12.
        In March 2020, we published:
      • “(T)o repeat, authors must disclose in their study methods the fact (or even the possibility) that their research includes reporting of outcomes on a patient or patients whose result(s) have been previously reported.”
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
      • ___True ___False
      • 13.
        Nevertheless, in 2022, authors continue to report outcomes on a patient or patients whose result(s) have been previously reported in other studies without proper disclosure in their methods.
      • ___True ___False
      • 14.
        In 2022, authors submitting original scientific articles to Arthroscopy will actually have to check a box prior to submission, answering the Yes/No question:
      • “Have you reported on a patient or patients whose result(s) have been previously reported in other studies without proper disclosure in the methods of your current submission?”
      • ___True ___False
      • 15.
        In 2022, authors checking the box described above as “Yes” will be instructed not to submit their paper until they revise their submission by disclosing that their research includes even the possibility that they are reporting outcomes on a patient or patients whose result(s) have been previously reported in other studies.
      • ___True ___False
      Section III: Answer Key
      The answer to all questions above is True. Submitting authors should consider themselves twice notified.

      References

        • Cote M.P.
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Misinterpretation of P values and statistical power creates a false sense of certainty: Statistical significance, lack of significance, and the uncertainty challenge.
        Arthroscopy. 2021; 37: 1057-1063
        • Lubowitz J.H.
        • Brand J.C.
        • Rossi M.J.
        Do some patients count more than others? Reporting outcomes of the same patient in more than one study requires disclosure.
        Arthroscopy. 2020; 36: 617-618