Volume 26, Issue 2 , Pages 144-146, February 2010
Selection Bias Results in Misinterpretation of Randomized Controlled Trials on Arthroscopic Treatment of Patients With Knee Osteoarthritis
Article Outline
It can be safely said that we all have bias, to some degree or another, stemming largely from what we have been taught, and tempered or reinforced by our own experience. Realizing that, most researchers take great pains to minimize if not eliminate it. The prospective randomized controlled trial (especially when double-blinded) has often been touted as achieving this goal, especially in terms of avoiding selection bias. However, unless randomization is done properly, selection bias may still be present. How this can occur despite overt randomization has been eloquently described by Randelli et al.,1 as have strategies to avoid it. Mathematical methods of randomization before study initiation, as well as statistical manipulations to account for possible bias-related variables after the study conclusion, have their limitations, especially if the bias is more basic, pervasive, and unrecognized.
One example of a published investigation with a major selection bias at its core is that of Moseley et al.2 Overtly, it is a randomized, prospective, sham-controlled, and double-blinded investigation about the efficacy of arthroscopic debridement in treating painful knees with underlying arthrosis. However, it has been criticized for methodologic errors, such as the use of a nonvalidated outcome measure, inclusion of patients with significant malalignment or advanced disease, and a focus primarily on older men in a Veterans Administration hospital. However, there is a more basic flaw in this investigation—that of selection bias.
I know that to be true because for the majority of the 3-year enrollment period of the investigation, I was in charge of the arthroscopy service at that very institution, and patients there complaining of knee pain were routinely referred to the knee and shoulder clinic that I ran. At evaluation, those patients I thought would benefit from knee arthroscopy were offered that procedure, whereas those I thought would not benefit were not offered such intervention. However, some of them still wanted something to be done, and so would find their way to Dr. Moseley's study, which was quite separate and apart from the clinical arthroscopy service. Thus, there was a significant selection bias because the majority of subjects recruited into that trial had already been deemed unlikely to respond to arthroscopic surgery.
The results of that study, therefore, do not in fact show knee arthroscopy to be of no value in joints with radiographic evidence of arthrosis. Rather, they simply confirm that patients with radiographically visible arthritis do not significantly improve from arthroscopic intervention if orthopaedic evaluation deems it unlikely. The same may be said for patients with no radiographic evidence of gonarthrosis; they are unlikely to benefit from knee arthroscopy if orthopaedic evaluation determines that they are unlikely to significantly improve.
More recently, another prospective, randomized clinical trial evaluating the efficacy of knee arthroscopy in patients with gonarthrosis was published, again, in the New England Journal of Medicine.3 This study compared arthroscopic surgery with medical treatment in a general population setting and used validated outcome measurement devices (WOMAC, SF-36, etc.). Exclusion criteria included severe disease or significant malalignment (more than 5° of varus or valgus deformity) so as to address some of the weaknesses of the previous investigation. Again, it overtly appears to be a well-designed study and, unlike that of Moseley et al., I have no special inside knowledge of selection bias. Perusing the article carefully, however, I found it quite odd that it took a group of 7 orthopaedic surgeons 6½ years merely to identify 270 patients for study inclusion. This means each surgeon could, on average, identify only approximately 6 patients a year with knee arthrosis and pain not stemming from a bucket-handle meniscal tear. Such a low subject collection rate for such a common condition strongly suggests that only those with arthrosis who were thought unlikely to benefit from arthroscopy were identified for study inclusion, similar to the case with the investigation of Moseley et al. Unlike the previous investigators, however, who clearly stated in their hypothesis their personal bias that they did not expect to find arthroscopic debridement would yield lasting benefit, the only explicitly stated hypothesis in the more recent article is the authors' belief that those with less severe arthrosis or those reporting mechanical symptoms would do better following surgical treatment. This hypothesis was not met—an unsurprising result if the recruited subjects were deemed unlikely to respond to arthroscopic surgery in the first place.
In both cases, we have a clear result, but what was the actual question? In this I am reminded of Douglas Adams' cult comedic science fiction trilogy, The Hitchhiker's Guide to the Galaxy, in which the ultimate supercomputer of an alien race, after centuries of computation, gives the answer to the fundamental question of life, the universe, and everything as “42,” and blames any failure to comprehend the result on poor framing of the original question. A clearly stated hypothesis not only allows for proper framing of the basic question(s) that a clinical investigation seeks to answer, but also for the expression of investigator bias. Such expression may take the form of an explicit statement of bias or a statement of expected outcome.
Now, both of these clinical investigations of knee arthroscopy in patients with arthrosis were conducted by very reputable researchers. However, no investigator likes to doubt the very data he or she is presenting, and so he or she often defends it even while following through with the requisite enumeration of study weaknesses. Clearly stating investigator bias in the hypothesis can serve as a reminder that some may be at play, especially if the results of a clinical investigation are in line with said bias. This may then help temper conclusions drawn from the study results, both by the authors and by their readers.
In summary, selection bias can cause serious misinterpretation of study results. The hypothesis of a clinical study affords opportunity for the investigators to articulate their own personal bias. For the sake of full disclosure, my view, based on the views of my teachers, including Dr. Moseley (who later was my partner for several years and is now a respected colleague), is that arthroscopic debridement does not give lasting relief from symptoms of knee arthritis, but can effectively address specific pain generators, such as large meniscal tears, loose bodies, and unstable articular cartilage flaps in knees with or without underlying arthrosis. That teaching has, over the years, been reinforced by my clinical experience, i.e., it is my bias.
References
PII: S0749-8063(09)01075-5
doi:10.1016/j.arthro.2009.12.004
© 2010 Arthroscopy Association of North America. Published by Elsevier Inc. All rights reserved.
Volume 26, Issue 2 , Pages 144-146, February 2010
