Advertisement

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

      Purpose

      The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model.

      Methods

      Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models—faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review.

      Results

      The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05).

      Conclusions

      The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.

      Clinical Relevance

      As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Arthroscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Williams R.G.
        • Verhulst S.
        • Colliver J.A.
        • Sanfey H.
        • Chen X.
        • Dunnington G.L.
        A template for reliable assessment of resident operative performance: Assessment intervals, numbers of cases and raters.
        Surgery. 2012; 152 (discussion 24-27): 517-524
        • Hodges B.D.
        A tea-steeping or i-Doc model for medical education?.
        Acad Med. 2010; 85: S34-S44
        • Pugh D.
        • Touchie C.
        • Wood T.J.
        • Humphrey-Murto S.
        Progress testing: Is there a role for the OSCE?.
        Med Educ. 2014; 48: 623-631
        • Long D.M.
        Competency-based residency training: The next advance in graduate medical education.
        Acad Med. 2000; 75: 1178-1183
        • Dwyer T.
        • Theodoropoulos J.S.
        • Herold J.
        • et al.
        Assessing competence of orthopaedic residents: The reliability and validity of an objective structured clinical examination after a sports medicine rotation.
        J Bone Joint Surg Am. 2013; 95: e177
        • Dwyer T.
        • Glover Takahashi S.
        • Kennedy Hynes M.
        • et al.
        How to assess communication, professionalism, collaboration and the other intrinsic CanMEDS roles in orthopedic residents: Use of an objective structured clinical examination (OSCE).
        Can J Surg. 2014; 57: 230-236
        • Reznick R.
        • Regehr G.
        • MacRae H.
        • Martin J.
        • McCulloch W.
        Testing technical skill via an innovative “bench station” examination.
        Am J Surg. 1997; 173: 226-230
        • Farnworth L.R.
        • Lemay D.E.
        • Wooldridge T.
        • et al.
        A comparison of operative times in arthroscopic ACL reconstruction between orthopaedic faculty and residents: The financial impact of orthopaedic surgical training in the operating room.
        Iowa Orthop J. 2001; 21: 31-35
        • Dwyer T.
        • Slade Shantz J.
        • Chahal J.
        • et al.
        Simulation of anterior cruciate ligament reconstruction in a dry model.
        Am J Sports Med. 2015; 43: 2997-3004
        • Martin J.A.
        • Regehr G.
        • Reznick R.
        • et al.
        Objective Structured Assessment of Technical Skill (OSATS) for surgical residents.
        Br J Surg. 1997; 84: 273-278
        • Sidhu R.S.
        • Grober E.D.
        • Musselman L.J.
        • Reznick R.K.
        Assessing competency in surgery: Where to begin?.
        Surgery. 2004; 135: 6-20
        • van Hove P.D.
        • Tuijthof G.J.
        • Verdaasdonk E.G.
        • Stassen L.P.
        • Dankelman J.
        Objective assessment of technical surgical skills.
        Br J Surg. 2010; 97: 972-987
        • Koehler R.J.
        • Amsdell S.
        • Arendt E.A.
        • et al.
        The Arthroscopic Surgical Skill Evaluation Tool (ASSET).
        Am J Sports Med. 2013; 41: 1229-1237
        • Carraccio C.L.
        • Benson B.J.
        • Nixon L.J.
        • Derstine P.L.
        From the educational bench to the clinical bedside: Translating the Dreyfus developmental model to the learning of clinical skills.
        Acad Med. 2008; 83: 761-767
        • Batalden P.
        • Leach D.
        • Swing S.
        • Dreyfus H.
        • Dreyfus S.
        General competencies and accreditation in graduate medical education.
        Health Aff (Millwood). 2002; 21: 103-111
        • Karam M.D.
        • Pedowitz R.A.
        • Natividad H.
        • Murray J.
        • Marsh J.L.
        Current and future use of surgical skills training laboratories in orthopaedic resident education: A national survey.
        J Bone Joint Surg Am. 2013; 95: e4
        • Anastakis D.J.
        • Regehr G.
        • Reznick R.K.
        • et al.
        Assessment of technical skills transfer from the bench training model to the human model.
        Am J Surg. 1999; 177: 167-170
        • Olson T.
        • Koehler R.
        • Butler A.
        • Amsdell S.
        • Nicandri G.
        Is there a valid and reliable assessment of diagnostic knee arthroscopy skill?.
        Clin Orthop Relat Res. 2013; 471: 1670-1676
        • Insel A.
        • Carofino B.
        • Leger R.
        • Arciero R.
        • Mazzocca A.D.
        The development of an objective model to assess arthroscopic performance.
        J Bone Joint Surg Am. 2009; 91: 2287-2295
        • Gomoll A.H.
        • O'Toole R.V.
        • Czarnecki J.
        • Warner J.J.
        Surgical experience correlates with performance on a virtual reality simulator for shoulder arthroscopy.
        Am J Sports Med. 2007; 35: 883-888
        • Gomoll A.H.
        • Pappas G.
        • Forsythe B.
        • Warner J.J.
        Individual skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study.
        Am J Sports Med. 2008; 36: 1139-1142
        • McCarthy A.D.
        • Moody L.
        • Waterworth A.R.
        • Bickerstaff D.R.
        Passive haptics in a knee arthroscopy simulator: Is it valid for core skills training?.
        Clin Orthop Relat Res. 2006; 442: 13-20
        • Pedowitz R.A.
        • Esch J.
        • Snyder S.
        Evaluation of a virtual reality simulator for arthroscopy skills development.
        Arthroscopy. 2002; 18: E29
        • Srivastava S.
        • Youngblood P.L.
        • Rawn C.
        • Hariri S.
        • Heinrichs W.L.
        • Ladd A.L.
        Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity.
        J Shoulder Elbow Surg. 2004; 13: 196-205
        • Butler A.
        • Olson T.
        • Koehler R.
        • Nicandri G.
        Do the skills acquired by novice surgeons using anatomic dry models transfer effectively to the task of diagnostic knee arthroscopy performed on cadaveric specimens?.
        J Bone Joint Surg Am. 2013; 95 (e15(1-8))
        • Chandra D.B.
        • Savoldelli G.L.
        • Joo H.S.
        • Weiss I.D.
        • Naik V.N.
        Fiberoptic oral intubation: The effect of model fidelity on training for transfer to patient care.
        Anesthesiology. 2008; 109: 1007-1013
        • Grober E.D.
        • Hamstra S.J.
        • Wanzel K.R.
        • et al.
        The educational impact of bench model fidelity on the acquisition of technical skill: The use of clinically relevant outcome measures.
        Ann Surg. 2004; 240: 374-381
        • Matsumoto E.D.
        • Hamstra S.J.
        • Radomski S.B.
        • Cusimano M.D.
        The effect of bench model fidelity on endourological skills: A randomized controlled study.
        J Urol. 2002; 167: 1243-1247
        • McDougall E.M.
        • Kolla S.B.
        • Santos R.T.
        • et al.
        Preliminary study of virtual reality and model simulation for learning laparoscopic suturing skills.
        J Urol. 2009; 182: 1018-1025
        • Zendejas B.
        • Wang A.T.
        • Brydges R.
        • Hamstra S.J.
        • Cook D.A.
        Cost: The missing outcome in simulation-based medical education research; a systematic review.
        Surgery. 2013; 153: 160-176
        • Alvand A.
        • Auplish S.
        • Gill H.
        • Rees J.
        Innate arthroscopic skills in medical students and variation in learning curves.
        J Bone Joint Surg Am. 2011; 93: e115(1-9)
        • Howells N.R.
        • Auplish S.
        • Hand G.C.
        • Gill H.S.
        • Carr A.J.
        • Rees J.L.
        Retention of arthroscopic shoulder skills learned with use of a simulator. Demonstration of a learning curve and loss of performance level after a time delay.
        J Bone Joint Surg Am. 2009; 91: 1207-1213
        • Howells N.R.
        • Gill H.S.
        • Carr A.J.
        • Price A.J.
        • Rees J.L.
        Transferring simulated arthroscopic skills to the operating theatre: A randomised blinded study.
        J Bone Joint Surg Br. 2008; 90: 494-499
        • Hodgins J.L.
        • Veillette C.
        • Biau D.
        • Sonnadara R.
        The knee arthroscopy learning curve: Quantitative assessment of surgical skills.
        Arthroscopy. 2014; 30: 613-621
        • Dwyer T.
        • Wright S.
        • Kulasegaram K.M.
        • et al.
        Competency-based medical education: Can both junior residents and senior residents achieve competence after a sports medicine training module?.
        J Bone Joint Surg Am. 2015; 97: 1985-1991
        • Dwyer T.
        • Wadey V.
        • Archibald D.
        • et al.
        Cognitive and psychomotor entrustable professional activities: Can simulators help assess competency in trainees?.
        Clin Orthop Relat Res. 2016; 474: 926-934
        • ACGME
        The Orthopaedic Surgery Milestone Project 2013.
        July 20, 2015
        • Korndorffer Jr., J.R.
        • Kasten S.J.
        • Downing S.M.
        A call for the utilization of consensus standards in the surgical education literature.
        Am J Surg. 2010; 199: 99-104
        • Datta V.
        • Bann S.
        • Beard J.
        • Mandalia M.
        • Darzi A.
        Comparison of bench test evaluations of surgical skill with live operating performance assessments.
        J Am Coll Surg. 2004; 199: 603-606
        • Ghaderi I.
        • Vaillancourt M.
        • Sroka G.
        • et al.
        Performance of simulated laparoscopic incisional hernia repair correlates with operating room performance.
        Am J Surg. 2011; 201: 40-45
        • McCluney A.L.
        • Vassiliou M.C.
        • Kaneva P.A.
        • et al.
        FLS simulator performance predicts intraoperative laparoscopic skill.
        Surg Endosc. 2007; 21: 1991-1995
        • Beard J.D.
        • Jolly B.C.
        • Newble D.I.
        • Thomas W.E.
        • Donnelly J.
        • Southgate L.J.
        Assessing the technical skills of surgical trainees.
        Br J Surg. 2005; 92: 778-782
        • Bann S.
        • Khan M.S.
        • Datta V.K.
        • Darzi A.W.
        Technical performance: Relation between surgical dexterity and technical knowledge.
        World J Surg. 2004; 28 (discussion 6-7): 142-146
        • Putnam M.D.
        • Kinnucan E.
        • Adams J.E.
        • Van Heest A.E.
        • Nuckley D.J.
        • Shanedling J.
        On orthopedic surgical skill prediction—the limited value of traditional testing.
        J Surg Educ. 2015; 72: 458-470
        • Van Heest A.
        • Putnam M.
        • Agel J.
        • Shanedling J.
        • McPherson S.
        • Schmitz C.
        Assessment of technical skills of orthopaedic surgery residents performing open carpal tunnel release surgery.
        J Bone Joint Surg Am. 2009; 91: 2811-2817
        • Brooks M.A.
        Medical education and the tyranny of competency.
        Perspect Biol Med. 2009; 52: 90-102
        • Ginsburg S.
        • McIlroy J.
        • Oulanova O.
        • Eva K.
        • Regehr G.
        Toward authentic clinical evaluation: pitfalls in the pursuit of competency.
        Acad Med. 2010; 85: 780-786