Advertisement

Live Observational Objective Assessment of Operative Performance in a Cadaveric Model is Equivalent to Delayed Video-Based Assessment

      Purpose

      The purpose of our study was to compare real-time, live observational scoring with delayed retrospective video review of operative performance and to determine whether the evaluation method affected the attainment of proficiency benchmarks.

      Methods

      Sixteen arthroscopy/sports medicine fellows and 2 senior residents completed training to perform arthroscopic Bankart repairs (ABRs) and arthroscopic rotator cuff repairs (ARCRs) using a proficiency-based progression curriculum. Each final operative performance for 15 randomly selected ABRs and 13 ARCRs performed on cadavers were scored live (observation during the operative performance) and on delayed video review (6-8 weeks) by 1 of 15 trained raters using validated metric-based (step and error) assessment tools. The inter-rater reliability (IRR) of live versus video review by a single rater was calculated, and changes to the trainee’s attainment of the proficiency benchmarks were noted. The correlation coefficient (r) and the R2 were also calculated for the paired scores from the randomly selected performances.

      Results

      No significant differences in the observed IRR agreement or the attainment of the proficiency benchmarks were found when comparing live to video assessment for either ABR or ARCR. The correlation coefficients r and R2 were considerably lower than the agreement coefficient (IRR) for rotator cuff steps (e.g., R2 = 0.74 vs. IRR = 0.97, P = 0.001); Bankart errors (R2 = 0.73 vs. IRR = 0.98, P = 0.006); and rotator cuff errors (R2 = 0.48 vs. IRR = 0.98, P = 0.0002).

      Conclusions

      Real-time live and delayed video-based scoring of operative performance are essentially equivalent for the metric-based assessments of operative performance in ABRs and ARCRs. When the IRR agreement coefficient was compared with the correlation coefficients, the former was found to have greater homogeneity and measurement precision.

      Clinical Relevance

      Metric-based live scoring is reliable and accurate for operative performance assessment, including high-stakes evaluations.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Arthroscopy
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Gallagher A.G.
        • O'Sullivan G.C.
        Fundamentals of surgical simulation : Principles and Practices.
        Springer, London, UK2012
        • Gallagher A.
        Metric-based simulation training to proficiency in medical education: What it is and how to do it.
        Ulster Med J. 2012; 81: 107-113
        • Gallagher A.G.
        • Ritter E.M.
        • Champion H.
        • et al.
        Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training.
        Ann Surg. 2005; 241: 364-372
        • Ericsson K.A.
        • Krampe R.T.
        • Tesch-Römer C.
        The role of deliberate practice in the acquisition of expert performance.
        Psychol Rev. 1993; 100: 363
        • Van Sickle K.R.
        • Ritter E.M.
        • Baghai M.
        • et al.
        Prospective, randomized, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying.
        J Am Coll Surg. 2008; 207: 560-568
        • Ahlberg G.
        • Enochsson L.
        • Gallagher A.G.
        • et al.
        Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies.
        Am J Surg. 2007; 193: 797-804
        • Seymour N.E.
        • Gallagher A.G.
        • Roman S.A.
        • et al.
        Virtual reality training improves operating room performance: Results of a randomized, double-blinded study.
        Ann Surg. 2002; 236: 458-464
        • Cates C.U.
        • Lönn L.
        • Gallagher A.G.
        Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators.
        BMJ Sim Technol Enhan Learn. 2016; 2: 1-5
        • Srinivasan K.K.
        • Gallagher A.
        • O’Brien N.
        • et al.
        Proficiency-based progression training: An "end-to-end"model for decreasing error applied to achievement of effective epidural analgesia during labour: A randomised control study.
        BMJ Open. 2018; 8e020099
        • Breen D.
        • O’Brien S.
        • McCarthy N.
        • Gallagher A.G.
        • Walshe N.
        Effect of proficiency-based progression simulation training and standard simulation training on isbar performance. A randomized controlled trial.
        BMJ Open. 2019; 9 (e025992)
        • Angelo R.L.
        • Ryu R.K.
        • Pedowitz R.A.
        • Gallagher A.G.
        Metric development for an arthroscopic Bankart procedure: Assessment of face and content validity.
        Arthroscopy. 2015; 31: 1430-1440
        • Angelo R.L.
        • Pedowitz R.A.
        • Ryu R.K.
        • Gallagher A.G.
        The Bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill.
        Arthroscopy. 2015; 31: 1639-1654
        • Angelo R.L.
        • Ryu R.K.
        • Pedowitz R.A.
        • Gallagher A.G.
        The Bankart performance metrics combined with a cadaveric shoulder create a precise and accurate assessment tool for measuring surgeon skill.
        Arthroscopy. 2015; 31: 1655-1670
        • Pedowitz R.A.
        • Nicandri G.T.
        • Angelo R.L.
        • Ryu R.K.
        • Gallagher A.G.
        Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
        Arthroscopy. 2015; 31: 1872-1879
        • Gallagher A.G.
        • Ryu R.K.
        • Pedowitz R.A.
        • Henn P.
        • Angelo R.L.
        Inter-rater reliability for metrics scored in a binary fashion: Performance assessment for an arthroscopic Bankart repair.
        Arthroscopy. 2018; 34: 2191-2198
        • Angelo R.L.
        • Tauro J.
        • Pierre P.S.
        • et al.
        Arthroscopic rotator cuff repair metrics: establishing face, content, and construct validity in a cadaveric model.
        Arthroscopy. 2020; 36: 71-79. e71
        • Angelo R.L.
        • Pierre P.S.
        • Tauro J.
        • Gallagher A.G.
        A proficiency-based progression simulation training curriculum to acquire the skills needed in performing arthroscopic Bankart and rotator cuff repairs: Implementation and impact.
        Arthroscopy. 2020; 37: 1099-1106
        • Angelo R.L.
        • Ryu R.K.
        • Pedowitz R.A.
        • et al.
        A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set.
        Arthroscopy. 2015; 31: 1854-1871
        • Driscoll P.J.
        • Paisley A.M.
        • Paterson-Brown S.
        Video assessment of basic surgical trainees' operative skills.
        Am J Surg. 2008; 196: 265-272
        • Aggarwal R.
        • Grantcharov T.
        • Moorthy K.
        • Milland T.
        • Darzi A.
        Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room.
        Ann Surg. 2008; 247: 372-379
        • Koehler R.J.
        • Amsdell S.
        • Arendt E.A.
        • et al.
        The arthroscopic surgical skill evaluation tool (Asset).
        Am J Sports Med. 2013; 41: 1229-1237
        • Kazdin A.E.
        Artifact, bias, and complexity of assessment: The ABCs of reliability.
        J Appl Behav Anal. 1977; 10: 141-150
        • Seymour N.E.
        • Gallagher A.G.
        • Roman S.A.
        • O'Brien M.K.
        • Andersen D.K.
        • Satava R.M.
        Analysis of errors in laparoscopic surgical procedures.
        Surg Endosc. 2004; 18: 592-595