Purpose
The purpose of our study was to compare real-time, live observational scoring with
delayed retrospective video review of operative performance and to determine whether
the evaluation method affected the attainment of proficiency benchmarks.
Methods
Sixteen arthroscopy/sports medicine fellows and 2 senior residents completed training
to perform arthroscopic Bankart repairs (ABRs) and arthroscopic rotator cuff repairs
(ARCRs) using a proficiency-based progression curriculum. Each final operative performance
for 15 randomly selected ABRs and 13 ARCRs performed on cadavers were scored live
(observation during the operative performance) and on delayed video review (6-8 weeks)
by 1 of 15 trained raters using validated metric-based (step and error) assessment
tools. The inter-rater reliability (IRR) of live versus video review by a single rater
was calculated, and changes to the trainee’s attainment of the proficiency benchmarks
were noted. The correlation coefficient (r) and the R2 were also calculated for the paired scores from the randomly selected performances.
Results
No significant differences in the observed IRR agreement or the attainment of the
proficiency benchmarks were found when comparing live to video assessment for either
ABR or ARCR. The correlation coefficients r and R2 were considerably lower than the agreement coefficient (IRR) for rotator cuff steps
(e.g., R2 = 0.74 vs. IRR = 0.97, P = 0.001); Bankart errors (R2 = 0.73 vs. IRR = 0.98, P = 0.006); and rotator cuff errors (R2 = 0.48 vs. IRR = 0.98, P = 0.0002).
Conclusions
Real-time live and delayed video-based scoring of operative performance are essentially
equivalent for the metric-based assessments of operative performance in ABRs and ARCRs.
When the IRR agreement coefficient was compared with the correlation coefficients,
the former was found to have greater homogeneity and measurement precision.
Clinical Relevance
Metric-based live scoring is reliable and accurate for operative performance assessment,
including high-stakes evaluations.
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to ArthroscopyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Fundamentals of surgical simulation : Principles and Practices.Springer, London, UK2012
- Metric-based simulation training to proficiency in medical education: What it is and how to do it.Ulster Med J. 2012; 81: 107-113
- Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training.Ann Surg. 2005; 241: 364-372
- The role of deliberate practice in the acquisition of expert performance.Psychol Rev. 1993; 100: 363
- Prospective, randomized, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying.J Am Coll Surg. 2008; 207: 560-568
- Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies.Am J Surg. 2007; 193: 797-804
- Virtual reality training improves operating room performance: Results of a randomized, double-blinded study.Ann Surg. 2002; 236: 458-464
- Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators.BMJ Sim Technol Enhan Learn. 2016; 2: 1-5
- Proficiency-based progression training: An "end-to-end"model for decreasing error applied to achievement of effective epidural analgesia during labour: A randomised control study.BMJ Open. 2018; 8e020099
- Effect of proficiency-based progression simulation training and standard simulation training on isbar performance. A randomized controlled trial.BMJ Open. 2019; 9 (e025992)
- Metric development for an arthroscopic Bankart procedure: Assessment of face and content validity.Arthroscopy. 2015; 31: 1430-1440
- The Bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill.Arthroscopy. 2015; 31: 1639-1654
- The Bankart performance metrics combined with a cadaveric shoulder create a precise and accurate assessment tool for measuring surgeon skill.Arthroscopy. 2015; 31: 1655-1670
- Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.Arthroscopy. 2015; 31: 1872-1879
- Inter-rater reliability for metrics scored in a binary fashion: Performance assessment for an arthroscopic Bankart repair.Arthroscopy. 2018; 34: 2191-2198
- Arthroscopic rotator cuff repair metrics: establishing face, content, and construct validity in a cadaveric model.Arthroscopy. 2020; 36: 71-79. e71
- A proficiency-based progression simulation training curriculum to acquire the skills needed in performing arthroscopic Bankart and rotator cuff repairs: Implementation and impact.Arthroscopy. 2020; 37: 1099-1106
- A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set.Arthroscopy. 2015; 31: 1854-1871
- Video assessment of basic surgical trainees' operative skills.Am J Surg. 2008; 196: 265-272
- Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room.Ann Surg. 2008; 247: 372-379
- The arthroscopic surgical skill evaluation tool (Asset).Am J Sports Med. 2013; 41: 1229-1237
- Artifact, bias, and complexity of assessment: The ABCs of reliability.J Appl Behav Anal. 1977; 10: 141-150
- Analysis of errors in laparoscopic surgical procedures.Surg Endosc. 2004; 18: 592-595
Article info
Publication history
Published online: May 04, 2021
Accepted:
April 23,
2021
Received:
November 18,
2020
Footnotes
The authors report the following conflicts of interest or sources of : Funding Provided by The Arthroscopy Association of North America Education Foundation. The authors report travel support from AANA for the conduct of the PBP Shoulder Training Course. P.S.P. reports consultancy and payment for lectures including service on speakers bureaus for Stryker, DJO Surgical, and Zimmer Biomet, outside the submitted work; and royalties from DJO Surgical and Zimmer Biomet, outside the submitted work. J.T. reports consultancy for ConMed, outside the submitted work; patent fees paid to institution from KFX Corporation; and having stock/stock options in KFX corporation. Full ICMJE author disclosure forms are available for this article online, as supplementary material.
Identification
Copyright
© 2021 by the Arthroscopy Association of North America
