Advertisement

Proficiency-Based Training Using Simulator-Based Tools Could be Validated for Certification of Surgical Procedural Proficiency

      Abstract

      Airline pilots can learn how to fly without ever leaving the ground, and pilots are required to pass proficiency checks using simulator-based tools to ensure they are competent, and remain competent, to fly specific types of planes under specific circumstances. In contrast, generally, surgical training is based on an apprenticeship model, and surgical certification does not require demonstration of hands-on skills in performing basic, advanced, or subspecialized procedures. Proficiency-based training, or PBT, also known as proficiency-based progression or PBP, is a tool that can be used to train surgeons to safely and efficiently perform surgical procedures. Moreover, PBT, simulators, and similar tools can be used to evaluate a surgeon’s proficiency. Ultimately, in the interest of quality, value, and patient safety, objective evaluation of hands-on surgical skill using simulators should be implemented. In the interest of fairness to surgeons, certification must be developed with a goal of not only testing but supporting development of procedural proficiency. As we follow our peers in the airline industry, pathways chosen to validate surgical competence must be evidence-based, fair, and valid.
      Flying has become the safest mode of transport. We note that most people drive (and many cycle) more than they fly; nevertheless, the lifetime odds of death as an airplane passenger is 1 in 188,364 compared with 1 in 4047 for a cyclist and 1 in 103 for a person in a car crash.
      National Safety council
      Preventable deaths. Odds of dying.
      How has air travel become safer? Improved protection by critical flight systems, better air traffic control technology, improved crashworthiness of airplanes, more frequent safety audits, and above all, pilot training. New pilot training protocols have significantly contributed to the safety of air passengers. Regulations limit the time pilots can spend in the cockpit, and pilots must have at least 10 hours off of work before flight duty, including 8 hours of sleep. Rigorous pilot training includes regular simulator training, including line checks; captains in training conduct line checks by observing pilots during simulated air travel while scrutinizing every aspect of pilot performance against a validated checklist. Flight simulators reproduce various and challenging flight situations. Pilots can now learn how to fly without ever leaving the ground.
      Readwrite
      Air travel is safer than you think: Here’s why.
      Pilot training consists of basic training as well as type-rating or training to fly a specific aircraft.
      • Sommer K.J.
      Pilot training: What can surgeons learn from it?.
      Type rating can be compared with a driver’s license that is valid for one specific car model alone. The pilot training process parallels how we as surgeons learn our profession. With basic training, we learn the skills that every orthopaedic surgeon should possess, while advanced training fine-tunes a surgeon’s skills to refine the performance of subspecialized procedures such as arthroscopic surgery. Like the one-aircraft approach, a focus on one specific field in orthopaedic surgery narrows one’s focus and specialization. However, this analogy is where the similarities end.
      Pilots are required to undertake proficiency checks to ensure they continue to be competent conducting particular kinds of operations. An airline pilot needs to complete at least 2 proficiency checks each year.
      Australian Government Civil Aviation Authority
      Proficiency checks.
      These mandatory checks are regulated by federal law.
      Federal Aviation Administration
      Regulations and policies.
      Without passing these compulsory checks, pilots do not fulfil their licensure requirements and, thus, cannot fly.
      Where do arthroscopic and related surgeons stand in comparison? Once we have passed our board examinations, are we allowed to practice without further regulation? Absolutely not. In 1986, the American Board of Orthopaedic Surgery implemented a recertification program with numerous pathways.
      • Rosnier R.N.
      Continuing competency in orthopaedics: The future of recertification.
      However, the program has been criticized for not having “kept up with the field.”
      • Bernstein J.
      Not the last word: The time to fix ABOS recertification has arrived.
      Most of all, the American Board of Orthopaedic Surgery certification and recertification process does not directly evaluate the hands-on surgical skills essential to our profession.
      Is there something that we can do to improve the surgical certification process? In a word, yes.
      Proficiency-based training or PBT, also known as proficiency-based progression or PBP, is a relatively new method that can be used to train surgeons to develop psychomotor skills to safely and efficiently perform surgical procedures.
      • St.Pierre P
      Editorial commentary: Proficiency-based training: Are we ready for a new way to train and test orthopaedic surgeons?.
      Angelo et al.
      • Angelo R.L.
      • Ryu R.K.
      • Pedowitz R.A.
      • Gallagher A.G.
      Metric development for an arthroscopic Bankart procedure: assessment of face and content validity.
      developed a metric-based assessment for an arthroscopic Bankart procedure with 45 steps and 13 phases, allowing characterization of the essential components of the procedure. In a different study, Angelo et al.
      • Angelo R.L.
      • Ryu R.K.
      • Pedowitz R.A.
      • et al.
      A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set.
      showed that PBT, coupled with simulator training, yielded a 56% lower error rate when performing arthroscopic Bankart repairs than with a traditionally based training method. In addition, the PBT-trained surgeons were 7.5 times more likely to achieve a final benchmark demonstrating proficiency. Other authors have confirmed Angelo et al.’s findings.
      • Rebolledo B.J.
      • Hammann-Scala J.
      • Leali A.
      • Ranawat A.S.
      Arthroscopy skills development with a surgical simulator: A comparative study in orthopaedic surgery residents.
      • Waterman B.R.
      • Martin K.D.
      • Cameron K.L.
      • Owens B.D.
      • Belmont Jr., P.J.
      Simulation training improves surgical proficiency and safety during diagnostic shoulder arthroscopy performed by residents.
      • Rahm S.
      • Wieser K.
      • Bauer D.E.
      • et al.
      Efficacy of standardized training on a virtual reality simulator to advance knee and shoulder arthroscopic motor skills.
      • Tofte J.N.
      • Westerlind B.O.
      • Martin K.D.
      • et al.
      Knee, shoulder, and fundamentals of arthroscopic surgery training: Validation of a virtual arthroscopy simulator.
      In sum, studies suggest that competency-based simulator training may serve to advance arthroscopic skills for residents, PBT is superior to the traditional apprenticeship model of surgical training, and PBT-based evaluation can be used to evaluate a surgeon’s proficiency in performing a specific surgical procedure. Theoretically, these methods could be used to certify that surgeons possess, and maintain, adequate hands-on skills required to perform surgery safely, efficiently, and with adequate competence.
      Other research supports this potential approach to assessment of competency. For example, the Fundamentals of Arthroscopic Surgery Training (FAST) program is a collaborative initiative between the Arthroscopy Association of North America and the American Academy of Orthopaedic Surgeons.
      • Pedowitz R.A.
      • Nicandri G.T.
      • Angelo R.L.
      • Ryu R.K.
      • Gallagher A.G.
      Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
      The FAST program allows task deconstruction and assessment of basic arthroscopic skills. In addition, the Arthroscopic Surgery Skill Evaluation Tool (ASSET) has been developed as a video-based assessment of technical skill that is suitable to evaluate multiple procedures in both the simulation laboratory and operative theater.
      • Koehler R.J.
      • Goldblatt J.P.
      • Maloney M.D.
      • Voloshin I.
      • Nicandri G.T.
      Assessing diagnostic arthroscopy performance in the operating room using the arthroscopic surgery skill evaluation tool (ASSET).
      ASSET, and similar tools, have also been shown to be a reliable and valid instrument to evaluate arthroscopic surgical skills.
      • Koehler R.J.
      • Goldblatt J.P.
      • Maloney M.D.
      • Voloshin I.
      • Nicandri G.T.
      Assessing diagnostic arthroscopy performance in the operating room using the arthroscopic surgery skill evaluation tool (ASSET).
      • Elliott M.J.
      • Caprise P.A.
      • Henning A.E.
      • Kurtz C.A.
      • Sekiya J.K.
      Diagnostic knee arthroscopy: A pilot study to evaluate surgical skills.
      • Slade Shantz J.A.
      • Leiter J.R.
      • Collins J.B.
      • Macdonald P.B.
      Validation of a global assessment of arthroscopic skills in a cadaveric knee model.
      • Alvand A.
      • Logishetty K.
      • Middleton R.
      • et al.
      Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair.
      In the end, simulators, PBT, FAST, ASSET, and other tools to evaluate surgical proficiency could be potentially analogous to airline pilot’s line check evaluation.
      Some might argue that these checks and balances are not required for arthroscopic surgeons. Certainly, experienced surgeons make fewer errors, have less sentinel events, and perform surgery faster and more efficiently than novices.
      • Angelo R.L.
      • Pedowitz R.A.
      • Ryu R.K.
      • Gallagher A.G.
      The Bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill.
      However, Pedowitz et al.
      • Pedowitz R.A.
      • Nicandri G.T.
      • Angelo R.L.
      • Ryu R.K.
      • Gallagher A.G.
      Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
      showed that surgeons are not infallible. For example, when checking knot-tying skills at an Arthroscopy Association of North America Resident Course, failure resulted from 24% of all knots performed by course faculty. The authors concluded that there is significant room for improvement, even among some “experienced” surgeons.
      • Pedowitz R.A.
      • Nicandri G.T.
      • Angelo R.L.
      • Ryu R.K.
      • Gallagher A.G.
      Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
      These data support the importance of continuing education and certification.
      Our Editor-in-Chief has previously suggested that the training curriculum of Angelo et al.
      • Pedowitz R.A.
      • Nicandri G.T.
      • Angelo R.L.
      • Ryu R.K.
      • Gallagher A.G.
      Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
      can be used as a tool to judge surgical expertise and that trainees whose performance falls below a specific benchmark require additional training to increase proficiency before performing live surgery.
      • Lubowitz J.H.
      • Provencher M.T.
      • Brand J.C.
      • Rossi M.J.
      Expert surgeons can be distinguished from trainees, and surgical proficiency can be defined, using validated metrics and shoulder models.
      The next logical step is to use this tool not only for residents but also for board-certified surgeons. Similar to the aviation industry, checklists could be employed to verify competencies.
      • Koehler R.J.
      • Goldblatt J.P.
      • Maloney M.D.
      • Voloshin I.
      • Nicandri G.T.
      Assessing diagnostic arthroscopy performance in the operating room using the arthroscopic surgery skill evaluation tool (ASSET).
      ,
      • Nwachukwu B.
      • Gaudiani M.
      • Hammann-Scala J.
      • Ranawat A.
      A checklist intervention to assess resident diagnostic knee and shoulder arthroscopic efficiency.
      ,
      • Gallagher A.G.
      • Ryu R.K.
      • Pedowitz R.A.
      • Henn P.
      • Angelo R.L.
      Inter-rater reliability for metrics scored in a binary fashion—Performance assessment for an arthroscopic Bankart repair.
      Whether the proficiencies tested with the simulator can replicate live surgery is a matter of debate,
      • Walbron P.
      • Thomazeau H.
      • Sirveaux F.
      Virtual reality simulation in orthopedics and trauma surgery in France: Current status and perspectives [in German].
      but several studies have suggested that simulator skills can be transferred to the operating room.
      • Vaghela K.R.
      • Lee J.
      • Akhtar K.
      Performance on a virtual reality DHS simulator correlates with performance in the operating theatre.
      • Howells N.R.
      • Gill A.S.
      • Carr A.J.
      • Price A.J.
      • Rees J.L.
      Transferring simulated arthroscopic skills to the operating theatre. A randomised blinded study.
      • Roberts P.G.
      • Alvand A.
      • Gallieri M.
      • Hargrove C.
      • Rees J.
      Objectively assessing intraoperative arthroscopic skills performance and the transfer of simulation training in knee arthroscopy: A randomized controlled trial.
      • Verdaasdonk E.G.G.
      • Dankelman J.
      • Lange J.F.
      • Stassen L.P.S.
      Transfer validity of laparoscopic knot-tying training on a VR simulator to a realistic environment: A randomized controlled trial.
      • Youngblood P.L.
      • Srivastava S.
      • Curet M.
      • Heinrichs W.L.
      • Dev P.
      • Wren S.M.
      Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance.
      Could proficiency training improve outcomes for our patients? Tam et al.
      • Tam V.
      • Borrebach J.
      • Dunn S.A.
      • Bellon J.
      • Zeh H.J.
      • Hogg M.E.
      Proficiency-based training and credentialing can improve patient outcomes and decrease cost to a hospital system.
      were able to demonstrate that proficiency-based robotic training for board-certified general surgeons reduced operative times and costs for hernia surgery providing a supportive argument.
      Four years ago, the editors of Arthroscopy wrote that they firmly believed in PBP training using simulators, as well as PBP training using cadavers, as evidence-based teaching tools.
      • Feldman M.D.
      • Brand J.C.
      • Rossi M.J.
      • Lubowitz J.H.
      Arthroscopic training in the 21st century: A changing paradigm.
      We look forward to research supporting the use of such tools for primary and continuing certification to perform orthopaedic surgery.
      However, difficult questions remain: What if the tests are “unreasonably” challenging, or worse, do not “emulate” what we practice? Can we test specific procedures (such as anterior cruciate ligament reconstruction) given that there are so many technique “variations” resulting in good and excellent outcomes? Alternatively, should we limit testing to the “basics,” such as knot tying, to ensure “fairness?” What if a “proficient” surgeon does not pass the test? What if the test is so easy that an “incompetent” surgeon receives a passing grade? How do we determine the threshold resulting in “certification?” It is critically important that tools used to perform such “high stakes” evaluation must be robust, reliable, validated, and reproducible.
      In the preceding paragraph, every sentence includes words in quotation marks, and all but the last sentence end with a question mark. These are obvious warning signs that clarify our assertion: surgical skills evaluation and certification is a goal. Research is required before this goal can be achieved in an evidence-based manner.
      In the interest of quality, value, and patient safety, objective evaluation of hands-on surgical skill using simulators should be developed and implemented to improve the application of surgeon licensure certification to clinical patient care. In the interest of fairness to patients and to surgeons, PBT should result in modification of the apprenticeship surgical training model; PBT is evidence-based superior. In all fairness to surgeons, certification must be developed with a goal of not only testing but supporting development of proficiency. As we follow our peers in the airline industry, pathways chosen to validate competence must be proven before implementation.

      References

        • National Safety council
        Preventable deaths. Odds of dying.
        (Available at:)
        • Readwrite
        Air travel is safer than you think: Here’s why.
        (Available at:)
        • Sommer K.J.
        Pilot training: What can surgeons learn from it?.
        Arab J Urol. 2014; 12: 32-36
        • Australian Government Civil Aviation Authority
        Proficiency checks.
        (Available at:)
        • Federal Aviation Administration
        Regulations and policies.
        (Available at:)
        • Rosnier R.N.
        Continuing competency in orthopaedics: The future of recertification.
        Clin Orthop Relat Res. 2006; 449: 72-75
        • Bernstein J.
        Not the last word: The time to fix ABOS recertification has arrived.
        Clin Orthop Relat Res. 2018; 476: 1928-1930
        • St.Pierre P
        Editorial commentary: Proficiency-based training: Are we ready for a new way to train and test orthopaedic surgeons?.
        Arthroscopy. 2018; 34: 2199-2200
        • Angelo R.L.
        • Ryu R.K.
        • Pedowitz R.A.
        • Gallagher A.G.
        Metric development for an arthroscopic Bankart procedure: assessment of face and content validity.
        Arthroscopy. 2015; 31: 1430-1440
        • Angelo R.L.
        • Ryu R.K.
        • Pedowitz R.A.
        • et al.
        A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set.
        Arthroscopy. 2015; 31: 1854-1871
        • Rebolledo B.J.
        • Hammann-Scala J.
        • Leali A.
        • Ranawat A.S.
        Arthroscopy skills development with a surgical simulator: A comparative study in orthopaedic surgery residents.
        Am J Sports Med. 2015; 43: 1526-1529
        • Waterman B.R.
        • Martin K.D.
        • Cameron K.L.
        • Owens B.D.
        • Belmont Jr., P.J.
        Simulation training improves surgical proficiency and safety during diagnostic shoulder arthroscopy performed by residents.
        Orthopedics. 2016; 39: e479-e485
        • Rahm S.
        • Wieser K.
        • Bauer D.E.
        • et al.
        Efficacy of standardized training on a virtual reality simulator to advance knee and shoulder arthroscopic motor skills.
        Orthopedics. 2016; 39: e479-e485
        • Tofte J.N.
        • Westerlind B.O.
        • Martin K.D.
        • et al.
        Knee, shoulder, and fundamentals of arthroscopic surgery training: Validation of a virtual arthroscopy simulator.
        Arthroscopy. 2017; 33: 641-646
        • Pedowitz R.A.
        • Nicandri G.T.
        • Angelo R.L.
        • Ryu R.K.
        • Gallagher A.G.
        Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.
        Arthroscopy. 2015; 31: 1872-1879
        • Koehler R.J.
        • Goldblatt J.P.
        • Maloney M.D.
        • Voloshin I.
        • Nicandri G.T.
        Assessing diagnostic arthroscopy performance in the operating room using the arthroscopic surgery skill evaluation tool (ASSET).
        Arthroscopy. 2015; 31: 2314-2319
        • Elliott M.J.
        • Caprise P.A.
        • Henning A.E.
        • Kurtz C.A.
        • Sekiya J.K.
        Diagnostic knee arthroscopy: A pilot study to evaluate surgical skills.
        Arthroscopy. 2012; 28: 218-224
        • Slade Shantz J.A.
        • Leiter J.R.
        • Collins J.B.
        • Macdonald P.B.
        Validation of a global assessment of arthroscopic skills in a cadaveric knee model.
        Arthroscopy. 2013; 29: 106-112
        • Alvand A.
        • Logishetty K.
        • Middleton R.
        • et al.
        Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair.
        Arthroscopy. 2013; 29: 906-912
        • Angelo R.L.
        • Pedowitz R.A.
        • Ryu R.K.
        • Gallagher A.G.
        The Bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill.
        Arthroscopy. 2015; 31: 1639-1654
        • Lubowitz J.H.
        • Provencher M.T.
        • Brand J.C.
        • Rossi M.J.
        Expert surgeons can be distinguished from trainees, and surgical proficiency can be defined, using validated metrics and shoulder models.
        Arthroscopy. 2015; 31: 1635-1636
        • Nwachukwu B.
        • Gaudiani M.
        • Hammann-Scala J.
        • Ranawat A.
        A checklist intervention to assess resident diagnostic knee and shoulder arthroscopic efficiency.
        J Surg Educ. 2017; 74: 9-15
        • Gallagher A.G.
        • Ryu R.K.
        • Pedowitz R.A.
        • Henn P.
        • Angelo R.L.
        Inter-rater reliability for metrics scored in a binary fashion—Performance assessment for an arthroscopic Bankart repair.
        Arthroscopy. 2018; 34: 2191-2198
        • Walbron P.
        • Thomazeau H.
        • Sirveaux F.
        Virtual reality simulation in orthopedics and trauma surgery in France: Current status and perspectives [in German].
        Unfallchirurg. 2019; 122: 439-443
        • Vaghela K.R.
        • Lee J.
        • Akhtar K.
        Performance on a virtual reality DHS simulator correlates with performance in the operating theatre.
        Surg Technol Int. July. 2019; 20 ([Epub ahead of print.])
        • Howells N.R.
        • Gill A.S.
        • Carr A.J.
        • Price A.J.
        • Rees J.L.
        Transferring simulated arthroscopic skills to the operating theatre. A randomised blinded study.
        J Bone Joint Surg Br. 2008; 90: 494-499
        • Roberts P.G.
        • Alvand A.
        • Gallieri M.
        • Hargrove C.
        • Rees J.
        Objectively assessing intraoperative arthroscopic skills performance and the transfer of simulation training in knee arthroscopy: A randomized controlled trial.
        Arthroscopy. 2019; 35: 1197-1209
        • Verdaasdonk E.G.G.
        • Dankelman J.
        • Lange J.F.
        • Stassen L.P.S.
        Transfer validity of laparoscopic knot-tying training on a VR simulator to a realistic environment: A randomized controlled trial.
        Surg Endosc. 2008; 22: 1636-1642
        • Youngblood P.L.
        • Srivastava S.
        • Curet M.
        • Heinrichs W.L.
        • Dev P.
        • Wren S.M.
        Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance.
        J Am Coll Surg. 2005; 200: 546-551
        • Tam V.
        • Borrebach J.
        • Dunn S.A.
        • Bellon J.
        • Zeh H.J.
        • Hogg M.E.
        Proficiency-based training and credentialing can improve patient outcomes and decrease cost to a hospital system.
        Am J Surg. 2019; 217: 591-596
        • Feldman M.D.
        • Brand J.C.
        • Rossi M.J.
        • Lubowitz J.H.
        Arthroscopic training in the 21st century: A changing paradigm.
        Arthroscopy. 2017; 33: 1913-1915