Abstract
Airline pilots can learn how to fly without ever leaving the ground, and pilots are required to pass proficiency checks using simulator-based tools to ensure they are competent, and remain competent, to fly specific types of planes under specific circumstances. In contrast, generally, surgical training is based on an apprenticeship model, and surgical certification does not require demonstration of hands-on skills in performing basic, advanced, or subspecialized procedures. Proficiency-based training, or PBT, also known as proficiency-based progression or PBP, is a tool that can be used to train surgeons to safely and efficiently perform surgical procedures. Moreover, PBT, simulators, and similar tools can be used to evaluate a surgeon’s proficiency. Ultimately, in the interest of quality, value, and patient safety, objective evaluation of hands-on surgical skill using simulators should be implemented. In the interest of fairness to surgeons, certification must be developed with a goal of not only testing but supporting development of procedural proficiency. As we follow our peers in the airline industry, pathways chosen to validate surgical competence must be evidence-based, fair, and valid.
Flying has become the safest mode of transport. We note that most people drive (and many cycle) more than they fly; nevertheless, the lifetime odds of death as an airplane passenger is 1 in 188,364 compared with 1 in 4047 for a cyclist and 1 in 103 for a person in a car crash.
1
How has air travel become safer? Improved protection by critical flight systems, better air traffic control technology, improved crashworthiness of airplanes, more frequent safety audits, and above all, pilot training. New pilot training protocols have significantly contributed to the safety of air passengers. Regulations limit the time pilots can spend in the cockpit, and pilots must have at least 10 hours off of work before flight duty, including 8 hours of sleep. Rigorous pilot training includes regular simulator training, including line checks; captains in training conduct line checks by observing pilots during simulated air travel while scrutinizing every aspect of pilot performance against a validated checklist. Flight simulators reproduce various and challenging flight situations. Pilots can now learn how to fly without ever leaving the ground.National Safety council
Preventable deaths. Odds of dying.
Preventable deaths. Odds of dying.
https://injuryfacts.nsc.org/all-injuries/preventable-death-overview/odds-of-dying/
Date accessed: September 4, 2019
2
Readwrite
Air travel is safer than you think: Here’s why.
Air travel is safer than you think: Here’s why.
https://readwrite.com/2018/12/21/air-travel-is-far-safer-than-you-think-heres-why/
Date accessed: September 4, 2019
Pilot training consists of basic training as well as type-rating or training to fly a specific aircraft.
3
Type rating can be compared with a driver’s license that is valid for one specific car model alone. The pilot training process parallels how we as surgeons learn our profession. With basic training, we learn the skills that every orthopaedic surgeon should possess, while advanced training fine-tunes a surgeon’s skills to refine the performance of subspecialized procedures such as arthroscopic surgery. Like the one-aircraft approach, a focus on one specific field in orthopaedic surgery narrows one’s focus and specialization. However, this analogy is where the similarities end.Pilots are required to undertake proficiency checks to ensure they continue to be competent conducting particular kinds of operations. An airline pilot needs to complete at least 2 proficiency checks each year.
4
These mandatory checks are regulated by federal law.Australian Government Civil Aviation Authority
Proficiency checks.
Proficiency checks.
https://www.casa.gov.au/standard-page/proficiency-checks
Date accessed: September 4, 2019
5
Without passing these compulsory checks, pilots do not fulfil their licensure requirements and, thus, cannot fly.Federal Aviation Administration
Regulations and policies.
Regulations and policies.
https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentID/1020469
Date accessed: September 4, 2019
Where do arthroscopic and related surgeons stand in comparison? Once we have passed our board examinations, are we allowed to practice without further regulation? Absolutely not. In 1986, the American Board of Orthopaedic Surgery implemented a recertification program with numerous pathways.
6
However, the program has been criticized for not having “kept up with the field.”7
Most of all, the American Board of Orthopaedic Surgery certification and recertification process does not directly evaluate the hands-on surgical skills essential to our profession.Is there something that we can do to improve the surgical certification process? In a word, yes.
Proficiency-based training or PBT, also known as proficiency-based progression or PBP, is a relatively new method that can be used to train surgeons to develop psychomotor skills to safely and efficiently perform surgical procedures.
8
Angelo et al.9
developed a metric-based assessment for an arthroscopic Bankart procedure with 45 steps and 13 phases, allowing characterization of the essential components of the procedure. In a different study, Angelo et al.10
showed that PBT, coupled with simulator training, yielded a 56% lower error rate when performing arthroscopic Bankart repairs than with a traditionally based training method. In addition, the PBT-trained surgeons were 7.5 times more likely to achieve a final benchmark demonstrating proficiency. Other authors have confirmed Angelo et al.’s findings.11
, 12
, 13
, 14
In sum, studies suggest that competency-based simulator training may serve to advance arthroscopic skills for residents, PBT is superior to the traditional apprenticeship model of surgical training, and PBT-based evaluation can be used to evaluate a surgeon’s proficiency in performing a specific surgical procedure. Theoretically, these methods could be used to certify that surgeons possess, and maintain, adequate hands-on skills required to perform surgery safely, efficiently, and with adequate competence.Other research supports this potential approach to assessment of competency. For example, the Fundamentals of Arthroscopic Surgery Training (FAST) program is a collaborative initiative between the Arthroscopy Association of North America and the American Academy of Orthopaedic Surgeons.
15
The FAST program allows task deconstruction and assessment of basic arthroscopic skills. In addition, the Arthroscopic Surgery Skill Evaluation Tool (ASSET) has been developed as a video-based assessment of technical skill that is suitable to evaluate multiple procedures in both the simulation laboratory and operative theater.16
ASSET, and similar tools, have also been shown to be a reliable and valid instrument to evaluate arthroscopic surgical skills.16
, 17
, 18
, 19
In the end, simulators, PBT, FAST, ASSET, and other tools to evaluate surgical proficiency could be potentially analogous to airline pilot’s line check evaluation.Some might argue that these checks and balances are not required for arthroscopic surgeons. Certainly, experienced surgeons make fewer errors, have less sentinel events, and perform surgery faster and more efficiently than novices.
20
However, Pedowitz et al.15
showed that surgeons are not infallible. For example, when checking knot-tying skills at an Arthroscopy Association of North America Resident Course, failure resulted from 24% of all knots performed by course faculty. The authors concluded that there is significant room for improvement, even among some “experienced” surgeons.15
These data support the importance of continuing education and certification.Our Editor-in-Chief has previously suggested that the training curriculum of Angelo et al.
15
can be used as a tool to judge surgical expertise and that trainees whose performance falls below a specific benchmark require additional training to increase proficiency before performing live surgery.21
The next logical step is to use this tool not only for residents but also for board-certified surgeons. Similar to the aviation industry, checklists could be employed to verify competencies.16
,22
,23
Whether the proficiencies tested with the simulator can replicate live surgery is a matter of debate,24
but several studies have suggested that simulator skills can be transferred to the operating room.25
, 26
, 27
, 28
, 29
Could proficiency training improve outcomes for our patients? Tam et al.30
were able to demonstrate that proficiency-based robotic training for board-certified general surgeons reduced operative times and costs for hernia surgery providing a supportive argument.Four years ago, the editors of Arthroscopy wrote that they firmly believed in PBP training using simulators, as well as PBP training using cadavers, as evidence-based teaching tools.
31
We look forward to research supporting the use of such tools for primary and continuing certification to perform orthopaedic surgery.However, difficult questions remain: What if the tests are “unreasonably” challenging, or worse, do not “emulate” what we practice? Can we test specific procedures (such as anterior cruciate ligament reconstruction) given that there are so many technique “variations” resulting in good and excellent outcomes? Alternatively, should we limit testing to the “basics,” such as knot tying, to ensure “fairness?” What if a “proficient” surgeon does not pass the test? What if the test is so easy that an “incompetent” surgeon receives a passing grade? How do we determine the threshold resulting in “certification?” It is critically important that tools used to perform such “high stakes” evaluation must be robust, reliable, validated, and reproducible.
In the preceding paragraph, every sentence includes words in quotation marks, and all but the last sentence end with a question mark. These are obvious warning signs that clarify our assertion: surgical skills evaluation and certification is a goal. Research is required before this goal can be achieved in an evidence-based manner.
In the interest of quality, value, and patient safety, objective evaluation of hands-on surgical skill using simulators should be developed and implemented to improve the application of surgeon licensure certification to clinical patient care. In the interest of fairness to patients and to surgeons, PBT should result in modification of the apprenticeship surgical training model; PBT is evidence-based superior. In all fairness to surgeons, certification must be developed with a goal of not only testing but supporting development of proficiency. As we follow our peers in the airline industry, pathways chosen to validate competence must be proven before implementation.
References
- Preventable deaths. Odds of dying.(Available at:)https://injuryfacts.nsc.org/all-injuries/preventable-death-overview/odds-of-dying/Date accessed: September 4, 2019
- Air travel is safer than you think: Here’s why.(Available at:)https://readwrite.com/2018/12/21/air-travel-is-far-safer-than-you-think-heres-why/Date accessed: September 4, 2019
- Pilot training: What can surgeons learn from it?.Arab J Urol. 2014; 12: 32-36
- Proficiency checks.(Available at:)https://www.casa.gov.au/standard-page/proficiency-checksDate accessed: September 4, 2019
- Regulations and policies.(Available at:)https://www.faa.gov/regulations_policies/orders_notices/index.cfm/go/document.information/documentID/1020469Date accessed: September 4, 2019
- Continuing competency in orthopaedics: The future of recertification.Clin Orthop Relat Res. 2006; 449: 72-75
- Not the last word: The time to fix ABOS recertification has arrived.Clin Orthop Relat Res. 2018; 476: 1928-1930
- Editorial commentary: Proficiency-based training: Are we ready for a new way to train and test orthopaedic surgeons?.Arthroscopy. 2018; 34: 2199-2200
- Metric development for an arthroscopic Bankart procedure: assessment of face and content validity.Arthroscopy. 2015; 31: 1430-1440
- A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set.Arthroscopy. 2015; 31: 1854-1871
- Arthroscopy skills development with a surgical simulator: A comparative study in orthopaedic surgery residents.Am J Sports Med. 2015; 43: 1526-1529
- Simulation training improves surgical proficiency and safety during diagnostic shoulder arthroscopy performed by residents.Orthopedics. 2016; 39: e479-e485
- Efficacy of standardized training on a virtual reality simulator to advance knee and shoulder arthroscopic motor skills.Orthopedics. 2016; 39: e479-e485
- Knee, shoulder, and fundamentals of arthroscopic surgery training: Validation of a virtual arthroscopy simulator.Arthroscopy. 2017; 33: 641-646
- Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester.Arthroscopy. 2015; 31: 1872-1879
- Assessing diagnostic arthroscopy performance in the operating room using the arthroscopic surgery skill evaluation tool (ASSET).Arthroscopy. 2015; 31: 2314-2319
- Diagnostic knee arthroscopy: A pilot study to evaluate surgical skills.Arthroscopy. 2012; 28: 218-224
- Validation of a global assessment of arthroscopic skills in a cadaveric knee model.Arthroscopy. 2013; 29: 106-112
- Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair.Arthroscopy. 2013; 29: 906-912
- The Bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill.Arthroscopy. 2015; 31: 1639-1654
- Expert surgeons can be distinguished from trainees, and surgical proficiency can be defined, using validated metrics and shoulder models.Arthroscopy. 2015; 31: 1635-1636
- A checklist intervention to assess resident diagnostic knee and shoulder arthroscopic efficiency.J Surg Educ. 2017; 74: 9-15
- Inter-rater reliability for metrics scored in a binary fashion—Performance assessment for an arthroscopic Bankart repair.Arthroscopy. 2018; 34: 2191-2198
- Virtual reality simulation in orthopedics and trauma surgery in France: Current status and perspectives [in German].Unfallchirurg. 2019; 122: 439-443
- Performance on a virtual reality DHS simulator correlates with performance in the operating theatre.Surg Technol Int. July. 2019; 20 ([Epub ahead of print.])
- Transferring simulated arthroscopic skills to the operating theatre. A randomised blinded study.J Bone Joint Surg Br. 2008; 90: 494-499
- Objectively assessing intraoperative arthroscopic skills performance and the transfer of simulation training in knee arthroscopy: A randomized controlled trial.Arthroscopy. 2019; 35: 1197-1209
- Transfer validity of laparoscopic knot-tying training on a VR simulator to a realistic environment: A randomized controlled trial.Surg Endosc. 2008; 22: 1636-1642
- Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance.J Am Coll Surg. 2005; 200: 546-551
- Proficiency-based training and credentialing can improve patient outcomes and decrease cost to a hospital system.Am J Surg. 2019; 217: 591-596
- Arthroscopic training in the 21st century: A changing paradigm.Arthroscopy. 2017; 33: 1913-1915
Article info
Identification
Copyright
© 2019 by the Arthroscopy Association of North America