Purpose or Case Report: We intended to develop and test a PACS based radiology simulation program as an objective tool to evaluate residents. We utilized the simulator as the pediatric “end of rotation” (EOR) exam and compared it to our institution’s traditional EOR review. We hypothesized that a simulator evaluation tool would better identify areas for residents to improve, create a clear gradient of resident performance, or identify a struggling resident.
Methods & Materials: The simulator we employed was developed internally and very closely approximates the appearance and functionality of the PACS in use at the hospital. The simulation cases are real cases that have been wiped of all patient identifiers. The simulator contains the entire study, closely mimicking the true PACS environment. Each resident was given the same 12 cases with a one hour time limit. Residents were instructed to provide a preliminary report for the study and complete as many studies as possible within one hour. Four senior residents took the simulation EOR exam: 2 PGY-5 residents and 2 PGY-4 residents. Three junior residents took the simulation EOR exam: 3 PGY-2 residents. The attending evaluator compared the preliminary report provided by the resident to the findings and impression of the final report for that study. A grading system was devised: correct, incorrect, and not attempted. A passing score was set as percentage correct greater than 60%. Our institution’s traditional EOR exam, where a resident takes a number of cases from Powerpoint slides one on one with the attending evaluator, was compared to the simulator. The traditional EOR exam is subjectively scored as pass/fail.
Results: All residents completed 12 cases in the allotted time. The percentage correct ranged from 67% to 92% with a mean score of 83% and standard deviation of 8%. All residents passed the simulator exam. The senior residents’ scores ranged from 83% to 92% with a mean of 87.5% and standard deviation of 5%. The junior residents’ scores ranged from 67% to 83% with a mean of 78% and standard deviation of 9%. The case most missed by junior residents: VCUG demonstrating a posterior urethral valve. The case most missed by the senior residents: normal elbow radiograph with multiple ossifications.
Conclusions: The simulator provided objective data allowing the evaluator to develop a clear distribution of resident performance as well as identify areas needing improvement. We demonstrated that the PACS based simulator is a superior evaluation tool for an EOR exam.