Purpose or Case Report: To assess the adoption and performance of a deep learning-based bone age interpretation tool deployed into the clinic over a period of 4 years.
Methods & Materials: The retrospective study was IRB approved. Analysis was performed using 10,529 pediatric radiologist-reported bone age hand radiograph exams performed between May 2021 and September 2025 from our clinical PACS system. For 7,352 of these exams, the radiologist was provided with a prepopulated imaging report with numerical bone age estimates from a previously published deep-learning bone age assessment algorithm, at which point the radiologist had the option to modify or accept the automated estimate. The original algorithm estimates were archived separately and were retrieved to compare against the final signed-off assessments stored in the PACS system.
Results: Individual radiologists accepted the deep-learning model estimates at frequencies ranging from 64% to 81%. When averaged across all radiologists, the overall acceptance rates slightly fluctuated over time, varying between 74% and 79%, with no statistically significant trend (p=0.88), suggesting rapid adoption and sustained usage by clinicians. Mean absolute error (MAE) between final radiologist reported bone ages and preliminary algorithmic assessments averaged 4.15 months over the study period, and trended downward slightly over time (4.19 months for year 1 vs. 3.82 months for year 4), although this trend was not statistically significant when evaluated year-by-year (p=0.19) or month-by-month (p=0.08).
Algorithmic bias was minimal, with a nearly balanced distribution of overestimations (48.1%) and underestimations (51.9%). Cases with significant adjustments (>12 months under or over) accounted for 55.2% of radiologist changes in total.
Conclusions: A deep-learning model for bone age assessment demonstrated excellent performance, although radiologist adjustments remain relatively common. Further model development and optimization are needed to continue to improve model accuracy. Our findings emphasize that continuous, real-world performance monitoring is important for understanding and refining clinical AI tools, building clinician confidence, and improving patient outcomes.