Diagnostic Performance of Artificial Intelligence and Radiologists in 3D Automated Breast Ultrasound System (ABUS) Interpretation: A Pilot Study

Abstract

Background and objectives: Breast cancer is currently the most commonly diagnosed cancer among women worldwide. In 2022, the World Health Organization reported approximately 2.29 million new cases and 666,103 deaths. In Thailand, breast cancer similarly ranks as the most frequently diagnosed cancer in women, with approximately 21,628 new cases and 7,599 deaths. Mammography, the standard screening tool, has limitations in women with extremely dense breast tissue, showing only 61% sensitivity compared to 86% in fatty breasts. 3D Automated Breast Ultrasound System (3D ABUS) is a technology suitable for dense breasts that provides standardized imaging. Artificial intelligence (AI) integration may improve diagnostic accuracy, reduce false positives, and support areas with limited access to breast imaging specialists. This study aimed to evaluate the accuracy, sensitivity and specificity of artificial intelligence in classifying breast cancer risk from 3D ABUS images compared with radiologists of different expertise levels and experience, and to study advantages and limitations of radiologists versus artificial intelligence (AI) in breast cancer screening using automated breast ultrasound (3D ABUS).

Methods: This retrospective study included 50 three-dimensional automated breast ultrasound (3D-ABUS) images from the TDSC-ABUS2023 dataset, without stratification by breast tissue density, comprising 25 benign and 25 malignant cases. Three radiologists participated: a breast imaging specialist (R1), a radiologist with over 15 years of experience (R2), and a radiologist with less than 15 years of experience (R3). Cases 1–10 were designated as a familiarization set to allow radiologists to acclimate to the imaging characteristics and were excluded from the final analysis. The remaining 40 cases (cases 11–50) constituted the evaluation set, in which radiologists were tasked with classifying the lesions as either benign or malignant along with recording the time spent interpreting each case (in seconds) and their level of diagnostic confidence using a confidence score rated on a scale of 1 to 10. R3 analyzed the images both independently and with AI assistance (R3&AI), with an inter-reading washout period of approximately 9 months. 3D Slicer software was used for image analysis. The AI system employed a combined model of Faster R-CNN with ResNet50 for detection, U-net for segmentation, and Radiomics with Machine Learning for classification. Accuracy, sensitivity, specificity, reading time, confidence scores AUC and statistical significance of Diagnostic performance were measured.

Results: In terms of diagnostic accuracy, R1 and AI demonstrated the highest accuracy (72.50%), followed by R3&AI (65.00%), R3 (62.50%), and R2 (60.00%). Of the 40 evaluated cases, 12 cases (30.0%) were correctly classified by all reader groups and AI. For sensitivity, R1 had the highest at 80.00%, followed by R3, R3&AI, and AI (75.00%), and R2 (70.00%). For specificity, AI had the highest at 70.00%, followed by R1 (65.00%), R3&AI (55.00%), and R2 and R3 (50.00%). AUC values for R1 and AI were highest (0.7250), followed by R3&AI (0.6500), R3 (0.6250), and R2 (0.6000). Comparison of diagnostic performance between each group and the AI revealed no statistically significant differences across all metrics (p > 0.05 for all groups). Pairwise comparisons of AUC values between radiologist groups and AI, as well as among radiologist groups themselves, showed no statistically significant differences in any comparison (p > 0.05). The interobserver reliability of confidence scores among radiologists was assessed using the intraclass correlation coefficient (ICC), which was 0.069 (95%CI: -2.543–0.976, p = 0.345) Reading times for R1, R2, R3 and R3&AI were 47.40, 47.05, 53.70, and 41.20 seconds, respectively, R3&AI condition demonstrated 23.28% reduction in reading time and no statistically significant differences (p > 0.05).  The AI required an average processing time of 12–15 seconds per case. R3&AI showed the highest confidence score (median = 8.0) and greatest consistency (IQR = 2.0).

Conclusions: AI demonstrated accuracy equivalent to breast imaging specialists and the highest specificity, reducing false positive diagnoses. AI can enhance the performance of less experienced radiologists by improving accuracy, specificity, and confidence. However, breast imaging specialists maintained the highest sensitivity for cancer detection. Furthermore, automation bias was observed in the R3&AI group, where increased confidence scores did not correspond to improved diagnostic accuracy, representing an important caution for clinical AI implementation. This study demonstrates the potential of integrating 3D ABUS with AI in breast cancer screening programs, especially in regions facing a shortage of subspecialized radiologists. However, policy-level implementation warrants further validation through large-scale prospective studies in real-world populations.