Structured ACR Bone-RADS Versus Unstructured Assessment of Bone Tumors on Radiographs: A Multicenter, Multireader Study Across Experience Levels.
PURPOSE To compare the diagnostic performance and interreader agreement of structured ACR Bone Reporting and Data System (Bone-RADS) versus unstructured assessment of bone tumors on radiographs and to assess their effects on management recommendations. MATERIALS AND METHODS Multicenter, multireader study with a primary retrospective cohort (n = 1,423; 4 centers), external retrospective cohort (n = 354; 6 centers), and prospective cohort (n = 152; 2 of these 10 centers; written informed consent obtained from all prospective participants). After a standardized training session with practice cases, nine musculoskeletal radiologists (early career 3-5 years; midcareer 15-20 years; late career >20 years) independently performed unstructured classification and Bone-RADS 1 to 4 grading with a standardized 4-week washout. Primary end point was reader-averaged difference in area under the curve (ΔAUC) via an Obuchowski-Rockette-Hillis multireader, multicase receiver operating characteristic model; interreader agreement and management recommendation reclassification were also assessed. RESULTS Overall reader-averaged AUC did not differ significantly between unstructured assessment and Bone-RADS across the primary retrospective, external retrospective, and prospective cohorts (ΔAUC: -0.0012, 0.0160, and 0.0053, respectively; all P > .05). However, early-career readers demonstrated significant AUC improvements with Bone-RADS in the primary and external cohorts ΔAUC: +0.0302 and +0.0615; both P < .001), with a similar but nonsignificant difference in the prospective cohort (ΔAUC = +0.0201; P = .572). In contrast, midcareer readers showed nonsignificant changes, and late-career readers exhibited slight performance declines. Interreader agreement increased across all cohorts (eg, in the prospective cohort, early-career Fleiss κ improved from 0.394 to 0.502). Feature-level agreement was highest for pathologic fracture but lower for margins and endosteal erosion (κ ∼0.518-0.660). Malignancy rates increased monotonically across Bone-RADS categories 1 to 4 (9.44%, 25.44%, 46.67%, and 79.67%, respectively). Among early-career readers, Bone-RADS shifted management recommendations by increasing referral or further workup recommendations for potentially malignant lesions in the primary (+8.61%), external (+15.30%), and prospective (+7.14%) cohorts, while concurrently reducing routine surveillance recommendation rates for benign lesions (-17.27%, -15.90%, and -19.51%, respectively). CONCLUSION Bone-RADS provides stable risk stratification, improves diagnostic performance and interreader agreement in early-career readers, and shifts management recommendations toward greater referral or workup of potentially malignant lesions. Its sensitivity-oriented design may increase overmanagement of some benign lesions. To mitigate this, future refinements should consider incorporating age and anatomic-site information, adopting context-aware management thresholds, and providing structured training on lower-agreement features such as margin classification and endosteal erosion.