Assessing clinical decision support system tools in precision oncology: piloting ring testing

Abstract

Background The EU4Health project PCM4EU aimed to improve survival rates and quality of life of patients with cancer based on precision cancer medicine. To achieve this, enhanced expertise and quality of molecular cancer diagnostics are key. Clinical decision support systems (CDSS) have become increasingly important after the introduction of comprehensive genomic diagnostic profiling. While external quality assessment schemes are mandatory for most diagnostic laboratory tests, similar programs for CDSS tools are currently lacking. To address this, we piloted an international ring test documenting CDSS usage, performance, and manual interpretation. Materials and methods Twenty synthetic datasets were generated, mimicking small variant call sets from a typical targeted 500-gene panel (VCF format) across multiple cancer types (10 tumour-normal pairs and 10 tumour-only). Participants received standardised instructions via e-mail and at a virtual meeting and submitted results using a structured response form. Results Eight laboratories from seven countries participated. All participants submitted results for the 10 tumour-only cases; one submitted results from two assessors. Tumour-only cases contained 6-18 variants where interpretation could be critical. Oncogenic calls for hotspot variants showed good agreement across the various CDSS tools applied; however, variability existed regarding reported variants and clinical interpretation. Conclusion The pilot ring test revealed clinically relevant discrepancies between laboratories and interpreters, underscoring the need for structured external quality assessment schemes for CDSS tools in addition to the existing laboratory workflow schemes. It also highlighted several challenges related to the generation of realistic synthetic data, the design of reporting formats, the definition of ground truth, and the manual interpretation of results.