Optimising Pharmacovigilance Efficiency with MLIT (Machine Learning for Intelligent Triage): A Tool for Statistical Safety Alerts.
TL;DR
This study highlights the potential of ML-based tools to improve pharmacovigilance by enhancing signal detection performance, reducing the likelihood of missed signals, while increasing operational efficiency, and strengthening reproducibility and transparency.
Abstract
Background
AND
Aim
Pharmacovigilance is essential to ensuring patient safety by enabling timely identification of adverse reactions in increasingly complex and voluminous data. Routine quantitative signal detection methods generate statistical alerts for product-event pairs based on predefined criteria; however, most alerts do not warrant further investigation, creating inefficiencies and significant time demands for pharmacovigilance teams. Manual triage of these alerts is often resource-intensive, prone to variability, and challenging to audit, highlighting the need for more reliable, transparent and efficient triage strategies. This study aimed to design, develop and prospectively evaluate an explainable Machine Learning for Intelligent Triage (MLIT) tool to assist pharmacovigilance teams in reviewing statistical alerts for vaccine and drug portfolios. The objective was to enhance signal detection performance without increasing the risk of missing signals, improving operational efficiency and maintaining decision traceability and regulatory compliance.
Methods
Alert and individual case safety report data were retrieved from the company's safety and signal management databases. Feature selection was guided by prior experience with a published case completeness tool, called Clinical Utility Score for Prioritisation (CUSP), and expert input. Of several ML methods explored, eXtreme Gradient Boosting (XGBoost) emerged as the optimal algorithm, with models trained and tested using a 75/25 split dataset. Iterative model refinement was conducted using Shapley Additive Explanations analyses to ensure explainability and alignment with safety reviewers' decision-making processes. Refined models underwent prospective validation in two four-month prospective validation studies, covering over 20 products across vaccine and drug portfolios. The prospective validations assessed concordance between model predictions and reviewers' decision under real-world conditions, as well as estimated time savings.
Results
The vaccine model demonstrated robust predictive performance, achieving a weighted-average F1 score of 0.81 and an accuracy of 0.79. In the prospective validation phase, 92% of vaccine alerts were closed in alignment with the model's top-ranked prediction, while 98% were closed within the top 3 predictions. The MLIT tool also identified inconsistencies and human errors in manual triage, highlighting its potential role as a quality-control mechanism. Safety reviewers reported a 24% reduction in time spent on triage activities, and explainability analyses confirmed that the model's decision-making was conceptually aligned with safety reviewers' logic. Comparable results were observed for the drug portfolio.
Conclusion
This study highlights the potential of ML-based tools to improve pharmacovigilance by enhancing signal detection performance, reducing the likelihood of missed signals, while increasing operational efficiency, and strengthening reproducibility and transparency. While MLIT demonstrated high concordance with expert decisions and provided meaningful time savings, human oversight remains essential, especially for low-confidence predictions. Ongoing refinement and user engagement will be critical for broader implementation and further automation, marking a significant step forward in ensuring safer and more efficient drug safety surveillance.