Explainable AI-based Real-Time Anomaly Detection in Server Logs using Isolation Forest and XGBoost

Abstract

The current state of IT infrastructure creates a huge volume of logs, which cannot be effectively monitored manually nor detected by rule-based methods due to rapidly changing cyber threats. Therefore, this paper presents a solution based on explainable AI approaches that focuses on detecting anomalies in server logs using a specialized LogHub HDFS v1 benchmark dataset containing 113,896 labeled blocks. For this purpose, several statistical aggregations and log vectorizations are performed to calculate 34 behavioral features (for example, Shannon Entropy or rare event ratio). The proposed method includes a hybrid engine that incorporates an unsupervised Isolation Forest algorithm used for discovering new threats and a supervised model called XGBoost that helps detect known failure types. In order to increase transparency of such black box models, SHAP explanations were used. By leveraging SHAP explanations, specific log events that triggered alerts can be determined (such as write exceptions, E2, or verification failures, E23). As per rigorous evaluation, it is seen that whereas the isolated forest alone provided a ROC-AUC score of 0.99, the XGBoost model attained near perfect accuracy with an F1-Score of 99.84%. The resultant combined decision-making process ensures the best possible compromise between security, resulting in an accurate score of 97.40% with the extremely important Recall rate of 99.97% such that just one of the 3,368 anomalies were missed. With the ability to provide valuable insights at the milliseconds’ notice, the system converts automated log analysis into meaningful decision support.