Skip to content
Conference Open access

Evaluating the effectiveness of the locally deployed GPT-OSS model in detecting SMB and RDP brute-force attacks from Windows security logs

2026 · Zbornik radova · 0 citations · 3 references

Abstract

Brute-force attacks targeting Server Message Block (SMB) and Remote Desktop Protocol (RDP) services remain among the most common techniques used to gain unauthorized access to Windows systems. Traditional detection approaches are typically based on manually defined rules and event correlation mechanisms, which may be difficult to maintain and adapt to evolving attack patterns. Recent advances in Large Language Models (LLMs) have demonstrated promising capabilities in log analysis and anomaly detection; however, their applicability to authentication-based attack detection in Windows environments remains insufficiently explored. This paper evaluates the effectiveness of a locally deployed GPT-OSS model in detecting SMB and RDP brute-force attacks from Windows Security logs. A controlled experimental environment was established consisting of a Windows host with SMB and RDP services enabled, a Kali Linux attack host executing brute-force attacks using Hydra, and an NXLog-based log collection pipeline. Authentication-related security events were preprocessed and analyzed using a context-aware approach in which the model received a sliding window of recent log entries and classified the latest event as benign or malicious. The experimental results demonstrate that the proposed approach achieves high detection performance, obtaining F1 scores of 0.999 for SMB and RDP brute-force attacks in contextual judgement. The findings indicate that combining targeted preprocessing of authentication logs with context-aware reasoning performed by a Mixture-of-Experts (MoE) LLM enables accurate identification of attack patterns without relying on handcrafted detection rules. The main contributions of this work include the development of a reproducible framework for LLM-based analysis of Windows Security logs, the creation of a labeled dataset of SMB and RDP brute-force events, and the experimental validation of GPT-OSS as an effective locally deployable solution for real-time security monitoring.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.