Skip to content
#machine learning #cybersecurity Conference Open access

What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs

Sep 2025 · Network and Distributed System Security Symposium · Vol abs/2509.22796 · 1 citation · 80 references
Computer Science

TL;DR

DUALLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model, achieves 87.4% accuracy and an F1-score of 0.875, significantly outperforming prior solutions.

Abstract

Open-source software projects are foundational to modern software ecosystems, with the Linux kernel standing out as a critical exemplar due to its ubiquity and complexity. Although security patches are continuously integrated into the Linux mainline kernel, downstream maintainers often delay their adoption, creating windows of vulnerability. A key reason for this lag is the difficulty in identifying security-critical patches, particularly those addressing exploitable vulnerabilities such as out-of-bounds (OOB) accesses and use-after-free (UAF) bugs. This challenge is exacerbated by intentionally silent bug fixes, incomplete or missing CVE assignments, delays in CVE issuance, and recent changes to the CVE assignment criteria for the Linux kernel. While fine-grained patch classification approaches exist, they exhibit limitations in both coverage and accuracy. In this work, we identify previously unexplored opportunities to significantly improve fine-grained patch classification. Specifically, by leveraging cues from commit titles/messages and diffs alongside appropriate code context, we develop DUALLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model. DUALLM achieves 87.4% accuracy and an F1-score of 0.875, significantly outperforming prior solutions. Notably, DUALLM successfully identified 111 of 5,140 recent Linux kernel patches as addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification (many do not have clear indications in patch descriptions). Moreover, we constructed proof-of-concepts for two identified bugs (one UAF and one OOB), including one developed to conduct a previously unknown control-flow hijack as further evidence of the correctness of the classification.

Read PDF

Similar papers

Preprint Aug 2026

eBPF Security in the Wild: Structural Concentration, Failure Mechanisms, and Discovery Gaps

Extended Berkeley Packet Filter (eBPF) is a security-critical in-kernel execution framework, yet its vulnerability landscape remains fragmented across components, semantic gaps, and testing techniques. We present an empirical study of observed eBPF vulnerabilities. We construct a multi-source dataset from Linux kernel...

Bai-Hong Chen, Ming Hua, Wei-Feng Pan et al. · 0 citations
#artificial intelligence Preprint Sep 2026

The History Is the Detector: Executing CVE Patch History, End-to-End

BUGSTONE-E2E, a framework that transforms vulnerability history into executable detection rules and validates their findings, demonstrates that CVE history can be turned into an executable workflow, transforming past vulnerabilities into reproducible detection and repair.

Qiu-Shi Wu, Kevin Eykholt, Youngja Park et al. · 0 citations

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.

Om Nepal, Sushant Aryal, Oluseyi Olukola et al. · 0 citations
Book Open access Aug 2026

Benchmarking Automated Security Patch Backporting: How Far Are We?

This work presents Porting Benchmark, a curated dataset of 1,234 security patch backporting cases spanning cross-version, cross-branch, and cross-repository scenarios, paired with a common evaluation framework and identifies four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-...

Jin-Cheng Yang, Yu-Long Fu, Cheng-Wei Liu et al. · 1 citation
#large language models Open access Sep 2026

Fixing, Breaking, or Faking It? An Execution-Calibrated Evaluation of LLM Vulnerability Patching in JavaScript, Python, Go, and Java, and the Limits of LLM-as-Judge

Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness, leaving no single judge trustworthy.

Patrick Deininger, Wolfgang Slany · 1 citation
Preprint Oct 2026

On the Reliability of LLM-Based Vulnerability Patching Benchmarks

These results show that benchmark scores are highly sensitive to evaluation design, and practical guidelines for more rigorous, reliable, and reproducible evaluation are provided.

Dang K. Le, Wen-Xuan Shi, Xin-Yu Xing · 0 citations

Related blog posts

Microsoft Research Blog Nov 3, 2026

Learn the rules: The hacker mindset

Research Manager Mike Walker reflects on a career spanning hacker culture, cybersecurity, and AI, sharing lessons about trust, teamwork, leadership, and why solving difficult problems requires curiosity, judgment, agility—and an understanding of the rules. The post Learn the rules: The hacker mindset appeared first on Microsoft Research.

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.