Jun 2026· Proceedings of the ACM on Software Engineering· Vol 3, pp. 4482 - 4504· 0 citations· 38 references
TL;DR
Lingfuzz is a linguistic-aware fuzzing framework for continuing red teaming LLMs that enables generating diverse malicious instructions due to the unlimited space of lexical and syntactic choices, while having a continued testing capability by iteratively mutating the mutants.
Abstract
Safety alignment aims to prevent Large Language Models (LLMs) from producing harmful content. However, safety alignment remains vulnerable to malicious instructions. Red teaming is a critical methodology for identifying such vulnerabilities in LLMs. Existing approaches often rely on jailbreak templates or rule-based transformation, limiting the diversity of generated tests and the continued testing capability of these test approaches. To address these limitations, we propose Lingfuzz, a linguistic-aware fuzzing framework for continuing red teaming LLMs. The key idea of this framework is to mutate the existing malicious instructions at the lexical and syntactic levels, while keeping the malicious intentions of the instructions. Such mutations enable generating diverse malicious instructions due to the unlimited space of lexical and syntactic choices, while having a continued testing capability by iteratively mutating the mutants. We evaluated Lingfuzz on five aligned commercial LLMs and one open-source LLM in black-box testing. The results show that Lingfuzz triggers safety alignment vulnerabilities in 71.0% of the cases in JailbreakBench benchmark, higher than the second-highest baseline of 63.8%. Lingfuzz also possesses outstanding multilingual testing capabilities that far exceed other baselines with approximately 24% ESR advantage testing in Jailbench benchmark. The malicious instructions generated by Lingfuzz have almost three times higher diversity than previous work according to the self-BLEU metric. Lingfuzz also demonstrates strong continued testing capability by showing five times less sensitivity to the LLM evolution than other approaches.
Findings show that insecure code generation is not merely a collection of independent defects, but a structured and prompt-conditioned phenomenon, motivating cluster-aware verification and prompt-level intervention for safer LLM-assisted programming.
The Security Mutation Score (SMS) is proposed, a metric that classifies mutant kills into semantic, functional, incidental, and crash categories using operator-aware heuristics, distinguishing genuine security awareness from coincidental detection.
: As modern software systems grow in complexity, the number of vulnerabilities has grown rapidly, rendering traditional security strategies—including manual code reviews and rule-based static analyzers—increasingly resource-intensive and unable to keep pace with emerging threats. This paper presents a hybrid pipeline combining static vulnerability candidate selection using Latent Semantic Analysis with dynamic verification through Large Language Model-assisted fuzzing. The approach identifies semantically similar code patterns to known vulnerabilities without exact syntactic matches, and automates the generation of fuzzing harnesses and seed inputs to overcome coverage plateaus. We evaluate our pipeline on real-world C libraries (LibTIFF, libexpat, FreeType) with documented vulnerabilities. Results demonstrate that vulnerability candidates are ranked within the Top 3–8% of all functions, and model-generated artifacts yield a 5-fold speedup over random fuzzing. The LLM Gemini 3 Pro achieves the highest harness generation success rate at 83% (Pass@3). Our analysis reveals fundamental limitations: the mixed-vector problem dilutes vulnerability signals in multifunctional code, safety alignment mechanisms can inadvertently neutralize vulnerabilities through defensive code insertion, and static feature extraction without build context achieves only 19.93% recall. The pipeline reduces manual effort while still requiring human validation.
Kevin Nguyen, Dominik Schoop· Proceedings of the 23rd Inte...· 0 citations
This work systematically generates syntactic variants of security-relevant code generation prompts and evaluates their impact on code security across multiple open LLMs and programming languages, identifying prompt syntax as a concrete security control surface and providing actionable guidance for reducing vulnerability risk in LLM-assisted development.
Matteo Cicalese, Antonio Della Porta, Stefano Lambiase et al.· 0 citations
Large Language Models (LLMs) are widely used for automated code generation. Their reliance on infrequently updated pretraining data can leave them unaware of newly discovered vulnerabilities and evolving security standards, making them prone to producing insecure code. In contrast, developer communities on Stack Overflow (SO) provide an ever-evolving repository of knowledge, where security vulnerabilities are actively discussed and addressed through collective expertise. These community-driven insights remain largely untapped by LLMs. This paper introduces SOSecure, a Retrieval- Augmented Generation (RAG) system that leverages the collective security expertise found in SO discussions to improve the security of LLM-generated code. We build a security-focused knowledge base by extracting SO answers and comments that explicitly identify vulnerabilities. Unlike common uses of RAG, SOSecure triggers after code has been generated to find discussions that identify flaws in similar code. These are used in a prompt to an LLM to consider revising the code. Evaluation across three datasets (SALLM dataset, LLMSecEval, and LMSys) shows that SOSecure achieves strong fix rates of 71.7%, 91.3%, and 96.7% respectively, compared to prompting GPT-4 without relevant discussions (49.1%, 56.5%, and 37.5%), and outperforms multiple other baselines. SOSecure operates as a language-agnostic complement to existing LLMs, without requiring retraining or fine-tuning, making it easy to deploy. Our results underscore the importance of maintaining active developer forums,
Manisha Mukherjee, Vincent J. Hellendoorn· AIware· 0 citations