Towards an Automated Test of LLM Security Knowledge

CWEEP: A Lexical Static Analysis Framework for CWE Early Prevention

As the hardware layer becomes a focus point for attackers, the need for improved hardware security verification techniques is more important than ever. State-of-the-art security verification techniques require significant manual effort from individuals with security expertise. Furthermore, there is no standard method to locate where the fault lies within the register transfer level (RTL) code. This paper presents CWEEP, a static analysis framework for detecting security weaknesses in RTL. CWEEP does not require a detailed security specification, so it can be used in the early stages of RTL development while properties are still under construction. Furthermore, CWEEP can identify the exact location in the RTL where the potential vulnerability resides and supports automatic code repair suggestions when applicable. Using datasets from the literature, we evaluate the performance of CWEEP on a set of two SoC designs with manually inserted bugs and on a large language model generated dataset, consisting of 3874 buggy modules. We find that CWEEP issues a correct warning up to 60.8% of the time. In contrast, the tool from a previous work issued a correct warning 17.5% of the time for the same dataset.

An Exploratory Study on Code Smells Detection and Refactoring using LLMs

With the observed progress in machine learning (ML), and particularly the introduction of Large Language Models (LLMs), several activities related to code maintenance could be automated. That includes not only detection and evaluation of design flaws, but also code transformation and refactoring. However, the general-purpose LLMs, while being commonly used and popular, have not been specifically trained for code analysis, and may not be suitable for conducting software maintenance tasks due to biases, and inherent shortcomings of the models. In this paper, we explore if the widely available LLMs could aid the detection and the refactoring of code smells. We focus on four common smells (God Class, Long Method, Feature Envy, and Refused Bequest) and consider five prompts of diverse complexity, asking the model for detecting and removing the identified code smells. Results suggest that general-purpose LLMs cannot be reliably used for that. They can effectively detect or remove code smells only in simple cases, and frequently produce invalid code. However, their performance depends on various factors, e.g., the model, the specific code smell or the prompt objective and composition.

Optimizing Snyk AI Results with Large Language Model (LLM) to Validate False Positive Rate

SOSecure: The Wisdom of the Crowd for Safer AI-Generated Code

Large Language Models (LLMs) are widely used for automated code generation. Their reliance on infrequently updated pretraining data can leave them unaware of newly discovered vulnerabilities and evolving security standards, making them prone to producing insecure code. In contrast, developer communities on Stack Overflow (SO) provide an ever-evolving repository of knowledge, where security vulnerabilities are actively discussed and addressed through collective expertise. These community-driven insights remain largely untapped by LLMs. This paper introduces SOSecure, a Retrieval- Augmented Generation (RAG) system that leverages the collective security expertise found in SO discussions to improve the security of LLM-generated code. We build a security-focused knowledge base by extracting SO answers and comments that explicitly identify vulnerabilities. Unlike common uses of RAG, SOSecure triggers after code has been generated to find discussions that identify flaws in similar code. These are used in a prompt to an LLM to consider revising the code. Evaluation across three datasets (SALLM dataset, LLMSecEval, and LMSys) shows that SOSecure achieves strong fix rates of 71.7%, 91.3%, and 96.7% respectively, compared to prompting GPT-4 without relevant discussions (49.1%, 56.5%, and 37.5%), and outperforms multiple other baselines. SOSecure operates as a language-agnostic complement to existing LLMs, without requiring retraining or fine-tuning, making it easy to deploy. Our results underscore the importance of maintaining active developer forums,

A Comparative Analysis of Security Vulnerabilities and Defense Mechanisms in Large Language Models

Large Language Models (LLMs) are now deployed at an unprecedented scale across many critical sectors, rapidly transitioning from experimental AI tools to embedded components of production software systems. This accelerated adoption, often enabled by low-code integrations, has lowered technical barriers while simultaneously expanding the attack surface of modern applications, particularly when deployments occur without sufficient domain-specific security expertise. In many cases, security maturity has not progressed at the same pace as capability expansion, creating systemic exposure across confidentiality, integrity, and availability dimensions. To provide structured clarity amid this rapid growth, this paper presents a comparative and standards-aligned analysis of LLM security risks and defense mechanisms grounded in the OWASP GenAI Top-10 (2025). We systematically examine each vulnerability class, map representative attack patterns to primary mitigation strategies, evaluate their security property impact, and analyze practical limitations and implementation trade-offs. In addition, we introduce a severity-based assessment to prioritize risks according to operational and systemic impact, offering a quantitative perspective on defensive readiness. Our findings indicate that current mitigation strategies are predominantly reactive, concentrated at inference time, and unevenly distributed across the LLM lifecycle. Controls addressing training pipelines, supplychain dependencies, and autonomous system behaviors remain comparatively less mature and less standardized. By integrating vulnerability classification, defense mapping, severity prioritization, and trade-off analysis within a unified framework, this study provides actionable guidance for strengthening secure, resilient, and standards-driven LLM deployment in high-stakes environments.

The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs

This work systematically generates syntactic variants of security-relevant code generation prompts and evaluates their impact on code security across multiple open LLMs and programming languages, identifying prompt syntax as a concrete security control surface and providing actionable guidance for reducing vulnerability risk in LLM-assisted development.