The Role of Prompt Patterns in LLM-Based Smart Contract Vulnerability Detection
Abstract
Smart contracts are immutable programs deployed on blockchain platforms, and their vulnerabilities can cause severe financial losses. Therefore, several tools have been proposed to detect these vulnerabilities. Among them, recent research has increasingly leveraged Large Language Models (LLMs) relying on carefully crafted prompts. Meanwhile, prior work has identified several prompt patterns, such as persona roles, chain-of-thought reasoning, and multi-agent interactions. However, the sensitivity of LLM performance to prompt design raises questions about the reliability of existing evaluations. This paper investigates how prompt patterns affect the effectiveness of LLMs in detecting smart contract vulnerabilities. Building on the zero-shot prompt used by Chen et al., which combines persona and chain-of-thought (CoT) patterns, we conducted an ablation study to measure the impact of removing specific patterns on detection accuracy. We further extended the original prompt with few-shot examples to evaluate potential performance gains over the baseline. Finally, we investigated the impact of prompt patterns on a reasoning model. The results show that LLMs without reasoning mechanisms obtain limited benefits from persona and CoT patterns, whereas few-shot prompting achieves the best precision-recall balance. In reasoning-enabled LLMs, simple task-definition prompts outperform persona and CoT patterns, whereas few-shot examples further improve precision and generalization.