This paper evaluates five prompt engineering strategies across five state-of-the-art open-source LLMs, and compares them against the same models fine-tuned with QLoRA on a standardized multi-label dataset of real-world Ethereum contracts to provide actionable guidance for researchers and practitioners designing LLM-based security tools for Solidity smart contracts.
Abstract
Large Language Models (LLMs) are increasingly being deployed for smart contract security, yet a fundamental question remains unresolved for practitioners: when confronted with the realistic, multi-label setting where a single contract may harbor several concurrent vulnerabilities, which deployment strategy is more effective — prompt engineering or fine-tuning? This paper provides a rigorous empirical benchmark to answer that question. We evaluate five prompt engineering strategies (Zero-shot, Chain-of-Thought, Structured Chain-of-Thought, Step-back, and Few-shot) across five state-of-the-art open-source LLMs (7B – 24B parameters), and compare them against the same models fine-tuned with QLoRA on a standardized multi-label dataset of real-world Ethereum contracts. Our benchmark yields three specific, quantified findings. First, the performance gap is large and consistent: the best prompt engineering configuration achieves a macro F1-score of 0.311, while fine-tuned models reach 0.79, a gap that holds across all five model architectures. Second, scale alone did not compensate for the absence of task-specific training in this setting: a 24B-parameter model under the best prompting strategy was outperformed by a 7B fine-tuned model, suggesting that the benefit of scale was not sufficient to close the paradigm gap on this structured multi-label task. Third, advanced reasoning strategies (Chain-of-Thought, Structured CoT, Step-back) provided negligible improvement over the zero-shot baseline for this task, suggesting that the evaluated prompt-only approaches struggled with structured multi-label output requirements rather than with reasoning quality per se. These results hold for the five open-source model architectures, five prompting strategies, and Slither-derived benchmark labels used in this study. These findings provide actionable guidance for researchers and practitioners designing LLM-based security tools for Solidity smart contracts.
A unified framework combining a novel Hierarchical Cross-Attention Subgraph Neural Network for detection with Large Language Models for explanation form a comprehensive framework that significantly enhances both the technical accuracy and operational usability of smart contract analysis.
The rapid advancement of cybersecurity agents in vulnerability discovery has increased the need for benchmarks that evaluate their capabilities accurately. However, synthetic benchmarks lack realism, real-life benchmarks are prone to contamination, and many existing benchmarks have already been saturated by current mod...
Rui Yang, Yi-Chao Xu, Zhuo-Hao Zhang et al.· 0 citations
Automated alpha factor discovery is critical for modern quantitative research. While large language models (LLMs) have increasingly been integrated into quantitative workflows, existing agent-based approaches remain highly susceptible to severe backtest overfitting. To address this challenge, we propose EvoQuantLab, a...
Hao-Lin Yang, Zhi-Qiang Qian, Chu-Tian Chen et al.· 0 citations
This work recasts vulnerability discovery as an input-prediction task with a closed, deterministic ground truth, and decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains, finding constraint inference, not navigation, is the dominant bottleneck.
Yuan-Xiang Shi, Jia-Yi Lin, Xuan-Yong Lin et al.· 0 citations
AiXamine is introduced, a unified black-box platform that evaluates LLM trustworthiness across safety, security, and privacy as interdependent properties and uncovers three cross-dimensional phenomena invisible to single-axis evaluation.
Fatih Deniz, Yazan Boshmaf, Dorde Popovic et al.· 0 citations
Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without c...
Arastoo Zibaeirad, Rodrigo Pato Nogueira, Marco Vieira· IEEE Transactions on Reliabi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.