Skip to content
Open access

Benchmarking Prompt Engineering Against Fine-Tuning for Multi-Label Vulnerability Detection in Solidity Smart Contracts: An Empirical Study

2026 · IEEE Access · Vol 14, pp. 138876-138892 · 0 citations · 37 references
Computer Science

TL;DR

This paper evaluates five prompt engineering strategies across five state-of-the-art open-source LLMs, and compares them against the same models fine-tuned with QLoRA on a standardized multi-label dataset of real-world Ethereum contracts to provide actionable guidance for researchers and practitioners designing LLM-based security tools for Solidity smart contracts.

Abstract

Large Language Models (LLMs) are increasingly being deployed for smart contract security, yet a fundamental question remains unresolved for practitioners: when confronted with the realistic, multi-label setting where a single contract may harbor several concurrent vulnerabilities, which deployment strategy is more effective — prompt engineering or fine-tuning? This paper provides a rigorous empirical benchmark to answer that question. We evaluate five prompt engineering strategies (Zero-shot, Chain-of-Thought, Structured Chain-of-Thought, Step-back, and Few-shot) across five state-of-the-art open-source LLMs (7B – 24B parameters), and compare them against the same models fine-tuned with QLoRA on a standardized multi-label dataset of real-world Ethereum contracts. Our benchmark yields three specific, quantified findings. First, the performance gap is large and consistent: the best prompt engineering configuration achieves a macro F1-score of 0.311, while fine-tuned models reach 0.79, a gap that holds across all five model architectures. Second, scale alone did not compensate for the absence of task-specific training in this setting: a 24B-parameter model under the best prompting strategy was outperformed by a 7B fine-tuned model, suggesting that the benefit of scale was not sufficient to close the paradigm gap on this structured multi-label task. Third, advanced reasoning strategies (Chain-of-Thought, Structured CoT, Step-back) provided negligible improvement over the zero-shot baseline for this task, suggesting that the evaluated prompt-only approaches struggled with structured multi-label output requirements rather than with reasoning quality per se. These results hold for the five open-source model architectures, five prompting strategies, and Slither-derived benchmark labels used in this study. These findings provide actionable guidance for researchers and practitioners designing LLM-based security tools for Solidity smart contracts.

Read PDF

Similar papers

Sep 2026

A Unified Framework for Function-Level Vulnerability Detection and Explanation in Smart Contracts

A unified framework combining a novel Hierarchical Cross-Attention Subgraph Neural Network for detection with Large Language Models for explanation form a comprehensive framework that significantly enhances both the technical accuracy and operational usability of smart contract analysis.

Ngoc Minh Nguyen, Le-Minh Nguyen · 0 citations
Preprint Oct 2026

Discovery or Memorization? A Faithful Benchmark for Real-World Business-Logic Vulnerability Discovery

The rapid advancement of cybersecurity agents in vulnerability discovery has increased the need for benchmarks that evaluate their capabilities accurately. However, synthetic benchmarks lack realism, real-life benchmarks are prone to contamination, and many existing benchmarks have already been saturated by current mod...

Rui Yang, Yi-Chao Xu, Zhuo-Hao Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

EvoQuantLab: Training-Free Dual-Loop Evolution for Auditable Quantitative Research

Automated alpha factor discovery is critical for modern quantitative research. While large language models (LLMs) have increasingly been integrated into quantitative workflows, existing agent-based approaches remain highly susceptible to severe backtest overfitting. To address this challenge, we propose EvoQuantLab, a...

Hao-Lin Yang, Zhi-Qiang Qian, Chu-Tian Chen et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction

This work recasts vulnerability discovery as an input-prediction task with a closed, deterministic ground truth, and decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains, finding constraint inference, not navigation, is the dominant bottleneck.

Yuan-Xiang Shi, Jia-Yi Lin, Xuan-Yong Lin et al. · 0 citations
2026

LLMKernelBench: Benchmarking Large Language Models on Software Vulnerability Detection in Linux Kernel

Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without c...

Arastoo Zibaeirad, Rodrigo Pato Nogueira, Marco Vieira · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.