Findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.
Abstract
Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.
Function calling enables Large Language Models (LLMs) to orchestrate backend services by invoking structured tools, extending their role from text generators to intelligent agents. While prior work evaluates tool-use accuracy, there is limited empirical evidence on the backend-level performance trade-offs between LLM-driven orchestration and traditional deterministic implementations in enterprise systems. This study presents a controlled, reproducible benchmark comparing Javabased deterministic backend logic with LLM function-calling pipelines using locally deployed open-weight models (Mistral 7B, Llama 3.1 8B, and Qwen 2.5 7B). Across three representative enterprise scenarios, data retrieval, data normalization, and command execution, we measure accuracy, latency, error rate, token efficiency, and development effort. Results show that deterministic implementations achieve near-instantaneous execution (0.2-1.8 ms) with 100% reliability, while LLM-driven approaches incur a $\mathbf{1 0 0} \times \mathbf{- 1 6 0 0} \times$ latency overhead (200-800 ms per request). Because of this significant delay overhead, LLM function calling is inappropriate for high-throughput and latencycritical backend services. However, selected 8B-class models (Llama 3.1 and Qwen 2.5) maintain 100% schema adherence under zero-temperature settings and reduce codebase size by approximately 33%, improving development agility. These findings demonstrate that LLM function calling is not a replacement for deterministic backend logic in latency-critical systems, but rather an effective orchestration layer for dynamic and natural language-driven workflows. We conclude by proposing a hybrid backend architecture that combines compiled logic for highperformance transactional paths with LLM-based reasoning for flexible interaction layers.
M. A. Ala’anzy, Adil Akhmetov, Amani Ibraheem et al.· 2026 6th International Confe...· 0 citations
This paper introduces T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement that improves trace-to-line localization.
The results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.
Yunhao Liang, Chengguang Gan, Ruixuan Ying et al.· 0 citations
Lightweight large language models (LLMs) are increasingly being deployed locally on personal computers and are expected to play a growing role in resource-constrained edge and mobile environments. In such settings, energy consumption, execution time, and memory usage directly affect practical usability, yet existing evaluations of LLM efficiency largely rely on proxy descriptors such as parameter count or FLOPs, often decoupled from task precision. This paper introduces a PTME-based experimental framework for the precision-aware profiling of lightweight LLM inference, jointly measuring Precision, execution Time, peak Memory usage, and Energy consumption through direct hardware-level measurements. The methodology is applied to a representative set of lightweight LLMs executed locally under edge-class resource envelopes on a controlled desktop platform, using benchmarks spanning code generation, mathematical reasoning, and multi-task understanding. We find that static proxy descriptors approximate inference cost well but fail to predict precision. Tightening the resource envelope increases cost without affecting precision, amplifying execution time more strongly than energy and penalizing larger models the most. Moreover, no single model dominates across all PTME dimensions, and a Pareto analysis reveals non-dominated configurations that would be hidden by accuracy-only or efficiency-only assessments, providing practical guidance for selecting models under different resource envelopes. These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.
Tomohiro Harada, Enrique Alba, Gabriel Luque· 0 citations