Reforge is presented, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification.
Abstract
Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.
WebAssembly (Wasm) has emerged as a widely adopted compilation target for systems programming languages, offering near-native performance and portability across platforms. However, the increasing use of code obfuscation techniques in Wasm binaries poses significant challenges for program analysis, malware inspection, and reverse engineering. In this paper, we investigate the capability of large language models (LLMs) to perform automated Wasm deobfuscation. We first provide a conceptual formulation of Wasm deobfuscation as a semantic recovery problem under unknown and potentially non-invertible program transformations, which naturally leads to an underdetermined reconstruction task. Building on this formulation, we introduce a three-tier evaluation hierarchy for assessing deobfuscation quality, consisting of syntax correctness, execution validity, and semantic similarity. We also employ fuzzing based validation as an intermediate evaluation procedure between execution validity and semantic similarity. By generating diverse input cases and comparing the resulting behaviors of the original and reconstructed programs, fuzzing enables us to assess whether semantic preservation holds beyond a limited set of manually specified test cases. We further interpret prompt design as an experimental configuration and study how it induces different inductive biases over the latent semantic hypothesis space explored by LLMs during reconstruction. To empirically study these factors, we construct a structured evaluation framework spanning multiple programming languages, obfuscation stages, and transformation regimes. Experiments with four state-of-the-art LLMs under multiple prompting strategies demonstrate that syntax-aware prompting significantly improves reconstruction accuracy; in particular, for the best performing LLM on C, applying the mono-shot prompting strategy under single obfuscation yields improvements of 33.06%, 7.43%, and 0.13 in syntax, execution, and semantic(LLM) performance, respectively, while virtualization-based obfuscation exposes an identifiability gap that limits semantic recovery without interpreter-level knowledge. These results provide empirical evidence and a conceptual framework for analyzing LLM-assisted deobfuscation as an inverse semantic reconstruction problem under adversarial program transformations.
A reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs, enabling reproducible evaluation of generative AI for hardware design workflows.
Decompilation aims to recover high-level, compilable, and semantically equivalent code from binaries. Traditional decompilers produce pseudo-C that is difficult to read and does not compile, while the recent LLM-assisted approaches generate readable, but semantically incorrect code. LLM-aided iterative recovery is an emerging branch of research, but prior works rely on supplied test suites for semantic recovery. In this work, we present CHISEL, a test suite-free framework to iteratively recover source code from Ghidra-derived pseudo-C. CHISEL uses simple yet effective feedback from a compiler (static analysis) and a coverage-guided fuzzer (differential analysis), augmented by rich observables for grounded divergence detection and feedback, cross-iteration divergence memory, and best candidate retention. We systematically evaluate CHISEL for compilation and semantic recovery, feedback oracle soundness, and iteration overhead on 120 ExeBench functions compiled for the x86-64 architecture, across four optimizations (O0-O3), in both stripped and unstripped variants, using the open-weight Gemma4:31b LLM. CHISEL, with all recommended features, achieves an average of 96.1% re-compilability and 79.8% re-executability rates at an average of 2.1 iterations. Significantly, CHISEL recovers 26% of first-generation execution errors. At the same time, CHISEL feedback oracle falsely accepts only 9.4% candidates. Lastly, CHISEL performs significantly better than two recent prior work on LLM-assisted decompilation.
Varun Kohli, N. Raghava, B. Sikdar et al.· 0 citations
Evaluating SEDCoT on a public COBOL-to-C dataset demonstrates that it outperforms state-of-the-art baselines by at least 12% while producing translations with substantially higher readability than rule-based alternatives.
Phillip Entin, Wenchao Gu, Alexander Knapp et al.· 0 citations
Function calling enables Large Language Models (LLMs) to orchestrate backend services by invoking structured tools, extending their role from text generators to intelligent agents. While prior work evaluates tool-use accuracy, there is limited empirical evidence on the backend-level performance trade-offs between LLM-driven orchestration and traditional deterministic implementations in enterprise systems. This study presents a controlled, reproducible benchmark comparing Javabased deterministic backend logic with LLM function-calling pipelines using locally deployed open-weight models (Mistral 7B, Llama 3.1 8B, and Qwen 2.5 7B). Across three representative enterprise scenarios, data retrieval, data normalization, and command execution, we measure accuracy, latency, error rate, token efficiency, and development effort. Results show that deterministic implementations achieve near-instantaneous execution (0.2-1.8 ms) with 100% reliability, while LLM-driven approaches incur a $\mathbf{1 0 0} \times \mathbf{- 1 6 0 0} \times$ latency overhead (200-800 ms per request). Because of this significant delay overhead, LLM function calling is inappropriate for high-throughput and latencycritical backend services. However, selected 8B-class models (Llama 3.1 and Qwen 2.5) maintain 100% schema adherence under zero-temperature settings and reduce codebase size by approximately 33%, improving development agility. These findings demonstrate that LLM function calling is not a replacement for deterministic backend logic in latency-critical systems, but rather an effective orchestration layer for dynamic and natural language-driven workflows. We conclude by proposing a hybrid backend architecture that combines compiled logic for highperformance transactional paths with LLM-based reasoning for flexible interaction layers.
M. A. Ala’anzy, Adil Akhmetov, Amani Ibraheem et al.· 2026 6th International Confe...· 0 citations