This paper proposes an automated approach to extract domain models from source code using lightweight, locally deployable LLMs and achieves high F1-scores on a dataset of ten projects, each comprising a curated domain model and its corresponding implementation, while remaining fully executable on locally deployable LLMs.
Abstract
Large language models (LLMs) have recently shown strong capabilities for code understanding, making them promising for reverse engineering domain models from source code. However, state-ofthe- art proprietary LLMs cannot be used in many industrial contexts due to privacy and confidentiality constraints, while compact open-source LLMs that can run locally are limited by their context window and cannot process large code bases directly. In this paper, we propose an automated approach to extract domain models from source code using lightweight, locally deployable LLMs. Our method combines structural and semantic heuristics with iterative LLM-based reasoning to overcome context limitations. By progressively analyzing ranked subsets of code elements, the approach identifies domain concepts and refines domain boundaries without requiring full-system context. Our approach achieves high F1-scores on a dataset of ten projects, each comprising a curated domain model and its corresponding implementation, while remaining fully executable on locally deployable LLMs. This makes it particularly suitable for reverse engineering tasks in privacy-sensitive industrial environments.
In fast-evolving software systems, effective 'natural language requirements parsing' and downstream change effect analysis capability across a multitude of codes represents low-hanging-fruit in this regard. We present a structured framework to deploy Large Language Models (LLMs) for automating two essential software engineering tasks, namely requirement interpretation and change impact analysis Utilizing the inherent understanding of semantics offered by transformer-based LLMs, the novel approach advances by converting vague and unstructured requirement documents into structured but machine-readable specifications to offer a direct traceability mapping from requirements to system components. Additionally, the framework leverages LLM-driven dependency analysis to predict and quantify how change effects percolate through connected modules which can minimize manual effort and human errors. This approach combines prompt engineering and retrieval-augmented generation (RAG) for domain-relevant accuracy plus fine-tuning techniques. On open-source and enterprise-grade software projects, experimental evaluations show that disambiguation accuracy, traceability precision, and change impact coverage of our approach are orders of magnitude better than state-of-the-art rule-based or static analysis tools. Notes: The results illustrate the application of LLMs at scale and demonstrate how these can alter software engineering workflows by removing bottlenecks (at a massive scale) at different stages of the software development lifecycle. In this research, we provide a generalizable pipeline that helps to bridge the gap from NLP advancements into practice for software lifecycle management.
Nithya Krishnan, Kumaran Ramanujam, Suresh Babu Narra et al.· 2026 International Conferenc...· 0 citations
The results show that appropriate model slicing significantly improves completion correctness while simultaneously reducing token usage across several structural and semantic evaluation metrics, and establish model slicing as a key factor in LLM-based model completion and provide guidance for effective context selection for other modeling tasks.
Large Language Models (LLMs) have advanced code completion, but their ability to generate API usages compatible with evolving third-party libraries (TPLs) remains uncertain. As TPL APIs frequently change, LLMs risk producing code incompatible with installed library versions, causing build failures or incorrect behaviors. We define such issues as Incompatible Third-party Library API Usage (ITAU) and conduct a systematic study to evaluate how state-of-the-art LLMs handle this challenge. To this end, we propose an automated framework that builds a versioned TPL API Knowledge Base and a large-scale benchmark with 10,867 realistic code completion tasks. Through comprehensive evaluation of six state-of-the-art LLMs, we find that even top-performing models frequently generate incompatible completions. We further propose two lightweight solutions, Real-time Detection and Lightweight Repair, to mitigate ITAUs. This framework and benchmark provide a foundation for more compatibility-aware code generation in evolving software ecosystems.
Li Lin, Yaorui Fei, Yunfeng Shen et al.· ACM Transactions on Software...· 0 citations
Code refinement, a process of revising flawed code changes based on the code review comments, is a fundamental practice in software development for ensuring software quality and maintainability. Recent approaches using large language models (LLMs) show promise in automating this process, yet they typically rely only on review comments and a single code snippet, neglecting broader repository context. When a code refinement task requires invoking reviewer-unspecified identifiers (e.g., variables, functions, or classes) that are defined elsewhere in the repository, the LLM must accurately predict the exact identifier signature and usage, otherwise it could easily result in fatal errors. Prior work demonstrates that 90.59% of such knowledge-conflicting hallucinations cause all test cases to fail. Notably, over 60% of real-world code refinement scenarios in most popular Github repositories depend on repository-level knowledge, yet none of the existing code refinement works leverage this information. Moreover, current datasets do not distinguish between context-dependent and context-independent instances, contain many low-quality instances, and rely on lexical metrics (e.g., BLEU) that poorly correlate with human judgment. These limitations hinder reliable evaluation of code refinement tools in real-world, context-dependent scenarios. To address this gap, we introduce ReCoRe-Bench, a high-quality benchmark of 1,000 code refinement instances from 328 Python, Java, and JavaScript repositories that focused on one of the most challenging code refinement scenarios that strictly requires repository-level knowledge reasoning. To ensure repository-level context is required and to achieve strong review-revision alignment, we applied a strict identifier-based heuristic filtering pipeline and manually filtered low quality instances containing vague comments or unrelated review comments. To better evaluate whether an LLM correctly retrieves and resolves identifiers defined in the repository, instead of inferring them from limited local context, we propose a new evaluation metric, ReDIs Match. Additionally, to address the weaknesses of current LLM-powered code refinement methods identified by ReCoRe-Bench, we propose a straightforward method, RepoRefiner, which retrieves repository-level context by collecting the full file content, extracting definitions of its identifiers, and summarizing these contexts to support code refinement. The experiment results show that the RepoRefiner outperforms current code refinement baselines, and the repository-level context is essential for the reliable code refinement. Our findings underscore both the promise and current limitations of LLMs in repository-level code refinement scenarios, highlighting the need for more effective static analysis techniques and prompt engineering for broader repository-level context exploration.
Ke Wang, Peng Lan, Jiakun Liu et al.· ACM Transactions on Software...· 1 citation
LLM-based repository-level code generation aims to generate code using the context available in a software repository, requiring LLMs to reason over complex code dependencies. Due to limited context windows and insufficient repository-specific understanding, LLMs typically rely on retrieval-augmented generation (RAG) to incorporate relevant code. Early RAG approaches primarily employ similarity-based retrieval, which often fails to retrieve code snippets that the target function depends on. Recent work introduces graph-based retrieval to model such dependencies, but typically relies on manually designed rules and static global graphs, leading to limited flexibility and high construction and maintenance costs. In contrast, human developers collect helpful context by implicitly constructing a partial dependency graph and iteratively inspecting along it. Inspired by this behavior, we propose DyRetriever, an efficient context retrieval method via partial dependency graphs. DyRetriever uses an LLM to first select a set of entry-point functions and then perform multi-hop reasoning along the code dependency graph. During multi-hop reasoning, it uses the LLM's semantic understanding to validate whether a function can help generate the target function, eliminating manually designed rules and enabling flexibility across scenarios. Instead of statically constructing a global dependency graph, DyRetriever builds a partial graph on demand and discards it after use, reducing construction and maintenance costs. We integrate DyRetriever with a similarity-based code retriever to build DyCoder and evaluate it on CoderEval and DevEval. Experimental results show that DyCoder achieves relative Pass@1 improvements of 25.63% and 59.73% on CoderEval and DevEval, respectively, compared with existing RAG-based methods, while being 7.4x faster than baselines based on static dependency graph construction.
Zhongxin Liu, Zhonghao Jiang, Zhifan Ye et al.· 1 citation
While Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, their effectiveness in unit testing is often constrained by insufficient context regarding external dependencies. This limitation is particularly pronounced in industrial settings, where proprietary code remains opaque to the model. To address this challenge, we present a systematic empirical study of multiple strategies for context enrichment and optimization in LLM‐based unit test generation, conducted on seven diverse projects (three open‐source and four proprietary industrial systems), encompassing 261 distinct methods. By evaluating seven implementations (ranging from basic prompts to optimized context reduction strategies) across 10 independent runs, we analysed a total of 28,710 test suites. Our results demonstrate that combining prompt engineering with external dependency retrieval achieves an average branch coverage increase of 11.52 percentage points on industrial software over the baseline, with statistically significant improvements across all competing implementations. Beyond coverage, richer context substantially reduces generation‐repair iterations, cutting median execution time by 51.3% in industrial projects. We further show that reducing external dependencies to method signatures alone decreases input token consumption by up to 46.6% (25.4% in industrial projects) while fully preserving the coverage and efficiency gains of the complete retrieval approach. To confirm that these benefits are not tied to a specific model, we replicate the core comparison across three LLM backends from different families, obtaining a consistent, statistically significant coverage improvement on industrial code in every case. These findings establish this optimized context strategy as a cost‐effective solution for scalable, industrial‐grade automated test generation.
Javier Ferrer, Francisco Chicano· Expert systems· 0 citations