This paper presents a unified catalogue of test refactoring patterns derived from a mixed-method approach involving repository mining, developer surveys, and analysis of StackOverflow discussions, to establish a rigorous benchmark for evaluating future automated recommendation systems, including those driven by Large Language Models.
Abstract
Automated software testing is critical for modern software development, yet test code itself requires maintenance and refactoring to prevent decay. While theoretical test-specific code smells are well-documented, the practical automation of test refactoring—both detection and recommendation—remains incipient, leading to fragmented knowledge and inadequate tooling. This paper bridges this gap by presenting a unified catalogue of test refactoring patterns derived from a mixed-method approach involving repository mining, developer surveys, and analysis of StackOverflow discussions. To operationalize this catalogue, we extend RefactoringMiner with test-aware detection capabilities and introduce PatternTracker, a tool for identifying and tracking the evolution of test patterns across version history. Leveraging this infrastructure, we analyzed 1.7 million Java projects to generate a high-confidence ground-truth dataset of test refactorings. Our results validate the prevalence of these patterns in open-source projects and establish a rigorous benchmark for evaluating future automated recommendation systems, including those driven by Large Language Models.
Results show that ML-enhanced recommendations outperform traditional methods in accuracy, relevance, and impact on maintainability metrics, and highlight the potential of integrating ML into modern development practices to support developers in producing cleaner, more maintainable software systems.
Rohit Malhotra· International Journal of Mod...· 0 citations
SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis, is proposed, and results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.
Maha Ayub, Michael Konstantinou, Ahmed Khanfir et al.· 0 citations
Code refinement, a process of revising flawed code changes based on the code review comments, is a fundamental practice in software development for ensuring software quality and maintainability. Recent approaches using large language models (LLMs) show promise in automating this process, yet they typically rely only on review comments and a single code snippet, neglecting broader repository context. When a code refinement task requires invoking reviewer-unspecified identifiers (e.g., variables, functions, or classes) that are defined elsewhere in the repository, the LLM must accurately predict the exact identifier signature and usage, otherwise it could easily result in fatal errors. Prior work demonstrates that 90.59% of such knowledge-conflicting hallucinations cause all test cases to fail. Notably, over 60% of real-world code refinement scenarios in most popular Github repositories depend on repository-level knowledge, yet none of the existing code refinement works leverage this information. Moreover, current datasets do not distinguish between context-dependent and context-independent instances, contain many low-quality instances, and rely on lexical metrics (e.g., BLEU) that poorly correlate with human judgment. These limitations hinder reliable evaluation of code refinement tools in real-world, context-dependent scenarios. To address this gap, we introduce ReCoRe-Bench, a high-quality benchmark of 1,000 code refinement instances from 328 Python, Java, and JavaScript repositories that focused on one of the most challenging code refinement scenarios that strictly requires repository-level knowledge reasoning. To ensure repository-level context is required and to achieve strong review-revision alignment, we applied a strict identifier-based heuristic filtering pipeline and manually filtered low quality instances containing vague comments or unrelated review comments. To better evaluate whether an LLM correctly retrieves and resolves identifiers defined in the repository, instead of inferring them from limited local context, we propose a new evaluation metric, ReDIs Match. Additionally, to address the weaknesses of current LLM-powered code refinement methods identified by ReCoRe-Bench, we propose a straightforward method, RepoRefiner, which retrieves repository-level context by collecting the full file content, extracting definitions of its identifiers, and summarizing these contexts to support code refinement. The experiment results show that the RepoRefiner outperforms current code refinement baselines, and the repository-level context is essential for the reliable code refinement. Our findings underscore both the promise and current limitations of LLMs in repository-level code refinement scenarios, highlighting the need for more effective static analysis techniques and prompt engineering for broader repository-level context exploration.
Ke Wang, Peng Lan, Jiakun Liu et al.· ACM Transactions on Software...· 1 citation
Code smells significantly threaten software maintainability by accumulating technical debt, yet developers often lack the resources to manually address these flaws under tight release schedules. While static analysis tools like SonarQube provide precise detection, they function largely as passive alert systems, leaving the burden of refactoring on developers. To bridge this gap, we present a novel cleaning tool, namely SmellCC, a Visual Studio Code extension that augments SonarQube with an LLM-based pipeline to automatically detect and refactor Python code smells. By employing Chain-of-Thought (CoT) and few-shot learning, SmellCC provides in-place, one-click remediation for the top-10 most frequent smells, effectively preventing the accumulation of technical debt during development. Our quantitative evaluation demonstrates that our SmellCC is promising in helping developers effectively eliminate code smells (96.8\% cleaning rate) with high accuracy (i.e., 91.3\%), ensuring that the refactored code remains syntactically correct and behavior-preserving, thereby significantly improving long-term software maintainability.
Xiaoting Zhang, Yujie Zhang, Zhipeng Gao et al.· 0 citations
Unlike traditional refactorings, research on test-specific refactoring is limited. This paper presents a comprehensive catalogue of 57 test-specific refactoring types, out of which only 33 were previously documented in the literature. We have identified and classified those refactorings using a mixed-method approach grounded on three sources of data: a) Existing datasets of test-modifying commits, b) Monitoring of latest contributions to popular GitHub repositories, and c) Stack Overflow discussions. Our findings reveal that tools have limited support for test-specific refactorings. And, unlike the existing literature that mainly associates refactoring activity in tests with test smell resolution, test refactoring is driven by various motivations, such as improving test understandability, accommodating changes in production code, and test framework migration. Finally, we provide actionable implications for researchers and tool builders.
Victor Guerra Veloso, Nikolaos Tsantalis, T. Chen· Empirical Software Engineeri...· 0 citations
This work proposes an approach based on a foundation model oracle that analyzes git-style diffs to identify behavioral changes introduced by Python refactorings and uncovered 13 distinct bugs among the seven refactoring types studied.
Jonhnanthan Oliveira, Rohit Gheyi, Márcio Ribeiro et al.· 0 citations