Skip to content
Preprint

SemaDiff: Identifying Semantic-Changing Commits with Generated Code and Tests

Jul 2026 · 0 citations · 42 references
Computer Science

TL;DR

SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis, is proposed, and results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.

Abstract

Distinguishing semantic-preserving commits from changing ones remains an open challenge in software repository mining. While existing approaches detect refactoring commits accurately, they cannot ensure that a commit is purely semantic-preserving, without any interleaving behaviour-changing modification. This limitation can impact several tasks, such as debugging, fault localisation, bug dataset construction, rollback analysis, and bug fixes backporting. To fill this gap, we propose SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis; comparison of similar test execution on pre- and post-commit versions. As code impacted by the refactoring is often hard to test and different accross both versions, we propose generating additional calling methods to that code, which serve as testing target. Given a commit, SemaDiff analyses the diff to identify modified code and extracts unchanged dependent code that calls it. It then generates an additional dependent class using a large language model to exercise the changed code in both versions, and automatically generates tests for the dependent code. This way, we obtain the same tests for the different code versions, enabling the behavioural-difference detection. The commit is classified as semantic-preserving only if all generated tests produce identical outcomes across the two versions. To evaluate SemaDiff, we construct and annotate manually a dataset of 183 commits, gathered from well-known open-source Java projects. The obtained results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.

View source

Similar papers

Review Jul 2026

From Discussion to Execution: Replicating Buggy and Correct Data Science Code

Reproducing reliable data science code from informal sources is challenging due to ambiguous problem specifications, missing dependencies, and performance bottlenecks. Although developer Q&A forums provide rich discussions on diagnosing and fixing real-world issues, the information is often incomplete and unstructured, limiting its use for automated debugging and verification. In this paper, we introduce Reprodgen, a large language model (LLM) based framework for automatically replicating executable buggy and patched data science programs from Q&A forum posts. Given a question and its corresponding answer, Reprodgen reconstructs the buggy behavior described in the question and the intended fix described in the answer, producing executable buggy and patched code pairs that reflect the original discussion. The framework builds structured representations of code intent (CI), functional requirements (FR), and Structured Chain of Thought (SCoT), and iteratively refines code using an LLM-based reviewer until it is executable and semantically consistent. We evaluate Reprodgen on Stack Overflow (SO) and GitHub Issues (GI) across seven data science libraries, including pandas, numpy, and scikit-learn, and construct a benchmark of runnable buggy and patched programs validated by human experts. Our pipeline uses LLMs for semantic assessment, while executability is verified through actual execution. Results show reliable replication with clear differences in model performance.

Ragib Shahariar Ayon, Mohammad Wardat, Shibbir Ahmed · 0 citations
Book Open access Jul 2026

When Testing Meets Refactoring: Catalogue, Detection, and Recommendation

This paper presents a unified catalogue of test refactoring patterns derived from a mixed-method approach involving repository mining, developer surveys, and analysis of StackOverflow discussions, to establish a rigorous benchmark for evaluating future automated recommendation systems, including those driven by Large Language Models.

Victor Guerra Veloso · 0 citations
Conference Jul 2026

Commit-Message-Augmented Static Analysis of Cross-Site Scripting Fixes in Java Web Applications

Cross-Site Scripting (XSS) remains a common and high-impact web vulnerability. Detecting XSS with high precision is difficult because exploitability depends on end-to-end data flow (sources, sanitizers, and sinks), framework-specific rendering semantics, and the correctness of context-aware output encoding. Although secure development lifecycles increasingly integrate static application security testing (SAST) into CI/CD pipelines, practical adoption is often limited by false positives and incomplete modeling of project-specific sanitization and templating behavior. In parallel, machine learning (ML) approaches show promise for learning vulnerability-relevant patterns from code, but they depend on the availability of realistic labeled data. This paper studies a low-cost repository-native signal that is routinely available but underused in ML-assisted static analysis: commit messages. We propose a mining and weak-labeling pipeline for Java repositories that identifies candidate XSS-fix commits using message heuristics grounded in OWASP guidance and CWE taxonomy, aligns commit messages with pre-/post-change code contexts extracted from diffs, and trains classifiers under two feature settings: (i) code-only and (ii) code+commit-message fusion. Across three model families (Random Forest, TextCNN, and BiLSTM), commit-message fusion consistently improves F1 and AUPRC and reduces false positives, suggesting that developer intent provides a useful auxiliary signal for security triage.

Abu Alam, Qka Mirza · 0 citations
Review Aug 2026

COMMITGUARD: Differential Slice Fuzzing for Commit-Induced Bug Detection

Modern software systems evolve through frequent commits that implement bug fixes, features, and security patches. Although code review and testing are widely used to check these changes, they often provide limited assurance for memory-safety issues. Code reviewers may miss subtle boundary, lifetime, or initialization errors, while existing tests may not exercise the specific paths affected by a commit. Fuzzing is effective at exposing such bugs, but applying it to every commit remains impractical because whole-program fuzzing is expensive, requires suitable harnesses, and may still fail to reach the code changed by a commit. In this paper, we introduce COMMITGUARD, a commit-aware differential slice-based fuzzing approach for verifying code changes. The key insight behind COMMITGUARD is that the pre-commit version of a modified function can serve as a behavioral baseline for interpreting bugs found after the commit. For each target commit, COMMITGUARD identifies modified functions, extracts compilable code slices from both the pre-commit and post-commit versions, and fuzzes the paired slices independently. It then compares sanitizer reports across the two versions and reports bugs that emerge only in the post-commit version as candidate commit-induced bugs. We evaluate COMMITGUARD on 300 commits from openSSL, libpcap and leptonica. Slice fuzzing initially produces 518 sanitizer reports across these commits. By comparing pre-commit and post-commit slices, COMMITGUARD narrows this large output to 7 candidate commit-induced bug reports that require manual triage. Manual validation confirms 5 of these reports as real bugs that were fixed by developers of the examined projects after we reported them, while only 2 reports were classified as false positives. COMMITGUARD analyzes a commit in 32.4 minutes on average and achieves 75.36% average coverage of modified functions.

Aniruddhan Murali, N. Mathews, Mahmoud Alfadel et al. · 0 citations
Preprint Aug 2026

Identifying Latent Declarative Representations of Code for Assisting Repository Migration

Legacy software repositories embed decades of domain knowledge in undocumented code, making understanding and modernization difficult. We treat a program as the implementation of an unobserved, declarative description of its computation and investigate whether making this latent declarative representation explicit improves repository-scale porting. ADFD-Migrate approximates the latent representation with an annotated data-flow diagram (ADFD) of processes, data stores, external entities, flows, and behavioral contracts. An LLM infers the source ADFD from bounded repository context, guided by static-analysis coverage checks. Dependency-aware chunking orders bounded process groups for target-language generation. Differences between the source ADFD and a statically recovered target ADFD then guide regeneration. We evaluate ADFD-Migrate on f2x50, a new benchmark of 50~Fortran repositories spanning 1.5k--1.6M lines of code and three complexity tiers, and assess the resulting ports along two dimensions: porting soundness, measured by source-oracle behavioral agreement, and porting completeness, measured by a composite migration outcome index. Against 382 curated Fortran-oracle probes, the generated Python passes 327 (85.6\%), with 40 repositories passing every attempted probe. ADFD-Migrate exposes all 382 planned behaviors as runnable targets, compared with 99 and 98 for direct and repository-context translation and 69 and 30 for the static-profile and dependency-chunking ablations. It also achieves a 93.1\% mean migration outcome index and a 17--59 percentage-point outcome-index advantage over direct translation on 47 repositories. These results suggest that an inspectable semantic bottleneck can improve the coverage and integration of repository-scale migration while enabling lower-cost generation for many repositories.

Shraddha Surana, Ashwin Srinivasan, Michael Bain · 0 citations