ReadU is an inconsistency-driven technique for just-in-time detection and repair of README bugs that applies a high-recall commit filter, runs internal and external consistency checkers in parallel, uses an alert judge to remove false positives, and automatically synthesizes documentation patches.
Abstract
Repository-level documentation, such as READMEs, is often the first point of contact between users and a repository. When this documentation is incorrect, users may encounter runtime errors or waste their time debugging. We call such mistakes in repository-level documentation README bugs. Addressing README bugs is challenging because documentation mixes prose with code, its connection to the source of truth is loose, and finding a bug still leaves developers to craft a repair. This paper presents READU, an inconsistency-driven technique for just-in-time detection and repair of README bugs. The key insight behind READU is that README bugs often manifest as inconsistencies between documentation and another source of truth: either repository-internal facts, such as source code, or repository-external facts, such as external dependencies. READU applies a high-recall commit filter, runs internal and external consistency checkers in parallel, uses an alert judge to remove false positives, and automatically synthesizes documentation patches. On 6,000 recent commits from six popular repositories including Linux and Spring Boot, READU detects 244 true positives with 75% precision, while consuming less than $0.01 and less than one minute per commit, on average. Of these true positives, READU correctly repairs 217. We report 66 found README bugs, of which (so far) 44 are confirmed and 26 are fixed.
Reproducing reliable data science code from informal sources is challenging due to ambiguous problem specifications, missing dependencies, and performance bottlenecks. Although developer Q&A forums provide rich discussions on diagnosing and fixing real-world issues, the information is often incomplete and unstructured, limiting its use for automated debugging and verification. In this paper, we introduce Reprodgen, a large language model (LLM) based framework for automatically replicating executable buggy and patched data science programs from Q&A forum posts. Given a question and its corresponding answer, Reprodgen reconstructs the buggy behavior described in the question and the intended fix described in the answer, producing executable buggy and patched code pairs that reflect the original discussion. The framework builds structured representations of code intent (CI), functional requirements (FR), and Structured Chain of Thought (SCoT), and iteratively refines code using an LLM-based reviewer until it is executable and semantically consistent. We evaluate Reprodgen on Stack Overflow (SO) and GitHub Issues (GI) across seven data science libraries, including pandas, numpy, and scikit-learn, and construct a benchmark of runnable buggy and patched programs validated by human experts. Our pipeline uses LLMs for semantic assessment, while executability is verified through actual execution. Results show reliable replication with clear differences in model performance.
Ragib Shahariar Ayon, Mohammad Wardat, Shibbir Ahmed· 0 citations
Verification-aware languages, like Dafny, integrate formal specifications directly into source code to enable static correctness checks. However, when verification fails, the feedback provided is often limited to the specific condition of the error, such as a violated postcondition, rather than the root cause of the fault. While Dafny's counterexample features provide concrete execution traces, these typically expose a single failing path per assertion failure, leaving the developer to manually look through the entire trace to locate the error. This paper investigates automated fault localization for verification-aware languages by comparing two paradigms: state-based and counterexample-based localization. Our state-based localization strategy replicates the ``snapshot''methodology of AutoFix by inferring invariants and predicates to identify suspicious program states. The counterexample-based strategy consists of a family of techniques that progressively enrich the use of verifier output: from raw counterexample extraction, to structured single-trace ranking, and to multi-trace aggregation. To validate these methods, we present an evaluation framework using MutDafny to generate a diverse mutant dataset from DafnyBench and measure localization effectiveness using the EXAM score. Our results show that counterexample-based approaches substantially outperform state-based localization in this setting. Structured ranking over a single trace yields the largest improvement over raw counterexample output, while multi-trace aggregation provides additional gains in robustness and debugging utility by increasing coverage and reducing path bias introduced by the solver. These findings demonstrate that effective fault localization in verification-aware languages depends both on using counterexample information, and how that information is structured and diversified.
Álvaro F. Silva, Isabel Amaral, João Pascoal Faria et al.· 0 citations
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
The first empirical study of attention patterns in LLM-based program repair is presented, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair, and indicates that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success.
Ramtin Ehsani, Irene Manotas, Saurabh Pujar et al.· 0 citations
SemaDiff, a novel approach for identifying semantic-preserving commits through behaviour-based analysis, is proposed, and results show that SemaDiff distinguishes accurately semantic-preserving from -- changing commits in about 76% of the cases, with a 100% precision in semantic-changing commit detection.
Maha Ayub, Michael Konstantinou, Ahmed Khanfir et al.· 0 citations
Code refinement, a process of revising flawed code changes based on the code review comments, is a fundamental practice in software development for ensuring software quality and maintainability. Recent approaches using large language models (LLMs) show promise in automating this process, yet they typically rely only on review comments and a single code snippet, neglecting broader repository context. When a code refinement task requires invoking reviewer-unspecified identifiers (e.g., variables, functions, or classes) that are defined elsewhere in the repository, the LLM must accurately predict the exact identifier signature and usage, otherwise it could easily result in fatal errors. Prior work demonstrates that 90.59% of such knowledge-conflicting hallucinations cause all test cases to fail. Notably, over 60% of real-world code refinement scenarios in most popular Github repositories depend on repository-level knowledge, yet none of the existing code refinement works leverage this information. Moreover, current datasets do not distinguish between context-dependent and context-independent instances, contain many low-quality instances, and rely on lexical metrics (e.g., BLEU) that poorly correlate with human judgment. These limitations hinder reliable evaluation of code refinement tools in real-world, context-dependent scenarios. To address this gap, we introduce ReCoRe-Bench, a high-quality benchmark of 1,000 code refinement instances from 328 Python, Java, and JavaScript repositories that focused on one of the most challenging code refinement scenarios that strictly requires repository-level knowledge reasoning. To ensure repository-level context is required and to achieve strong review-revision alignment, we applied a strict identifier-based heuristic filtering pipeline and manually filtered low quality instances containing vague comments or unrelated review comments. To better evaluate whether an LLM correctly retrieves and resolves identifiers defined in the repository, instead of inferring them from limited local context, we propose a new evaluation metric, ReDIs Match. Additionally, to address the weaknesses of current LLM-powered code refinement methods identified by ReCoRe-Bench, we propose a straightforward method, RepoRefiner, which retrieves repository-level context by collecting the full file content, extracting definitions of its identifiers, and summarizing these contexts to support code refinement. The experiment results show that the RepoRefiner outperforms current code refinement baselines, and the repository-level context is essential for the reliable code refinement. Our findings underscore both the promise and current limitations of LLMs in repository-level code refinement scenarios, highlighting the need for more effective static analysis techniques and prompt engineering for broader repository-level context exploration.
Ke Wang, Peng Lan, Jiakun Liu et al.· ACM Transactions on Software...· 1 citation
