The quality of research software directly impacts the quality of the research results. Peer code review can identify defects, improve design, and facilitate knowledge sharing. Previously, we conducted a survey to understand how Research Software Engineers (RSEs) view and use code review. Because the prior survey did not examine how RSEs implement code review or the challenges they face, we study here how RSEs implement code review improvement practices, the challenges that limit adoption, and the additional strategies they propose. To gather this insight, we conducted 20 semi-structured interviews of RSEs. The results show that RSEs most often improved code review by using pull requests, defining review expectations, adding lightweight process steps, and using tools and automation (e.g., CI checks) to reduce effort and increase consistency. Key challenges included limited reviewer capacity and time, gaps in Git and software engineering skills, social and authority barriers to setting norms, and weak documentation or enforcement of standards. They also proposed lightweight training, clearer reviewer guidance (e.g., checklists), and better recognition of review and mentoring work. To ease peer code review adoption in research software, teams should begin with lightweight pull-request workflows, document a few clear review rules, use CI to enforce basics, and support reviewers with simple guidance, training, and recognition.
Md. Ariful Islam Malik, Jeffrey C. Carver, Nasir U. Eisty· Practice and Experience in A...· 0 citations
The oracle problem (determining the correct expected outcome for a test) remains a major bottleneck in automated testing, and is increasingly relevant as non-experts rely on AI-generated code they cannot reliably validate. We study whether large language models (LLMs) can generate generalizable test oracles directly from natural-language business requirements, without access to source code or example input-output pairs. We propose a reproducible, requirement-driven pipeline grounded in Defects4J. For each of 10 real bugs from Defects4J Lang (Bugs 1 and 3-11), we (i) extract behavioral changes via buggy/fixed diffs, (ii) manually translate the change into a business requirement, (iii) construct a requirement-derived oracle (REQ) as a gold standard, and (iv) prompt five LLMs (DeepSeek-V3, Gemma-3n, Llama-3, Mistral-7B, and Qwen-3) to generate Java oracle code. We evaluate oracle correctness and generalization under two targets: agreement with REQ and agreement with the system under test (SUT), reporting macro-averaged accuracy, precision, recall, and F1. LLMs achieve non-trivial generalization but with substantial bug- and model-level variance. Generated oracles align more closely with REQ than with SUT, and correlations between requirement technicality/ambiguity ratings and oracle accuracy are weak with wide confidence intervals. No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness. As a pilot proof of concept, these findings are preliminary and are intended to establish feasibility and motivate larger-scale empirical investigation.
