While Large Language Model (LLM) agents are increasingly applied to automated software repair, misalignment remains in how humans and agents judge issue conditions and how agents’ pre-execution self-assessments relate to repair competence. Human developers rely on diagnostic cues such as reproduction steps and stack traces to judge whether an issue is sufficiently specified, whereas LLM agents often fail to recognize missing information. We present the first systematic empirical study of misalignment between human and agent judgments and between agent judgments and repair performance. Using SWE-bench, controlled ablation experiments establish a causal link between removing human-valued cues and reduced LLM repair success. Specifically, LLM judges show limited agreement with human problem-specification ratings, and agents’ pre-execution self-assessments only weakly track repair degradation when key cues are removed. Our trajectory analysis further reveals distinct behavioral responses to missing information, while post-execution self-judgment signals add useful discriminative information when combined with behavioral traces. These findings reveal a persistent gap between how current repair agents assess issue conditions and how they perform during repair, highlighting the need for judgment-aware and correction-aware agent design.
Mingyue Yuan, Jieshan Chen, Dehai Zhao et al.· ACM Transactions on Software...· 0 citations
Deep learning models that predict molecular phenotypes directly from DNA sequence offer a powerful framework for interpreting genomic variation. Recently, AlphaGenome was introduced as a deep sequence-to-function architecture capable of predicting observations that historically required experiments. While the model has shown high accuracy, it was primarily evaluated on human variants scored against a reference genome. Here, we test performance on mouse data, the other species AlphaGenome was trained on although with fivefold fewer features than human (1,128 versus 5,930). We demonstrate that AlphaGenome’s predictive performance varies considerably depending on the functional task. Specifically, predicted quantitative expression effects are directionally weak and compressed roughly 100-fold relative to empirical benchmarks across both reconstructed-haplotype and single-variant regimes. In contrast, canonical splice-site disruptions are recognized with near-identical accuracy in mouse and human (AUC 0.96 versus 0.98), displaying no cross-species divergence in predicted effect magnitude. We developed a scoring-approach for AI-agents to autonomously assess AlphaGenome prediction confidence and accurately differentiate between AlphaGenome’s robust sequence-level recognition across species and its current limitations when interpreting un-fine-mapped regulatory variants. This demonstrates how GenAI innovations that are still under development can safely be harnessed by wrapping a responsible AI layer around the call to intercept flawed results, thereby adhering to international standards, such as the Australian Voluntary AI Safety Standard (VAISS).
Priya Ramarao-Milne, Suyu Ma, L. Sng et al.· bioRxiv· 0 citations