Analysis of instruction-preserving paraphrases shows that execution robustness is a distinct, model-dependent property not captured by accuracy alone and highlights the importance of evaluating prompt behavior under paraphrase in structured-output settings.
Experiments on ESCI-English show that free-generation and likelihood-based forced-choice evaluation can yield qualitatively different robustness conclusions even when valid-label generation is nearly perfect and average task performance is similar, and SFT does not uniformly reduce instruction sensitivity.
System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA)...
Muhammad Usama, Dong Eui Chang· arXiv.org· 0 citations
Face/Off is introduced, a semantics-preserving identifier-renaming framework, and progressive naming conditions across multiple models and code-comprehension tasks are evaluated, revealing a systematic vulnerability in how current LLMs balance lexical cues against program structure.
Xin-Peng Shen, San-Zhuo Xi, Ya-Li Du et al.· arXiv.org· 0 citations
Large Language Models (LLMs) are increasingly incorporated into verifier-in-the-loop software engineering systems, yet it remains unclear how the representation of verifier feedback affects their ability to repair incorrect programs. We study this question by isolating feedback representation from model, task, and prom...
Tao-Er Guo· Proceedings of the 7th Inter...· 0 citations
A graded, multi-family, failure-aware framework for stress-testing reasoning models that exposes structure that an aggregate score hides and exposes consistent weaknesses across all models.
This work investigates whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity, and identifies a difficulty-dependent breakdown in equivalence judgment.
Hui Sun, Anderson G. Uchôa, Rohit Gheyi et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.