Deep learning models have shown significant promise in code-related tasks. While confidence calibration techniques have been proposed to improve model reliability, their effectiveness in real-world tasks with strict accuracy requirements remains uncertain. In this paper, we investigate two key questions: first, whether...
Large language models (LLMs) are increasingly used for automated code generation, but generated programs can appear syntactically plausible while still failing execution-based correctness checks. Existing validation methods, such as testing and program analysis, remain essential but are often incomplete, costly, or app...
RankBuffer is introduced, which maintains an ordered, query-specific buffer of previously judged responses as a reusable quality scale that achieves nearly on-par performance with the strongest ranking-based reward baseline while substantially reducing judging cost.
Zi-Xuan Yang, Yi-Qun Chen, Qi Liu et al.· 0 citations
WatchPoint is introduced, a simulated-user system that mimics real developer behavior by generating and executing diagnostic scripts against the running application, producing structured observations that guide the coding model's retry and identifies a pattern of capability gaps that governs when simulated-user feedbac...
This study studies whether correctness-based evaluation identifies kernels unsuitable as library replacements, why such failures occur, and how they can be detected without exhaustive benchmark coverage.
