Skip to content

Probing Instruction Execution Stability of Large Language Models under Semantic Paraphrase

· 2 citations · 12 references

TL;DR

Analysis of instruction-preserving paraphrases shows that execution robustness is a distinct, model-dependent property not captured by accuracy alone and highlights the importance of evaluating prompt behavior under paraphrase in structured-output settings.

View source

Similar papers

Preprint Aug 2026

When Does Supervised Fine-Tuning Reduce Instruction Sensitivity?

Experiments on ESCI-English show that free-generation and likelihood-based forced-choice evaluation can yield qualitatively different robustness conclusions even when valid-label generation is nearly perfect and average task performance is similar, and SFT does not uniformly reduce instruction sensitivity.

Jaekeol Choi · 0 citations

The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models

System prompts are the primary lever practitioners use to control language model behavior, yet what they actually do to the computation inside the transformer remains poorly understood. Across 17 instruction-tuned models spanning 8 architecture families and 1.5B to 72B parameters, we use Centered Kernel Alignment (CKA)...

Muhammad Usama, Dong Eui Chang · 0 citations

On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality

Face/Off is introduced, a semantics-preserving identifier-renaming framework, and progressive naming conditions across multiple models and code-comprehension tasks are evaluated, revealing a systematic vulnerability in how current LLMs balance lexical cues against program structure.

Xin-Peng Shen, San-Zhuo Xi, Ya-Li Du et al. · 0 citations
Book Open access Oct 2026

Evaluating Symbolic Feedback Granularity for LLM-Based Program Repair

Large Language Models (LLMs) are increasingly incorporated into verifier-in-the-loop software engineering systems, yet it remains unclear how the representation of verifier feedback affects their ability to repair incorrect programs. We study this question by isolating feedback representation from model, task, and prom...

Tao-Er Guo · 0 citations
Preprint Aug 2026

Evaluating Language Models on Cross-Language Code Functional Equivalence

This work investigates whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity, and identifies a difficulty-dependent breakdown in equivalence judgment.

Hui Sun, Anderson G. Uchôa, Rohit Gheyi et al. · 2 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.