Skip to content
Log in
Preprint

From Failing to Passing: Evolving Natural Language Prompt Optimization Rules for LLM Code Generation

Amal Akli Melissa Akli Cedric Richter Mike Papadakis Y. Traon
Jul 2026 · 0 citations · 38 references
Computer Science

TL;DR

This work introduces a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance, and proposes DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures.

Abstract

Large language models are known to be sensitive to prompt formulation. Even minor variations in wording can substantially degrade performance. This sensitivity reveals an opportunity: if prompt phrasing can harm performance, can it be used to improve it? To investigate this question, we introduce a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance. We then propose DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures. A key strength of our approach lies in its generality: the evolved rules are error-agnostic, reusable across problems, and transferable across models. We evaluate DUALFIX against execution-feedback repair baselines across three models on two challenging benchmarks, LiveCodeBench and APPS. Our results show that the evolved transformations fix from 10-30% of failing cases, including 12-17% of failures that execution-based repair alone cannot resolve. Overall, DualFix recovers up to 30% of baseline failures and fixes 3-5 times more failing cases than Self-Fix across all evaluated settings. Furthermore, we also show that rules evolved on one model transfer zero-shot to other models, outperforming execution-feedback repair without any re-optimization.

arXiv Semantic Scholar
View source

Similar papers

Preprint Aug 2026

Route-Align-Verify for Functional Correctness in Code Generation

The results indicate that functional correctness in code generation can be meaningfully improved without modifying the backbone architecture, by jointly optimizing how tasks are prompted, how the model is adapted, and how final outputs are selected.

Erxue Zhou, Jing Meng, Aofan Liu · 0 citations
Conference Jul 2026

Toward Reliable LLM Code Generation: Adaptive Routing Framework for Ambiguous Requirements

Natural-language requirements for program synthesis are often incomplete or ambiguous, yet large language models are commonly expected to generate code in a single pass. Prior clarification-based methods address this issue by asking follow-up questions when sampled candidate programs disagree, but fixed clarify-on-disagreement policies can overuse clarification and can also overtrust weak behavioral agreement. We present an adaptive routing framework for LLM-based program synthesis that treats clarification as an inference-time control decision. The framework augments a ClarifyGPT-style pipeline with execution-driven confidence estimation, semanticdifference analysis, and bounded candidate expansion, allowing the system to choose among direct generation, additional evidence gathering, and clarification. We evaluate the framework on MBPP, HumanEval, and extended-test variants using GPT-4.1 mini, Claude Haiku 4.5, and GPT-5.4 mini. Adaptive routing improves pass@1 accuracy by up to 7.60 percentage points over single-pass baselines. Compared with fixed-policy clarification, it preserves accuracy while reducing token usage by up to 57.2% for GPT-4.1 mini, and reallocates computation toward harder cases for Claude Haiku 4.5. These results suggest that clarification is most useful when triggered selectively based on execution evidence and semantic disagreement, even when ambiguity is observed indirectly through candidate behavior rather than through explicitly annotated ambiguous requirements.

Muhammad Ahmed, Edwar Tiu, Niyati Nikunj Kapadia et al. · 0 citations
Review Aug 2026

Refine After Generation: Toward Correct and Concise Patches in LLM-based Program Repair

This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.

Wenqiang Luo, J. Keung, Xiaoyu Shi et al. · 0 citations
Open access Aug 2026

How well do LLMs understand code?

SemBench is introduced, a novel benchmark consisting of 1000 diverse C programs sourced from the CodeParrot GitHub-code dataset, with 15,404 semantic questions spanning six basic but fundamental properties: dead code-statement, data dependency, function reachability, dominator, dead code-loop, and liveness.

Jade Xu, Renliang Sun, Zijian Ding et al. · 0 citations