It is shown that prompt engineering "ages" in a model-family-specific way: Newer GPT models exhibit diminishing or even negative marginal gains from structured prompting, suggesting that instruction-following and reasoning scaffolds are increasingly internalized, whereas Qwen models continue to benefit substantially from Few-Shot and CCoT.
Abstract
Prompt engineering and prompt engineering techniques (PETs) have become an integral part of software engineering for AI systems. However, new LLMs are released frequently and it remains unclear how the effectiveness of prompt engineering techniques changes across successive generations of Large Language Models (LLMs). To this end, we conduct a partial replication of the study by Khojah et al. (2025). We evaluate five techniques - Zero-Shot, Few-Shot, Chain-of-Thought (CoT), Contrastive Chain-of-Thought (CCoT), and an adapted version of Program-of-Thought (PoT) - on six instruction-tuned models grouped into three version pairs: GPT-3.5-Turbo/GPT-4o, Qwen2 7B Instruct/Qwen2.5 7B Instruct, and Mistral-7B-Instruct/Mistral-Large. We use a cleaned subset of the CodePromptEval dataset with 218 context-rich Python functions and 19,620 total generations assessed via pass@k-based functional correctness to evaluate model pairs on function-level code generation tasks. We show that prompt engineering"ages"in a model-family-specific way: Newer GPT models exhibit diminishing or even negative marginal gains from structured prompting, suggesting that instruction-following and reasoning scaffolds are increasingly internalized, whereas Qwen models continue to benefit substantially from Few-Shot and CCoT. Mistral models show mixed behavior with persistent gains from CCoT but attenuated benefits from CoT and PoT. Our results imply that effective prompting strategies must be adapted per model family and generation rather than transferred unchanged. This motivates future work on adaptive, model-aware prompting and broader, multi-dimensional code quality evaluation.
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wenqiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
This work introduces a search-based approach that identifies and evolves a set of natural language transformation rules with strong downstream effects on coding performance, and proposes DUALFIX, a staged repair pipeline that combines the evolved transformation rules with execution-feedback repair, addressing both specification-level and implementation-level failures.
Amal Akli, Melissa Akli, Cedric Richter et al.· 0 citations
This report synthesizes peer-reviewed and archival research published between 2019 and 2024 on the use of large language models and multi-agent architectures for automated vulnerability repair, with a specific focus on how these techniques apply, or fail to apply, to legacy software systems. The review traces a clear methodological progression across this five-year window: from template and heuristic program repair, through neural and transfer-learning models, to fine-tuned transformer-based repair, zero-shot and conversational prompting of general-purpose LLMs, and finally to multi-agent pipelines in which specialized agents divide the work of detecting, localizing, patching, and validating a vulnerability. Two figures accompany the discussion. The first depicts a generalized multi-agent repair pipeline synthesized from the architectures described across the reviewed literature, showing how detection, localization, patch generation, and validation agents typically interact, including the validation feedback loop that distinguishes conversational and agentic systems from earlier single-shot approaches. The second traces the field's methodological evolution chronologically, from 2019's template-based baselines through 2024's agentic systems. A comparative table consolidates fifteen of the most relevant studies discussed in the body of the report, listing the approach, target language, and publication venue for each. The report concludes that while multi-agent LLM pipelines show real promise for legacy code specifically, since their capacity for autonomous context reconstruction and iterative, feedback-driven refinement directly addresses the sparse documentation and thin test coverage that define legacy environments, the literature to date has been validated almost exclusively on actively maintained, comparatively well-organized open-source repositories. The gap between these benchmarks and the scale and disorganization of real enterprise legacy systems has not yet been closed empirically, and closing it, together with building trust and explainability mechanisms suitable for security-critical maintenance work, is identified as the field's most consequential open problem.
Harsh Verma· International Journal of Sci...· 0 citations
An empirical study of configuration prompt files in Cursor, a widely used AI-assisted code editor, shows that .cursorrules files emerged rapidly from mid-2024 and shows that there is a continuity of themes and topics between the now-legacy .cursorrules files and the current standard .mdc files.
Shuang Sun, Jafar Akhoundali, Arina Kudriavtseva et al.· Proceedings of the 21st Inte...· 1 citation· ⚡1