When Words Change the Model: Sensitivity of LLMs for Constraint Programming Modelling

Abstract

Abstract Automatically deriving executable and efficient constraint programming models from natural language descriptions has long been a central goal in optimisation and constraint programming. Large Language Models (LLMs) appear to bring this vision closer, showing promising results in generating models for classical benchmark problems. However, part of this apparent success may stem from data contamination rather than genuine reasoning capabilities, as many standard CP problems are likely represented in the training data of modern LLMs. In this initial investigation, we study the robustness of LLM-based constraint modelling by rephrasing and perturbing a set of 10 constraint problems while preserving their underlying structure. The modifications alter contextual elements and introduce misleading or distracting information to explore whether models rely on semantic understanding or superficial textual cues. We compare the models generated by three representative LLMs across the original and modified problem descriptions. Our qualitative analysis shows that, although LLMs frequently produce syntactically valid and semantically plausible models, their performance degrades significantly under contextual and linguistic variation, suggesting strong sensitivity to wording and contextual cues.