What really happens when a dev vibes with the code? An empirical study on LLM behavioral divergence in response to expressive code comments
TL;DR
These findings position inline comments as model-sensitive latent semantic prompts, with implications for AI-in-the-loop development and design of comment conventions for AI-assisted maintenance.
Abstract
Introduction We investigate how expressive inline code comments written in various developer styles, functional to progressively poetic, philosophical, and misleading, affect large language model (LLM) behavior during code optimization. Methods In this pilot study, we used a controlledmerge sort implementation across five stylistic variants and evaluated GPT-5 and Claude Opus 4.1 under standardized console prompts, isolating the effect of embedded comment semiotic variation. Seven expert developers (three senior, four mid-level) scored model outputs against adapted ISO/IEC 25010 criteria and novel LLM suggestibility index (LSI) framework. Results Semiotic character of comments measurably altered code quality, with consensus-score reliability ICC(2, k) = 0.65–0.81 for six of seven dimensions; single-rater Krippendorff's α = 0.232 reflects substantial interpretive variability. Claude exhibited higher interpretive sensitivity (mean behavioral divergence 4.00; SD 1.16), while GPT-5 maintained stronger architectural fidelity (mean divergence 3.58; SD 1.26). Reflective comments (philosophical, conversational) were associated with Claude's highest maintainability scores in our panel (both M = 4.00, ~8% above stock M = 3.71), while the same philosophical comments reduced GPT-5 maintainability (M = 2.86), suggesting asymmetric model responses to expressive context. Conclusions These findings position inline comments as model-sensitive latent semantic prompts, with implications for AI-in-the-loop development and design of comment conventions for AI-assisted maintenance.