Self-Adaptive Code Representation: Grammar-Guided or Token-Based?

Abstract

Recent advances in code intelligence are largely driven by large language models (LLMs), which represent code as token sequences and achieve strong results on many software engineering tasks. However, standard token-based approaches often struggle to capture the inherent structural regularities of programming languages. While grammar-guided approaches can alleviate this limitation by modeling syntax more explicitly, they may sacrifice flexibility. In this work, we propose the Self-Adaptive Code Representation (SACR) framework, which enables models to dynamically select between token-based and grammar-guided representations depending on the problem context. SACR operates in two stages: first, the model is jointly trained on both representations to learn their respective strengths; second, an adaptive mechanism quantifies the generation difficulty for each sample and selects the model’s preferred code representation for training. We implement SACR in three versions of SacrCoder with parameter sizes of 0.5B, 1.3B, and 3B, and evaluate their effectiveness across a range of code generation and code repair benchmarks, including two repository-level pragmatic benchmarks built from real open-source projects. Through empirical analysis, we first demonstrate that token-based and grammar-guided code representations exhibit complementary strengths. Building on this observation, SACR selectively exploits these complementary representations during training to improve overall performance. Experimental results show that SacrCoder consistently achieves state-of-the-art performance. For example, the 3B model obtains a Pass@1 of 84.1% on HumanEval and 30.3% on APPS, corresponding to relative improvements of 9.5% and 32.3% over grammar-only and dual-format baselines, respectively, and SacrCoder also outperforms all single-representation baselines on the real-world programming benchmarks at every scale. Further ablation studies confirm that the effectiveness of SACR in leveraging complementary code representations.