Skip to content

CodeSkill: Latent Skill Abstraction for Long-Horizon Code Agents

Sep 2026 · 0 citations · 47 references
Computer Science

TL;DR

By shifting RL from token-level exploration to experience-level reasoning, CodeSkill improves optimization efficiency and long-horizon behavioral coherence, highlighting the effectiveness of explicit behavioral abstraction for scalable agentic code generation.

Abstract

Code agents require long-horizon decision-making over complex interaction trajectories. However, existing reinforcement learning (RL) approaches typically optimize behavior at the token level, creating a mismatch between low-level generation and high-level behavioral reasoning. This limitation leads to inefficient exploration and weak credit assignment under sparse rewards. Moreover, while large-scale agent trajectories often contain recurring multi-step behavioral patterns, their noisy token-level representations hinder effective experience reuse. To address these challenges, we propose CodeSkill, a framework that adapts hierarchical latent skill modeling to the code agent domain. CodeSkill first leverages a teacher model to distill both successful and failed trajectories into multi-level textual abstractions. It then integrates temporal variational inference with reinforcement learning to map these discrete semantics into continuous latent variables, while an adaptive boundary mechanism dynamically gates skill transitions based on execution feedback. The learned skills are injected into a frozen LLM policy as latent semantic prefixes, enabling optimization in a compact semantic space rather than over raw token sequences. By shifting RL from token-level exploration to experience-level reasoning, CodeSkill improves optimization efficiency and long-horizon behavioral coherence. Extensive experiments demonstrate that CodeSkill achieves highly competitive performance against strong open-weight baselines across diverse general and industrial coding benchmarks. Furthermore, the learned skills exhibit strong transferability and robust cross-domain generalization, highlighting the effectiveness of explicit behavioral abstraction for scalable agentic code generation.

View source

Similar papers

Oct 2026

Enhancing Language Model Reasoning with Structured Multi-Level Modeling

Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, e...

Siheng Xiong, Ali Payani, F. Fekri · 19 citations · ⚡1
#artificial intelligence Preprint Oct 2026

From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, lea...

Sheng Lu, Zi-Xuan Yang, Rong-Xu Zhao et al. · 0 citations
Preprint Oct 2026

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a traj...

Zhengping Jiang, William Jurayj, Liaoyaqi Wang et al. · 0 citations
Preprint Oct 2026

Productive State-Space Exploration for Multi-turn RL in LLM Agents

Reinforcement Learning (RL) for Large Language Model (LLM) agents in multi-turn, long-horizon environments remains challenging due to inefficient action selection and repetitive behavioral patterns. In particular, agents often become trapped in high-confidence yet unproductive behaviors, repeatedly executing invalid ac...

Mohammed Haroon Dupty, Yong-Liang Goh, Song Guo et al. · 0 citations
Review Open access Sep 2026

Reinforcement Learning Post-Training for Reasoning Large Language Models: Methods, Systems, and Evaluation

This survey develops a mechanism-driven framework for separating effects of reinforcement learning, and organizes methods along three axes— learning-signal type, policy/data regime, and optimization granularity—and decomposes their training stacks into reusable motifs spanning advantage construction, drift control, den...

Liu Yang, Han Zhu, Zheng-Yang Zhong et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.