Skip to content

GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

Aug 2026 · arXiv.org · Vol abs/2608.11674 · 0 citations · 26 references
Computer Science

TL;DR

GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction, consistently outperforms GRPO and recent variants.

Abstract

On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.

View source

Similar papers

GRPO Training Dynamics for Small Language Models

Group Relative Policy Optimization (GRPO) has emerged as a memory-efficient reinforcement fine-tuning (RFT) technique for reasoning-intensive tasks. How- ever, GRPO training dynamics on small language models (SLMs) remain poorly understood, limiting its reliable adoption and reproducibility in open and resource- constr...

Rajat Ghosh, Vaishnavi Bhargava, Henry Wong et al. · 0 citations
#natural language process... Preprint Oct 2026

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

Reinforcement learning has greatly advanced the capabilities of large language models, but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations su...

Shao-Kun Zhang, Yi-Fan Zhang, Jian Hu et al. · 1 citation · ⚡1
#artificial intelligence Preprint Nov 2026

Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yield...

Hei-Yi Mak, H. Le, Mehran Taghian Jazi et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

The Alpha-Stabler framework is proposed, a plug-and-play framework with a Predictor that monitors principal-subspace intrusion for early collapse warnings, and a Controller that removes the principal-subspace component of activation gradients during backpropagation while preserving the orthogonal complement.

Yu-Chen Cai, Ding Cao, Qi-Xiang Yin et al. · 1 citation
#artificial intelligence Preprint Oct 2026

RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning

Reinforcement learning is crucial for improving large language models'reasoning and generalization. It relies on massive rollouts whose lengths become increasingly long-tailed as context windows grow. In on-policy training, these long-tail rollouts can result in GPU bubbles, reducing system utilization and limiting RL...

Yong-Qiang Yao, Jing-Ru Tan, Kai-Huan Liang et al. · 1 citation
#machine learning Preprint Oct 2026

Mixture of GFlowNets

This work explores whether this mode-bias problem can be addressed with a population of asynchronously trained policies which divide the state space between them, and evaluates both methods by keeping the environment-interaction budget fixed while scaling the community size, turning parallel compute into mode coverage...

Omar G. Younis, Joseph D. Viviano, Chirayu Anant Haryan et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.