Skip to content

Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning

Sep 2026 · 0 citations · 31 references
Computer Science

TL;DR

WIDER is proposed, a lightweight spectral regularizer for implicit CoT that improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps.

Abstract

Chain-of-thought (CoT) reasoning improves the reasoning ability of large language models by introducing intermediate computation, but explicit rationales increase decoding length, latency, and context cost. Implicit CoT offers a more efficient alternative by moving intermediate reasoning into continuous latent states. However, latent reasoning can be unstable: successive latent states may become overly similar and collapse toward a shared dominant direction, reducing the diversity of the reasoning trajectory. In this work, we identify $\textit{latent rank collapse}$ and propose $\textbf{WIDER}$, a lightweight spectral regularizer for implicit CoT. During training, WIDER estimates the shared direction of each latent trajectory and penalizes projections onto this direction, encouraging latent states to span a broader representational subspace. The method is plug-and-play and leaves the backbone model, latent schedule, and inference-time decoding procedure unchanged. We further formulate this collapse as a geometric bottleneck in implicit reasoning, casting its mitigation as a training-time regularization problem rather than an inference-time decoding change. Extensive experiments show that WIDER improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps. These results highlight latent subspace utilization as an important factor for efficient continuous reasoning, providing a geometric perspective for analyzing and improving implicit CoT. Code is available at https://github.com/whitesweater/WIDER.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM

A*-Thought-V2 is presented, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture that reflects broader soft targets that encourage richer step-level feature learning.

Xiao-An Xu, Si-Yuan Liu, Shuo Wang et al. · 1 citation
Preprint Aug 2026

ChainPrune: Evaluating and Reducing Redundancy in Long Chain-of-Thought Reasoning

This work proposes ChainPrune, a novel reasoning path semantic structural optimization method to efficiently and controllably synthesize self-generated high-quality training data and incorporates a DPO-based preference learning method combined with supervised loss, effectively mitigating false reward suppression.

Wei-Hang Pan, Zhengxu Yu, Yuxiang Zhang et al. · 2 citations
#artificial intelligence Preprint Sep 2026

Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs

Latent Recurrent Thoughts substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.

Zhao-Xing Chen, Jie Fu · 0 citations
#artificial intelligence Preprint Sep 2026

When Steering Fails in Latent Reasoning: A Latent-to-Language Transition Gap

Activation steering has become a widely used approach for controlling language models during explicit chain-of-thought (CoT) reasoning, motivating its extension to latent CoT. However, we find that steering continuous thoughts produces substantially weaker effects on subsequent language generation than steering explici...

Gao-Xiang Huang, Lei Qi · 0 citations
#artificial intelligence Preprint Sep 2026

Structural Process Supervision for Latent Chain-of-Thought Reasoning

The proposed Prototype-Mediated Process Supervision (PMPS), which introduces learnable reasoning prototypes as semantic anchors to provide structural process-level supervision for latent reasoning, consistently attains the highest accuracy among all latent reasoning methods with comparable output length.

Yi-Qi Li, Xu Chen, Chen Ju et al. · 1 citation

Efficient Reasoning via Constrained Optimization in Latent Space

This paper proposes a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance, and investigates the latent representations and proposes a novel training-free framework.

Zhi-Nan Hou, Xing-Chen Li, Ke-You You · 2 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.