Readout Feedback (RoFB), a test-time intervention that converts intermediate predictions into token-wise pairwise coupling forces injected into the latent dynamics, is introduced, suggesting that closed-loop steering of latent dynamics can serve as a complementary inference-time control mechanism for recurrent reasoning models.
Abstract
Recurrent models, which repeatedly update latent states with shared computation blocks, have emerged as powerful architectures for solving complex reasoning tasks. Existing inference-time methods scale computation by running more steps or sampling more trajectories, but ignore information revealed within each trajectory. Here we show that recurrent models can be improved at inference time by using their own readout probabilities to steer latent dynamics without retraining. We introduce Readout Feedback (RoFB), a test-time intervention that converts intermediate predictions into token-wise pairwise coupling forces injected into the latent dynamics. Across three recurrent models (AKOrN, ItrSA++, TRM) on Sudoku and Maze, RoFB yields clear gains in four of six model-task pairs, achieving performance unattainable by merely running more steps or selecting from multiple trajectories, at comparable or lower computational cost. These results suggest that closed-loop steering of latent dynamics can serve as a complementary inference-time control mechanism for recurrent reasoning models.
Penelope is introduced, an efficient latent-reasoning framework for pretrained decoder-only Transformers that localizes recurrent computation to a selected decoder interval and attains competitive accuracy relative to established latent-reasoning models while reducing measured inference latency.
Yutong Chen, Shouqian Shi, Xinran Liu et al.· 0 citations
Large language models (LLMs) possess extensive latent knowledge yet remain largely static at inference. Once prompted, their generation policy typically cannot evolve, and post-hoc ''self-reflection'' methods provide no explicit principled learning signals. To address this limitation, we formally model iterative research idea optimization as a finite-horizon Markov Decision Process and propose InfRL (Inference-time Reinforcement Learning), a framework designed for effective policy improvement at inference without updating model weights. InfRL coordinates three specialized LLM agents: (i) a State Transition Agent that proposes candidate ideas; (ii) a Policy Update Agent that learns feedback strategies based on idea trajectories; and (iii) a Reward Agent that assigns normalized, comparative rewards, enabling nuanced reinforcement signals. This empowers the Policy Update Agent to dynamically refine feedback strategies, progressively leveraging latent knowledge encoded within the LLM. We evaluate InfRL on five balanced datasets covering 500 recent research papers from health, genetics, environment, neuroscience, and engineering domains. Compared to a single-pass GPT-4o baseline and a strong self-reflection baseline, InfRL improves the novelty of generated ideas by 3.93%--32.7% and the feasibility of generated ideas by 70.1%--448.6% with GPT-4o, exhibiting consistent improvements across ten inference-time iterations. Ablation studies underscore the critical roles of our reward formulation and modular agent design, while human evaluations confirm alignment between reward trends and perceived idea quality. InfRL thus offers a practical and domain-agnostic approach to harness reinforcement learning during inference, bridging the gap between static prompting and computationally intensive parameter-level fine-tuning. The code and the dataset we use are provided at: https://github.com/amir-hassan25/InfRL
Sikun Guo, Amir Hassan Shariatmadari, Jiuqi Wang et al.· Proceedings of the 32nd ACM...· 0 citations
Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive"reasoning."However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$\pi$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.
Chuning Zhu, E. Xu, Jose A. Barreiros et al.· 0 citations
World models are becoming core infrastructure for embodied intelligence, with action-conditioned video generation providing controllable predictions of how scenes evolve after agent interventions. Yet existing models are commonly trained with space-time-uniform mean squared error, allowing abundant background tokens to dominate the gradient while sparse interaction dynamics remain under-optimized; such uniform fitting rewards reconstructing appearance rather than learning how actions change the world. We introduce Causal Action Effect Reweighting (CAER), a general training paradigm that redistributes supervision toward the tokens whose predicted future is causally affected by the action. CAER contrasts the model's own predictions with and without action conditioning to localize these tokens online, then normalizes the resulting effect map into a weight that preserves the total coefficient mass and changes only where it is spent. This online signal requires no external annotations or offline preprocessing, avoids additional data-processing time, and scales naturally with model and dataset size. Experiments across heterogeneous action-conditioned world-model tasks show that CAER converges to better solutions than uniform MSE training, with consistent improvements in the physical consistency, controllability, and visual quality of generated videos.
Jianjie Fang, Xvyuan Liu, Ziyou Wang et al.· 0 citations
Results show that the benefits of recurrent depth can be retained without repeatedly executing the dense FFN, and view recurrence as repeated composition of a state update and argue that an application is valuable when it exposes a new cross-position influence direction that remains observable at the task readout.
Ruhai Lin, Yi Guo, Ruiming Zhu et al.· 0 citations
This thesis proposes a unified two-axis framework that organizes SFT and RL methods along a data axis (off-policy to on-policy) and a loss function axis (positive-only to positive-plus-negative to GRPO) and enables controlled ablations of individual components.
G. Kim, Chair Chenyan Xiong, Aditi Raghunathan· 0 citations