Skip to content

Learning Native Reflection in Unified Models with Interleaved Reinforcement Learning

Sep 2026 · 0 citations · 45 references
Computer Science

TL;DR

UMM-Reflection is introduced, which applies reinforcement learning (RL) to complete reflection trajectories inside one unified model: sibling trajectories share one initial image, so the group-relative advantage compares reflection strategies, and one trajectory-level advantage updates both the reflection tokens and the flow-based revisions, avoiding the combinatorial blow-up of per-round credit assignment.

Abstract

Unified multimodal models can both look at and render images, so in principle they can repair their own generations: diagnose what an image gets wrong, revise it, observe the result, and diagnose again. Whether a revision helps is known only after it is rendered, so the reflection text and the image generation must be learned jointly, over the whole loop. Supervised fine-tuning (SFT) on reflection trajectories gives a cold start but does not find the high-success repair paths, and naive RL that optimizes only the renderer or only one head leaves most of the gain untapped. We introduce UMM-Reflection, which applies reinforcement learning (RL) to complete reflection trajectories inside one unified model: sibling trajectories share one initial image, so the group-relative advantage compares reflection strategies, and one trajectory-level advantage updates both the reflection tokens and the flow-based revisions, avoiding the combinatorial blow-up of per-round credit assignment. Unlike single-round editing or pipelines with an external critic, credit flows across rounds and to both roles of the same model, and no verifier is needed at inference. On BAGEL, UMM-Reflection improves GenEval by 12.05 points over SFT, and the gains transfer to WISE (+10.97), OneIG-Bench (+3.48), and T2I-CompBench++ (+4.63), none of which is used in training.

View source

Similar papers

Review Open access Aug 2026

Reinforcement Learning for Multimodal Foundation Models: A Survey

A coherent map of the rapidly expanding landscape of visual RL is provided to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry.

Weijia Wu, Chen Gao, Joya Chen et al. · 0 citations
Preprint Aug 2026

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

LoongReflect is proposed, a training framework that formulates reflection as a memory-control policy over a reversible trajectory tree using explicit reflect and backtrack actions and demonstrates consistent improvements over outcome-only reinforcement learning and self-distillation baselines.

Zhi-Xin Zhang, Xin-Ke Jiang, Zhi-Bang Yang et al. · 0 citations
Preprint Aug 2026

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

Noise-Contrastive GRPO is introduced, which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

Michael M. Jerge, Joseph Pelczar, J. Downes · 1 citation
#artificial intelligence Preprint Oct 2026

Selective Transfer of RL Updates for Visual Reasoning

This work isolates the RL-stage update, retains its dominant matrix-wise directions with magnitude preservation, and transfers them to the language modules of a VLM, which improves full-update interpolation in 12 of 15 comparisons, including an 8.55 percentage-point MathVision gain on the Qwen recipient.

Su-Xin Ji, Hung-Tao Wan, Ming-Jun Liu et al. · 0 citations
#machine learning Preprint Oct 2026

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

This work improves the two components of training that shape these fixed points: the depth prior and input injection, and learns the prior from prediction feedback, with an entropy term that keeps it broad.

Benhao Huang, Chu-Fan Shi, Jun-Lin Chen et al. · 5 citations · ⚡1
#artificial intelligence Preprint Sep 2026

UniEvo-VL: An On-policy Self-Distillation Training Recipe for Multimodal Model Self-improvement

UniEvo-VL, a self-evolving framework for multimodal models to learn from this constructive self-correction feedback during test-time compute, is introduced to enhance the user experience when using multimodal models without external supervision or guidance.

Fang Wu, Dan-Lei Xing, Yan-Jie Huang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.