BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention is proposed.
Abstract
Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited. While hybrid architectures integrating MLLMs with diffusion backbones have shown strong advantages in image synthesis, such designs remain underexplored in video generation, where existing approaches often treat MLLMs primarily as frozen feature encoders rather than semantic generators. To fill this gap, we systematically study how an MLLM should be integrated with a DiT for video generation by answering three questions: what intermediate representation should bridge the MLLM and DiT, how the MLLM should generate it, and how the DiT should incorporate it during diffusion rendering. Our analysis reveals three key findings: (1) discrete semantic visual tokens produced by an EMA-based tokenizer provide a stable and expressive interface, (2) autoregressive causal modeling is effective for generating these tokens, and (3) explicit visual-token conditioning is more effective than prompt refinement or latent bridging. Based on these findings, we propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention. Extensive experiments show that BiVidGen improves semantic alignment and temporal coherence over a fine-tuned DiT baseline, achieving stronger performance on VBench-Long. These results demonstrate that explicit MLLM-based visual planning provides an effective intermediate interface for text-to-video generation beyond text-only conditioning.
A training-free, interpretable mid-generation correction framework that integrates multimodal large language model (MLLM) feedback directly into the diffusion sampling loop and achieves diffusion trajectory correction by injecting semantic evaluation signals during video synthesis, enabling the model to optimize the generated content through continuous self-reflection.
Junhao Chen, Zheqi Lv, Keting Yin et al.· 0 citations
This work proposes to erase concepts by directly manipulating the model's internal representations by operating exclusively on the sparse text-branch tokens and leveraging the straight sampling trajectory of rectified flow, achieving effective concept erasure with negligible overhead and without any training.
Qiao Li, Xiaomeng Fu, Yuanshu Zhao et al.· 0 citations
Multimodal Large Language Models (MLLMs) have achieved strong progress in video understanding, yet it remains challenging because the token limitation makes MLLMs difficult to capture temporally sparse evidence. Existing methods typically rely on uniform sampling, or frame selection, but these strategies usually optimize either broad temporal coverage or local relevance, making it difficult to preserve both global storyline context and fine-grained evidence. We propose VideoRouter(VR) that rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. It first organizes each video into a question-agnostic temporal hierarchy, which partition the video into coarse-to-fine temporally coherent segments. In this hierarchy, upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This naturally gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router to determine which view is better supported by the selected evidence and select the final answer. We validate the effectiveness of the proposed design through extensive experiments, showing that the verification-guided router effectively coordinates global and local reasoning, and that, on VideoMME, our method outperforms state-of-the-art frame selection methods by 2.9 points, respectively, under the LLaVA-Video-7B backbone. We will release the code.
The inference performance of Multimodal Large Language Models (MLLMs) typically relies on a massive number of visual block tokens, leading to substantial computational overhead. However, existing training-free compression methods often sacrifice semantic relevance, disrupt spatial coherence, or require extensive architectural modifications. Consequently, we propose Text-Conditioned Spectral Prototyping for Multimodal Token Compression (TCSP). We construct a joint affinity matrix based on both text relevance and visual similarity to derive boundary-continuous and semantically aligned segments via spectral embedding, subsequently replacing redundant information within these segments with weighted prototypes to reduce visual tokens for downstream inference. This method is entirely training-free and plug-and-play, allowing it to be inserted into intermediate layers of the vision encoder or before/after the projector module as needed. Evaluations on LLaVA-1.5-7B demonstrate that after a approximately $3 \times$ reduction in visual tokens, the overall performance remains baseline levels, even achieving marginal denoising-like improvements in several general QA benchmarks. More aggressive compression results in a graceful performance degradation, whereas tasks emphasizing compositional reasoning exhibit higher sensitivity to the compression scale. TCSP integrates text-awareness, spectral partitioning, and prototype replacement within a minimally invasive design. It achieves high, near-lossless compression ratios and a robust accuracy-efficiency trade-off without training, providing a concise and reliable pathway for general reasoning and resource-constrained deployment.
Text-Image-to-Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-video (T2V) and image-to-video (I2V) generation. Given a high-quality first frame or a detailed textual prompt, TI2V models unlock substantially better visual quality than their T2V mode, raising a natural question: can the capability elicited by such privileged conditions be internalized into the model's own base generation ability? A common approach toward this goal is model self-distillation. However, the most straightforward solution, supervised fine-tuning, follows an off-policy strategy: its supervision is confined to teacher-generated endpoints from a fixed offline distribution rather than student-visited states, lacking precise correction tailored to the evolving policy. Recent on-policy distillation methods instead suffer from condition-state mismatch, where supervision is steered toward the given first frame instead of the student's actual content, misleading the correction. To achieve self-distillation that absorbs the teacher's privileged prior while retaining precise policy correction, in this work, we propose Hybrid-Policy Self-Distillation (HPSD), a novel self-distillation framework where a single TI2V model acts as both teacher and student under different conditions: the teacher operates in TI2V mode with a high-quality first frame and an enhanced prompt, while the student runs in the base T2V mode with only the vanilla prompt. Specifically, the student inherits off-policy teacher trajectory points as anchors, locally refines them toward its own policy, and finally receives velocity-level supervision on these self-generated roll-outs. Extensive experiments demonstrate that HPSD significantly improves T2V performance while also delivering notable TI2V gains, effectively strengthening the model's base generation ability.
Jiazi Bu, Pengyang Ling, Yujie Zhou et al.· 1 citation
Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass, is introduced, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.
Michael King, Aravindh Mahendran, M. Grimes et al.· 0 citations