Skip to content
Preprint

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

Aug 2026 · 0 citations · 55 references
Computer Science

TL;DR

A training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible.

Abstract

Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (\textbf{C}ontext-\textbf{A}ware \textbf{C}luster \textbf{D}ecoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.

View source

Similar papers

#natural language process... Preprint Aug 2026

Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs

IGFD is proposed, a training-free decoding strategy that ranks candidates using token confidence, neighborhood uncertainty, and structural commitment risk and consistently outperforms existing decoding strategies across the majority of benchmarks and diffusion MLLM backbones under identical decoding budgets.

Xing-You Fang, Jin Zhong, Xiaosong Yuan et al. · 0 citations
Preprint Aug 2026

Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference

DARD is proposed, a training-free framework that separates tokens into masked, candidate, and unmasked states and adaptively regulates their influence on subsequent decoding, and consistently improves the speed-quality Pareto frontier over recent revocable decoding methods.

Woo-Soon Park, Insu Lee, Minyoung Noh et al. · 1 citation
Open access Sep 2026

Short-Text Clustering Enhancement Based on Semantic-Aware Iterative Refinement

Short-text clustering (STC) remains a critical challenge in natural language processing because of the inherent sparsity, noise, and semantic ambiguity in brief textual data. Traditional methods such as TF-IDF and static embeddings are hindered by polysemy or noise challenges. This study introduces Semantic-Aware Itera...

Ali Sabah · 0 citations

Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding

This work introduces Elastic Threshold Attention, an end-to-end trainable architecture that rivals dense model quality under hardware-aligned block-sparse decoding, and introduces an offline calibration algorithm for domain-specific deployments that freezes per-head constant thresholds, cutting attention compute by an...

Themistoklis Haris, Henry Li, Maryam Karimzadehgan · 1 citation · ⚡1

VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMs

VideoMM is introduced, which marks a paradigm shift from model-centric downsizing to adaptive perceptual granularity and accelerates inference by 2.73 times over current leading methods, establishing a highly scalable paradigm for long-video understanding.

Hao-Yu Guo, Yuan Feng, Jun-Lin Lv et al. · 1 citation
#machine learning Preprint Sep 2026

Guiding the coarse levels of semantic IDs makes the fine levels learnable

Generative retrieval represents each item by a short Semantic ID and casts recommendation as autoregressive generation of that sequence. Because the tokenizer is trained independently to reconstruct an item embedding, its codes are aligned with neither the downstream LLM nor the end task. Nearly every SID system theref...

Bin Wang, Zheng-Yu Zhang · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.