Skip to content

Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling

Aug 2026 · 0 citations · 24 references
Computer Science

TL;DR

ITS is proposed, a query-efficient framework that uses an attention-derived pairwise dependency proxy to approximate the CMI-aware selection criterion and incorporates a diversity-promoting strategy to improve token coverage across sampling rounds, yielding aggregated token-wise signals that are less affected by dependency-induced approximation error.

Abstract

Diffusion large language models (dLLMs) offer a compelling alternative to autoregressive models, yet they may expose sensitive training data during denoising. Detecting such usage is challenging because dLLMs lack the efficient one-pass probability decomposition of causal architectures. Existing methods rely on random masking to obtain tractable token-wise detection signals under limited query budgets, but fail to control dependencies among masked tokens. We demonstrate that this token-wise approximation introduces a non-negative structural estimation error, which is theoretically characterized by the cumulative conditional mutual information (CMI) among masked tokens and can obscure subtle memorization signals. This insight suggests that reliable detection requires masked token sets with weak internal dependency. To avoid the prohibitive cost of directly estimating CMI over token combinations, we propose \textit{Independent Token Sampling} (ITS), a query-efficient framework that uses an attention-derived pairwise dependency proxy to approximate the CMI-aware selection criterion. ITS further incorporates a diversity-promoting strategy to improve token coverage across sampling rounds, yielding aggregated token-wise signals that are less affected by dependency-induced approximation error. Experiments on multiple datasets show that ITS consistently outperforms state-of-the-art baselines across different models and datasets, achieving an AUC improvement of 0.18 on the ArXiv dataset while maintaining strong performance under limited query budgets. The code is available at https://github.com/Chrisqcwx/DLLM-MIA .

View source

Similar papers

#natural language process... Open access Sep 2026

Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models

This work identifies a systematic upstream/downstream structure in dLLM rollouts and proposes Informed Masking (IM), which derives a per-token priority score from the denoising trajectory at zero extra inference cost and biases mask sampling toward downstream tokens.

Xiao-Yi Yu, E. Sangineto, Pei Fu et al. · 0 citations

Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models

Masked diffusion language models (dLLMs) have shown strong potential for faster inference through parallel token generation when combined with confidence-based samplers. However, recent work has shown that such methods can defer unmasking high-entropy fork positions at which multiple plausible continuations exist. This...

S. Frkovic, Metod Jazbec, C. Naesseth · 0 citations
Preprint Aug 2026

DIME: Query-Efficient Framework for Membership Inference on Diffusion Models

DIME (Denoiser Ideal Membership Error), a theoretically grounded and query-efficient framework for membership inference on diffusion models, is introduced, which reveals that membership leakage is governed by the denoiser's implicit reconstruction error.

Tue Do, Daniel Alabi · 0 citations
#natural language process... Preprint Sep 2026

Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry

This work identifies a phenomenon termed token-level memorization asymmetry through theoretical analysis of diffusion training dynamics and proposes Q-Skew, a quantile-weighted skewness-based indicator for membership inference on finetuned DLMs.

Sheng-Fang Zhai, Leo Marchyok, Yu-Ling Shi et al. · 1 citation · ⚡1
Preprint Aug 2026

Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference

DARD is proposed, a training-free framework that separates tokens into masked, candidate, and unmasked states and adaptively regulates their influence on subsequent decoding, and consistently improves the speed-quality Pareto frontier over recent revocable decoding methods.

Woo-Soon Park, Insu Lee, Minyoung Noh et al. · 1 citation
#machine learning Preprint Sep 2026

Denoising Surface: Modeling and Predicting Inference Cost for Diffusion LLM Serving

As diffusion large language models (dLLMs) become more capable, they are moving from research settings to real-world \textit{serving}, where request management (such as scheduling and resource allocation) relies on accurate estimation of per-request inference cost. However, common cost proxies fall short for dLLMs: out...

Hao-Yu Zheng, Fang-Cheng Fu, Bin-Hang Yuan et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.