Skip to content
Preprint

Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models

Oct 2026 · 0 citations · 24 references
Computer Science

TL;DR

How hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation are illustrated.

Abstract

Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.

View source

Similar papers

Sep 2026

Your Model Is Leaking: Covert Information Transfer through LLM Residual Streams

A residual-stream covert-channel attack is introduced that maps messages to codewords and injects them into an intermediate residual stream through a compromised runtime hook and is scaled with the local residual norm using the signal-to-residual-norm ratio.

Ming-Yuan Li, Yan-Na Jiang, Guang-Sheng Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers

This work audits reading for routing entropy in Attention-Residual variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, establishing control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.

Wenhao Liang, Lin Yue, W. Zhang et al. · 0 citations
#machine learning Preprint Oct 2026

When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

Mixture-of-Experts (MoE) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety auditing. Unlike ordinary model outputs, this telemetry reveals a view of the model's internal computation, raising a privacy question: can it reveal w...

Yi-Xin Tan, Jia-Yang Liu, Lu-Peng Sun et al. · 0 citations

Controlled Decoding Attacks on Black-Box LLMs

This work introduces \method{}, a framework for jailbreaking through text-only continuation interfaces that permit repeated sampling and assistant-prefix continuation, and achieves the highest mean score most comparisons against baselines.

Jesson Wang, Shawn Li, Wei Yang et al. · 2 citations
#machine learning Preprint Oct 2026

CascadeLedger: Counterfactual Auditing of Information and Resource Integrity in LLM Cascades

We introduce CascadeLedger, a counterfactual execution-audit framework for budgeted language-model cascades. The protocol enforces call-before-read semantics, binds successful reads to service and cache identities, records availability and component-wise budget transitions, and freezes the event trace before any target...

Miao-Bo Hu, Shu-Hao Hu, Xiao-Bo Guo et al. · 0 citations
Preprint Oct 2026

Context-Shielded Backdoor Attack against Large Vision-Language Models

Large vision-language models (LVLMs) routinely use inference-time visual-token compression to prune hundreds of patch tokens down to a compact budget. Existing LVLM backdoors fire whenever a triggered image appears, whereas prior deployment-triggered attacks activate only when model weights are quantized or pruned. How...

Zhi-Fang Zhang, Qi-Rui Huang, Chen-Peng Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.