A lightweight gate is trained that predicts high-NLL tokens from early-layer hidden states, enabling token selection during mid-prefill via early exit, motivated by the hypothesis that less predictable tokens are less recoverable from surrounding context and therefore benefit more from late-position repetition.
Abstract
While decoder-only LLMs excel at a vast array of natural language tasks, it suffers from an asymmetric information flow induced by causal attention: later tokens are richer in contextual grounding than earlier ones. A simple and effective remedy is prompt repetition -- just appending a second copy of prompt before generation can redistribute grounding across positions and improve reasoning performance. However, full repetition of the original prompt doubles the KV cache footprint and quadruples attention cost during prefill, making it impractical for long-context settings. We propose PartRep, a selective augmentation method that appends only the most informative tokens -- rather than the entire prompt. We use token-wise negative log-likelihood (NLL) as a selection signal, motivated by the hypothesis that less predictable tokens are less recoverable from surrounding context and therefore benefit more from late-position repetition. To avoid the heavy cost of a full forward pass for scoring, we train a lightweight gate that predicts high-NLL tokens from early-layer hidden states, enabling token selection during mid-prefill via early exit. Across eight benchmarks (including MMLU, GSM8K, and RULER) and three model families (Qwen2.5, Llama3.2, Gemma4), PartRep retains most of the gains of full repetition while using only 59.4\% of its KV cache and 79.0\% of its prefill FLOPs.
Large language models (LLMs) generate text by auto-regressively sampling the next token. This inherently leads to a many-to-many mapping between prompts and responses, complicating the task of inferring prompts from observed outputs. Prior work on LLM inversion frames prompt recovery as a semantic reconstruction task. They rely on fine-tuning pretrained sequence-to-sequence models on large external datasets--and requiring access to model weights or logits--to generate semantically plausible prompts. In contrast, we present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids. We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself. Analogous to forward next-token prediction, our inverse model is trained using previous-token prediction, establishing a generative link between the forward and inverse processes that enables faithful prompt reconstruction. Moreover, it naturally supports diverse prompt reconstructions through sampling, whereby all such prompts induce similar responses under the forward, target LLM. Our approach generalises across datasets and exhibits transferability in reconstructing prompts from responses generated by different LLMs. Further, across the set of token based evaluation metrics for prompt and response reconstructions, our approach outperforms prior work.
Pirzada Suhail, Nagasai Saketh Naidu, Atanu R. Sinha et al.· 0 citations
The results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.
Xuanru Zhou, Yiwen Shao, Jiahong Li et al.· 1 citation
Training minimal-capacity linear probes on frozen hidden states of three open-weight 7-8B models across seven completion-style datasets finds three converging pieces of evidence that LLMs maintain a plan-like internal representation of output length, interpreted as evidence that LLMs maintain a plan-like internal representation of output length.
Mohamed Amine Merzouk, Dmitri Carpov, Mirko Bronzi et al.· 1 citation
This work presents Consistency Forcing (CForce) for dLLMs, a distillation method to force the mask predictions of early stages to align with those of later stages, thereby improving training-inference alignment.
Yujie Ren, Chenkai Xu, Zhuocheng Gong et al.· 0 citations
These results show that long-context memory can be organized along the layer axis, not only the token axis, and expose both the benefits of bounded retrieval and its in-window compression tax.
Han-Lin Liu, Xuan Qi, Chunyu Liu et al.· 0 citations
Vision-language models (VLMs), such as contrastive language-image pre-training (CLIP), exhibit powerful zero-shot generalization capabilities. Parameter-efficient fine-tuning (PEFT) techniques, notably prompt learning, have been extensively explored to adapt these models to downstream tasks. However, their efficacy remains constrained when transferred to specialized domains like remote sensing. We argue that the bottleneck stems not merely from the limited parameters of prompts, but essentially from the disruption of the input’s original image–text features and the lack of deep cross-modal alignment. In particular, existing methods typically rely on global attention or coarse-grained feature mapping. This inadvertently corrupts the original input representations, thereby impairing the model’s inherent generalization. Furthermore, their isolated unimodal gradient updates fail to bridge the semantic gap inherent in complex remote sensing scenes. To address these challenges, we propose tokenwise prompt-free learning (Tiper), shifting the optimization paradigm from introducing external prompts to precisely recalibrating the critical tokens that govern classification outputs. In particular, Tiper employs a hierarchical learner to supersede global prompts. Crucially, this learner intervenes exclusively on the specific core tokens (i.e., the CLS token in the visual branch and the EOT token in the textual branch), leaving other original input representations unperturbed. This fine-grained strategy effectively balances domain adaptation with the preservation of inherent generalization. Finally, we design the learner as a cross-modal coupled bridge with shared weights, enabling it to synchronously receive gradient feedback from both modalities and fostering profound multimodal collaboration. Extensive experiments validate our method on eight public remote sensing datasets covering diverse scenes and resolutions. In the base-to-new generalization task, Tiper outperforms the strong baseline MaPLe with a significant 3.7% improvement in the harmonic mean (HM). Notably, without relying on any external large-scale domain models, Tiper surpasses the latest domain-specific prompt learning methods (e.g., domain-controlled prompt learning (DCPL), domain prompt learning with quaternion networks (DPLQ)), demonstrating its superior adaptability for remote sensing image scene classification.
Tengfei Gong, Jun-Lin Wu, Yaxioong Chen et al.· IEEE Transactions on Geoscie...· 0 citations