A mechanistic intervention that reweighs attention away from the patch tokens in the identified lexical heads at test-time is performed and a test-time adaptive attention head routing is proposed to further reduce the impact on performance at matched robustness, pushing beyond the Pareto frontier of current robustness-...
Bo-Han Liu, Wenqian Ye, Guang-Zhi Xiong et al.· 0 citations
In-context learning (ICL) enables language models to perform new tasks from demonstrations without weight updates. However, every ICL inference requires processing the full set of examples, resulting in inefficient deployments, and how ICL works mechanistically is not fully understood. Prior work compresses ICL into fi...
Guang-Zhi Xiong, Zheng-Hao He, Bo-Han Liu et al.· 0 citations
Masked diffusion language models (MDLMs) can generate text efficiently by predicting multiple masked tokens in parallel, but predictions from the same forward pass are not necessarily reliable when committed together. We study when parallel commitment is reliable. Our diagnostics show that confidence alone does not det...
This work proposes to view reasoning paths as phase-structured trajectories within fixed questions as PAIR, short for Phase-Aligned Intra-question Reasoning, and improves within-question trajectory ranking and Best-of-N trajectory selection across models and benchmarks.
It is suggested that CoT prompting activates specific latent features to trigger reasoning, and that targeted intervention on these features offers an alternative pathway to elicit efficient reasoning behavior without explicit CoT prompting.
Zhenghao He, Guangzhi Xiong, Bohan Liu et al.· 6 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.