On-policy distillation (OPD) samples trajectories from the current student policy and minimizes token-level divergence between student and teacher next-token distributions at prefixes along those trajectories. This aligns the distillation states with the student's own generation distribution. However, it still assumes that the complete teacher distribution is an appropriate target across student capacities. In vision--language reasoning, teacher corrections can depend on visual distinctions that a compact student cannot represent. Our target-scaling study shows that, as the target approaches the complete teacher distribution, the student realizes less of the prescribed shift and obtains worse downstream performance. We therefore propose \emph{Fisher-Projected On-Policy Distillation} (FP-OPD), which distills only locally realizable teacher corrections. FP-OPD uses continuous visual perturbations to estimate the student's local visual tangent space and projects the centered teacher--student log-probability gap onto this space under the student's Fisher metric. The resulting capacity-aware target is optimized with full-vocabulary reverse KL on student trajectories, retaining the standard OPD framework. In 8B-to-2B distillation, FP-OPD improves all seven evaluated multimodal benchmarks. It raises the average score by 2.77 points over the pretrained student and by 1.60 points over standard OPD. These results demonstrate that locally realizable teacher corrections provide a more effective target for distilling compact vision--language models.
Leyan Xue, Feng Xiong, Ming-Jun Ma et al.· 0 citations
Tool-level attacks on Large Language Model (LLM) agents—poisoned tool descriptions, prompt injection, and capability misrepresentation—are universally effective, yet no existing defense provides comprehensive protection. We propose Architectural Intent Collapse (AIC), a formal framework capturing the systematic loss of communicative intent when text from heterogeneous sources is flattened into a single context window. Grounded as a novel instantiation of the Confused Deputy Problem, AIC reveals that the missing boundary is not permission but intent: the architecture cannot distinguish descriptive statements from prescriptive commands. We formalize AIC via an architectural collapse operator, introduce Intent Separation Degree (ISD) as a measurable metric, and develop a mechanism-based taxonomy of five intent-disguise attack types, including two previously undescribed (Conditional Latency and Inference Inducement). Experiments across 25 framework–model combinations (employing GPT-4o, Claude-4-Sonnet, Gemini-2.5-Pro, DeepSeek-V3, and Qwen3-32B as LLM backends) confirm that ISD degrades with description verbosity, strongly predicts defense effectiveness (r=−0.97), and is uniformly low across all current frameworks. Three root-cause defense principles are derived; one retains substantial protection against adaptive attackers. This research is useful for agent framework designers, security practitioners, and researchers seeking a principled understanding of why tool-level attacks succeed and how architectural defenses can address their root cause.
Zhaowen Feng, Zhenhui Liu, Mingjun Ma et al.· Electronics· 0 citations