A trajectory-guided procedure that identifies candidate injection points and inserts reproducible visual payloads into the observation stream is proposed, which is used to construct MMA-SafetyBench, which contains 1,083 adversarial trajectories across web automation, desktop GUI control, mobile navigation, document understanding, and video processing.
Abstract
Multimodal agents increasingly rely on visual interfaces to perceive state and select actions, creating an attack surface not captured by text-only safety evaluations. We study Visual Semantic Hijacking, in which deceptive but semantically meaningful interface elements, such as forged system alerts, cause an agent to depart from its authorized task. We propose a trajectory-guided procedure that identifies candidate injection points and inserts reproducible visual payloads into the observation stream. Using this procedure, we construct MMA-SafetyBench, which contains 1,083 adversarial trajectories across web automation, desktop GUI control, mobile navigation, document understanding, and video processing. Across nine evaluated models, Attack Success Rate (ASR) reaches 87.96%, while Semantic Compromise Rate (SCR) reaches 98.09%. A 1--3--5 scoring scheme distinguishes target-matching outputs, semantic disruption without a target match, and successful preservation of the benign task. Controls covering all nine models and five domains show that attacks persist under neutral prompts. A safety-priority prompt lowers ASR without eliminating the failure mode.
Results show that skill-bundled images can induce unauthorized actions even as agents complete legitimate tasks, so task success alone does not establish safe skill use.
Ling-Qi Jiang, Jia-Luo Chen, Jia-Nan Ma et al.· 0 citations
This paper introduces AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types, and introduces learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to...
Chen-Xu Wang, Xiaoyu Wen, Han Qi et al.· 0 citations
Closed-loop safety evaluation requires traffic agents that expose planner weaknesses without producing arbitrary behavior. Existing adversarial tests often couple behavior generation, risk search, and failure counting, limiting control over how interactions are created and whether repeated events constitute new evidenc...
J. Lai, Meng Wang, Zhen Zhang et al.· Systems· 0 citations
Agents are increasingly susceptible to hijack attacks, where malicious instructions embedded in cross-modal inputs (e.g., adversarial images or retrieved web content) are used to deceive the agent's execution. Existing defenses typically rely on strict filtering or refusal mechanisms, which suffer from a critical limit...
Che Wang, Fu-Yao Zhang, Jiaming Zhang et al.· Proceedings of the 34th ACM...· 0 citations
The results demonstrate that StressSim exposes robustness gaps missed by standard and hand-designed user simulators while maintaining conversation realism, achieved through lightweight prompt optimisation alone, without model fine-tuning.
A. Sharma, Brian Formento, A. Lomuscio· 0 citations
DIBench provides a unified, reproducible benchmark to quantify the risk of in-task goal deviation in mobile agents and enable comparable evaluations of safety defenses, as well as providing a decision integrity benchmark for measuring this risk in mobile agents.
Li Hu, Kang-Hua Mo, Ying-Zi Jin et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.