Personal agents increasingly connect to services across the web and applications by using tools. As the effects of their actions accumulate and interact across services, unintended or harmful outcomes can arise when agents fulfill benign user requests. Proactively identifying these compositional failures requires syste...
Yeonsung Jung, Zhi-Yuan Wang, Zi-Fan Wang et al.· 0 citations
Prompt injection is widely recognized as a major security threat to AI agents that interact with untrusted external data, such as websites, documents, and emails. Prior work has shown that, in the text domain, black-box prompt injection can achieve near-perfect attack success rates (ASRs). In the image domain, however,...
Si-Zhe Chen, Yu-Lin Tsai, Ivan Evtimov et al.· 1 citation
RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), a principled reinforcement learning (RL) method for post-training that explicitly teaches models to override flawed reasoning trajectories and reroute to safe and helpful responses, substantially improves safety and jailbreak robustness, reduces overrefusa...
Sheng-Hsuan Peng, E. Smith, Ivan Evtimov et al.· arXiv.org· 10 citations· ⚡2
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.