Multi-dimensional Vulnerability Analysis of LM Safety Guards via Role-Goal-Format Decomposition
Abstract
Current safety guard models for Large Language Models (LLMs) often fail to block complex Jailbreaking attacks because they focus on the structure of a prompt rather than its true intent. To evaluate this problem, we introduce the Role-Goal-Format (RGF) framework, which analyzes how the combination of a user’s persona, their objective, and the text format can bypass safety filters. We compared two small safety guard models: Prompt Guard 2 and Llama Guard 3. The results show that even with the same overall accuracy, they have completely opposite weaknesses. Prompt Guard 2 is easily bypassed by unusual text formats. On the other hand, Llama Guard 3 lacks semantic understanding, missing 76.7% of virus-related attacks. Furthermore, both models exhibit a common vulnerability to contextual breaking; they fail to activate safety alignment when harmful objectives are framed within authoritative and benign personas, such as Teacher, Engineer, and Investigator. These findings highlight that small safety guard models have clear limitations and require multi-layered defenses that understand both text structure and actual intent.