This work examines emoji-augmented prompts as a test case for robustness, evaluating 50 prompts across four open-source LLMs, showing substantial variation in robustness.
Abstract
Safety evaluations of large language models (LLMs) predominantly rely on text-based adversarial prompts, potentially overlooking vulnerabilities arising from alternative input representations. This work examines emoji-augmented prompts as a test case for this gap, evaluating 50 prompts across four open-source LLMs (Mistral 7B, Qwen 2 7B, Gemma 2 9B, Llama 3 8B). Results show substantial variation in robustness: Gemma 2 9B and Mistral 7B exhibit non-zero success rates (10%), Llama 3 8B 6%, while Qwen 2 7B shows complete resistance (0% success rate). A chi-square test ($\chi^2 = 32.94, p<0.001$) confirms significant differences in outcome distributions. These findings indicate that robustness is sensitive to input representation, and that evaluations restricted to standard text prompts may underrepresent model vulnerabilities.
TanglishGuard is introduced, a novel benchmark designed to systematically assess the robustness of LLM safety mechanisms against code-mixed inputs and contributes to the development of more robust and equitable AI systems by ensuring safety mechanisms are tested against authentic communication patterns rather than sanitised English-only datasets.
Vivina Vijesh, Dr. E. Gothai, M. Student· International Journal of Inn...· 0 citations
Large language models (LLMs) are increasingly deployed in multilingual settings, yet their safety behavior under Turkish harmful prompts and prompt injection attempts remains insufficiently characterized. This study evaluates the adversarial robustness of 55 open- and closed-source LLMs under paired Turkish and English harmful prompt conditions. We constructed a benchmark of 790 Turkish adversarial prompts, translated the prompts into English for cross-lingual comparison, and applied both prompt sets to the model pool. Model responses were labeled as harmful, harmless, or hallucinatory, and safety was analyzed using safety scores, Turkish–English ranking differences, and inter-rater reliability based on Fleiss’ kappa. The results reveal substantial variation across models. Closed-source systems generally achieved higher safety scores and stronger filtering behavior, whereas open-source and Turkish-oriented models showed a wider performance distribution. GPT-5.4 ranked first in the Turkish tests with a 99.37% safety score but decreased to 96.71% in the English tests, while Qwen3.5:27B ranked first in English with 97.47%. These differences suggest that safety mechanisms are not fully language-invariant. Hallucination also emerged as a distinct safety risk, particularly in Turkish evaluations. The findings indicate that Turkish LLM safety cannot be inferred from general model capability alone and should be assessed through language-specific, culturally aware, and continuously updated adversarial benchmarks.
Öner Aytaş, Tuğçe Şen, Banu Diri et al.· Applied Sciences· 1 citation
This work presents the most comprehensive evaluation of LLM safety capabilities to date, systematically testing models across datasets that are organized into four distinct categories, and uncovers critical blind spots.
Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet how different factors shape their jailbreak vulnerabilities remains poorly understood. Existing benchmarks often couple harmful intent, prompt framing, visual semantics, and instruction carrier within individual jailbreak instances, obscuring the specific sources of observed vulnerabilities. To address this limitation, we introduce MMJailBench, a factorized benchmark that systematically varies and combines these factors under controlled configurations, enabling fine-grained comparison and factor-level attribution. Large-scale evaluations across 16 open-weight and proprietary MLLMs reveal highly heterogeneous and model-dependent vulnerability profiles. Jailbreak vulnerability varies markedly across harm domains, exposing uneven coverage in current multimodal safety alignment. Prompt framing emerges as the dominant source of variation, task-relevant visual semantics systematically increase jailbreak susceptibility with authority-like cues exposing particularly pronounced vulnerabilities, and visually rendered instructions do not consistently increase jailbreak susceptibility relative to direct textual instructions. To further investigate the risks introduced by multimodal context, we conduct diagnostic analyses on a representative open-weight model and identify vulnerability-associated patterns in internal representations and cross-modal interactions. Finally, we develop a modular multimodal jailbreak evaluation suite with full and lightweight configurations, multiple judge options, and multidimensional metrics, enabling reproducible, scalable, and cost-efficient multimodal jailbreak auditing.
Tianshi Wang, Jing-Song Wang, Ya-Fei Huang et al.· 0 citations
This work proposes an explanation-aware safety framework that augments binary harmfulness detection with structured, human-interpretable explanations capturing severity, strategies, trigger spans, ratio-nales, and derived safety factors, and introduces a human–LLM hybrid annotation and canonicaliza-tion pipeline.
Sunghee Dong, Sungwon Yi, Kangmin Bae et al.· 0 citations
Despite extensive safety alignment, large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safeguards to elicit harmful content. While prior work attributes this vulnerability to safety training limitations, the internal mechanisms by which LLMs process adversarial prompts remain poorly understood. We present a mechanistic analysis of the jailbreaking behavior in a large-scale, safety-aligned LLM, focusing on LLaMA-2-7B-chat-hf. Leveraging edge attribution patching and subnetwork probing, we systematically identify computational circuits responsible for generating affirmative responses to jailbreak prompts. Ablating these circuits during the first token prediction can reduce attack success rates by up to 80\%, demonstrating its critical role in safety bypass. Our analysis uncovers key attention heads and MLP pathways that mediate adversarial prompt exploitation, revealing how important tokens propagate through these components to override safety constraints. These findings advance the understanding of adversarial vulnerabilities in aligned LLMs and pave the way for targeted, interpretable defense mechanisms based on mechanistic interpretability.
Paria Mehrbod, Boris Knyazev, Guy Wolf et al.· 0 citations
