Jul 2026· Practice and Experience in Advanced Research Computing· 0 citations· 14 references
Computer Science
TL;DR
This work presents a two-phase evaluation of ten Llama variants using the OWASP Top 10 for LLM Applications, and applies nine encoding obfuscations to the same prompts, which fully bypasses all text-only models.
Abstract
As large language models (LLMs) are deployed in security-critical environments, evaluating their robustness against adversarial threats is essential. We present a two-phase evaluation of ten Llama variants using the OWASP Top 10 for LLM Applications. In Phase I we benchmark 100 OWASP-based adversarial prompts; Llama-Guard-3-1B leads (76% detection, 0.165 s latency, 0.94 GB VRAM), while base models detect nothing. In Phase II we apply nine encoding obfuscations to the same prompts (9,000 inferences): Base64 stays detectable (up to 100%), hexadecimal collapses to 2%, and ASCII art fully bypasses all text-only models. Model scale alone does not ensure robustness; targeted safety training and encoding-aware defenses are required. Both benchmark datasets are released for reproducibility [3].
Such failures are called Representation-Confusion Attacks in Reverse Engineering (RARE): the pipeline promotes a correctly extracted observation to instruction authority, claim-validating evidence, or trusted analysis state without the authority or support that role requires.
ExE-Bench assesses performance, temporal and adversarial robustness, and computational overhead, aggregating them into a single score for direct and fair model comparison, and highlights how evaluations conducted only after deployment are suboptimal and unable to provide a complete picture of their performance.
Andrea Ponte, Daniel Gibert, M. Kozák et al.· 0 citations
This paper presents a framework for evaluating prompt injection attacks against LLM-based log interpretation using log traces generated during real cyber attacks, and creates adversarial examples through generic injection generation, refinement, and attack-specific optimization.
Max Landauer, Florian Skopik, Markus Wurzenberger et al.· 0 citations
The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.
Berkay Özçam, İrem Önen, E. I. Tatli et al.· Signal Processing and Commun...· 0 citations
This study proposes an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another, and introduces a model capable of generating attack inputs in the Turkish language.
Berkay Ozcam, Irem Onen, M. Amasyalı et al.· 0 citations
The Security Mutation Score (SMS) is proposed, a metric that classifies mutant kills into semantic, functional, incidental, and crash categories using operator-aware heuristics, distinguishing genuine security awareness from coincidental detection.
