EADC, a novel advanced evaluation benchmark of LLMs based on an AI compliance knowledge graph and AI compliance legal experts, is introduced, offering a rigorous, AI laws and regulations-aligned benchmark to safeguard high-level and deep compliance in the application of LLMs.
Abstract
Large Language Models (LLMs) have been used in various industries. However, ensuring their compliance with complex laws and regulatory frameworks remains a great challenge. Existing evaluation paradigms mainly rely on static benchmarks that suffer from three severe limitations: First, the compliance rules being used do not comply with the requirements of Artificial Intelligence (AI) laws and regulations; Second, they only handle apparent, explicit compliance risks, leaving implicit and covert compliance risks undetected; Third, they fail to track the systematic propagation of risks along logical dependency chains or evaluate compliance within nuanced, context-based real-world scenarios. To bridge this critical gap, we introduce EADC, a novel advanced evaluation benchmark of LLMs based on an AI compliance knowledge graph and AI compliance legal experts. By mapping abstract legal rules into structured logical multi-relational graphs, our framework enables automated, evolving agents to distill and synthesize highly sophisticated adversarial scenarios. This compliance benchmark is reviewed and corrected by human AI legal experts throughout the whole process. The resulting dataset (4,435+ QA pairs) provides an extensive, multi-dimensional taxonomy covering critical regulatory frontiers, including bias and discrimination, fairness, personal privacy protection, and values. Crucially, our compliance dataset moves beyond shallow string-matching by incorporating contextual long-horizon interactions and logic-driven hazard chains, capturing deeply embedded compliance anomalies that bypass traditional filters. Experiment evaluations demonstrate that our framework exposes critical regulatory blind spots in state-of-the-art LLMs, offering a rigorous, AI laws and regulations-aligned benchmark to safeguard high-level and deep compliance in the application of LLMs.
The safety alignment of large language models (LLMs) faces persistent challenges from jailbreak attacks. While existing methods mostly leverage prompt engineering or adversarial optimization, we identify and formalize an underexplored semantic isomorphism vulnerability where harmful and safe scenarios share highly cons...
Fan Yang, Ke Wang, Wenzhou Dou et al.· IEEE Transactions on Audio,...· 0 citations
A Domain-Specific Adversarial Prompt (DSAP) dataset of 4,240 prompts aimed at testing safety boundaries of LLMs focusing on indirect adversarial prompts in the context of realistic professional settings is presented and novel information regarding the weaknesses of contemporary language models is provided.
Gaurang Mishra, Shikha Maheshwari· Natural Resources for Human...· 0 citations
This work proposes a novel neuro-symbolic approach to static analysis, named ELSA, which comprises two key modules, namely, LLM-assisted static analysis and analyzer ensemble (the ensemble of multiple LLM-assisted static analyzers), each incorporating two distinct strategies.
Jun-Xiang Wang, Fu Song, Miao-Miao Zhang et al.· Proceedings of the ACM on So...· 0 citations
A defense taxonomy spanning three axes, namely prompt-level, inference-time, and training-time interventions, is proposed, within which 30 mitigation mechanisms published from 2024 onwards are systematically analyzed, demonstrating that no single defense mechanism provides comprehensive protection, and that robust depl...
Berkay Ozcam, Mustafa Kara, Muhammet Ali Aydin et al.· Electronics· 0 citations
This work proposes an explanation-aware safety framework that augments binary harmfulness detection with structured, human-interpretable explanations capturing severity, strategies, trigger spans, ratio-nales, and derived safety factors, and introduces a human–LLM hybrid annotation and canonicaliza-tion pipeline.
Sunghee Dong, Sungwon Yi, K. Bae et al.· Proceedings of the Thirty-Fi...· 0 citations
EvoSafeHarness is a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules.
Nanxi Li, Ying-Zi Ma, Yu-Long Cao et al.· 4 citations· ⚡1