The proposed Recursive Self-Correction approach raises model performance from a Political Neutrality Likert scale baseline of 2.14 to 4.56, averaged across all models, demonstrating effective inference-time mitigation of political bias in LLM-generated summaries.
Abstract
As Large Language Models (LLMs) become the mainstay for information retrieval and summarization tasks, ensuring that they are always non-partisan and invulnerable to political bias is a critical step towards safer and more trustworthy Artificial Intelligence (AI). Current model alignment paradigms, such as reinforcement learning from human feedback (RLHF), make LLMs follow overarching safety instructions. However, this instruction tuning can be exploited via adversarial prompt injection and be used to generate unsafe content. In particular, political bias has not been specifically targeted by modern alignment techniques as harmful and biased content. To address this vulnerability of LLMs, we propose mitigation strategies using Chain of Thought (CoT) prompting and Direct Preference Optimization (DPO). Using a public dataset of legislative videos, we generate summaries using LLMs, inject bias via adversarial prompting and evaluate their performance on a four axis scale designed for political summarization. In this paper, we present different methods to shield LLMs against the injection of political bias. Our results demonstrate that the proposed Recursive Self-Correction approach raises model performance from a Political Neutrality Likert scale baseline of 2.14 to 4.56, averaged across all models, demonstrating effective inference-time mitigation of political bias in LLM-generated summaries.
This work investigates the impact of cross-lingual prompt injection attacks on LLM-based relevance judgments using TREC Deep Learning collections and two open-weight models under established prompting frameworks, and demonstrates that multilingual query-based injections are highly effective in inflating relevance scores while simultaneously evading existing prompt-injection defenses.
This work proposes B-APO (Bias-Targeted Adversarial Preference Optimization), which casts debiasing as a bias-targeted min-max game: it generates hard negatives by applying small adversarial perturbations in the latent space to maximally induce language-vision-prior reliance, and then performs preference alignment to enlarge the margin between clean and adversarial responses.
Pinlong Zhao, Zike Ding, Zengshu Ye et al.· Annual Meeting of the Associ...· 0 citations
Membership inference attack (MIA) has emerged as a promising tool for auditing the training data of LLMs, supporting data privacy and copyright protection. Most existing MIA methods rely on the assumption that LLMs assign higher confidence scores to training samples than to non-training ones. However, since LLMs generate text by sampling high-confidence tokens, they naturally produce AI-generated texts (AIGTs) that also satisfy this assumption. In this work, we empirically confirm that such AIGTs, regardless of whether they are generated by the target LLM, can lead existing MIAs to assign even higher membership likelihoods than those of true training samples, thereby significantly un-dermining their reliability. To address this challenge, we propose a robust membership inference framework for reliably identifying training data. Our method adopts a mixture-of-experts formulation to jointly model interactions across complementary features derived from multiple MIA methods and AIGT detectors, which can remain robust against ad-versarially generated samples. Furthermore, by leveraging expert components, our method provides explainable insights into the characteristics of member data. Experiments on various datasets and LLMs show that adversarial samples substantially degrade the performance of baselines, whereas our method preserves performance close to that of the unattacked setting. Codes and datasets are released at https://github.com/kong-hyh/MoMIA .
Yuanhong Huang, Huili Wang, Xueying Bai et al.· Annual Meeting of the Associ...· 0 citations
This study investigates response-only adversarial classification of LLM outputs through a comparative evaluation of lexical, contrastive, and contextual representations and provides a realistic benchmark for response-only adversarial detection and clarify the limits of contrastive representations when prompt information is absent.
A. Nasution, Ahmet Emre Ergün, Aytu˘g Onan et al.· IEEE Access· 0 citations
Large Language Models (LLMs) often produce outputs that reflect social biases, toxicity, or unfair treatment of demographic groups, undermining trust and fairness. While prior mitigation strategies frequently rely on complex architectures, access to model internals, or costly fine-tuning, we argue that simplicity can be a strength. We introduce StarDTox, a lightweight, critique-and-revise multi-agent framework that leverages the LLM's own internal knowledge, via a small number of coordinated prompts, to self-correct harmful outputs. Dedicated agents independently assess bias and overall output quality, and their feedback is integrated to guide prompt-based revision. Without modifying model weights or requiring any extra finetuning, StarDTox offers strong bias mitigation and high-quality outputs across both open-ended text generation and structured tasks, outperforming other baselines. For the text generation task, on the RealToxicityPrompt dataset, it reduces toxicity by over 50% compared to other baselines, while maintaining over 90% fluency. In addition, in structured tasks, on the BBQ benchmark, it achieves the lowest bias scores across both ambiguous and disambiguated examples, without sacrificing accuracy.
Shirin Tahmasebi, Narjes Nikzad, A. H. Payberah et al.· Annual International Compute...· 0 citations
With the rapid deployment of neural machine translation (NMT) and large language models (LLMs), AI-assisted translation has become a cornerstone of multilingual communication. Despite achieving impressive fluency, these systems often perpetuate subtle yet systematic cultural biases embedded in training corpora, model architectures, and inference pipelines. This paper presents a systematic review of cultural bias in AI translation, organized around three research questions: (1) how cultural bias manifests, (2) how it can be identified, and (3) how it can be mitigated. Drawing on recent advances in machine translation, multilingual NLP, and AI fairness, this study analyzes manifestations across gendered stereotyping, religious oversimplification, regional framing, and cultural normalization; and then synthesizes detection methods, including benchmark-based evaluation, contrastive probing, embedding association tests, and human-in-the-loop assessment. For mitigation, this paper proposes a five-layer framework spanning data auditing, model adaptation, inference-time intervention, post-editing, and governance. To validate the framework, we conduct five proof-of-concept experiments: cross-lingual gender bias detection with statistical testing, systematic cultural fidelity evaluation under prompt engineering, contrastive sentiment analysis under high-/low-risk contexts, word embedding association tests (WEAT) with permutation-based significance, and an integrated audit pipeline with automated mitigation. Results demonstrate significant gender bias (χ²=29.99, p<0.001), a pervasive "male-as-default" phenomenon, significant gains from culture-aware prompting (p=0.03), and robust embedding-space bias (permutation test p=0.0001). The audit pipeline successfully integrates detection and mitigation into an actionable workflow. We conclude by outlining future directions for low-resource languages, intersectional bias, and production-level deployment.
Dongyao Zhang· Applied and Computational En...· 0 citations
