Aug 2026· ACM Transactions on Intelligent Systems and Technology· 0 citations· 69 references
TL;DR
This work investigates whether large language models (LLMs) can automatically adapt template-based bias datasets to specific domains using zero-shot prompting and shows that domain-adapted templates capture real-world bias patterns more faithfully than standard templates.
Abstract
As natural language processing (NLP) systems increasingly operate on heterogeneous data, faithful social bias evaluation requires benchmarks that are both domain-sensitive and scalable. However, existing resources still face a core trade-off: template-based benchmarks scale easily but often lack linguistic authenticity, whereas naturally occurring examples (NOEs) reflect real usage but are expensive to curate and often provide limited coverage across domains and social groups. To bridge this gap, we investigate whether large language models (LLMs) can automatically adapt template-based bias datasets to specific domains using zero-shot prompting. We evaluate the effectiveness of LLMs along three dimensions: bias consistency, which measures agreement between bias estimates derived from adapted templates and NOEs; content preservation, which assesses whether the original semantic intent is maintained; and domain adherence, which evaluates how well the generated text reflects the linguistic characteristics of the target domain. Across the Equity Evaluation Corpus (EEC) and the Identity Phrase Templates Test Set (IPTTS), and across three domains—IMDb, Twitter, and Wikipedia Talk Pages—using both automatic and human evaluations, we show that domain-adapted templates capture real-world bias patterns more faithfully than standard templates. Overall, our results highlight the importance of domain variation in fairness research and position LLM-based adaptation as a scalable and reproducible framework for robust social bias assessment in NLP.
A hybrid detection framework which combines semantically deep embeddings from the RoBERTa transformer with a set of carefully designed language statistics and linguistic statistics and shows excellent resistance to the surface-level adversarial paraphrasing strategy.
Anita Rani, Suman· International Journal of Sci...· 0 citations
Language models (LMs) often acquire various biases during pre-training and may express them in interactions, potentially causing social harm. Existing methods often rely on counterfactual augmentation or representation projection. These strategies remain limited in practice due to their high computational costs and difficulty in scaling to larger models. Additionally, many of these strategies require manual data annotation, narrowing their scope to specific cultures and bias categories. To overcome these limitations, we propose HEIMAT, a HEurIstic-style autoMATic debiasing framework for LMs. HEIMAT consists of two main steps: bias disclosure and debiasing fine-tuning. In the first step, it uses simple templates to construct heuristic prompts, which are applied to reveal model biases and generate corresponding context prompts. In the second step, it fine-tunes the model by minimizing the Jensen-Shannon divergence of predictions on these context prompts to reduce bias. Extensive experiments show that HEIMAT effectively mitigates bias in different cultures while maintaining the model's natural language understanding (NLU) performance.
Tian Lan, Yemin Wang, Chuancheng Shi et al.· 0 citations
Societal stereotypes are often reflected in, and can be reinforced by, machine learning models and linguistic resources such as word embeddings. While various benchmarks and bias detection methods have been proposed, most focus exclusively on English. When applied to other languages, these approaches typically rely on direct translations of English resources, overlooking language- and culture-specific nuances. In this paper, we introduce BIAS-WEAT and BIAS-SEAT, two novel metrics designed to detect biases in word embeddings and language models for Dutch, German, Icelandic, Italian, Norwegian and Turkish. Drawing on real-world biases identified through co-creation workshops with native speakers in the context of a hiring situation, we translated these insights into technical evaluation metrics that are applicable to general-purpose language resources. Our interdisciplinary study demonstrates how language models embed and reproduce biases that are specific to their linguistic and geographic contexts, underscoring the need for culturally grounded approaches to bias detection.1
Mascha Kurpicz-Briki, Catherine Ikae, A. Puttick et al.· Frontiers in Artificial Inte...· 0 citations
A hybrid approach that combines contextual embeddings from RoBERTa with handcrafted stylometric features to capture both high-level semantic patterns and low-level stylistic cues enables robust generalization across heterogeneous text types is adopted.
Britt van Leeuwen, S. Bhulai, R. V. D. Mei· 0 citations
A fundamental disconnect is suggested between a model's capacity for factual accuracy and its ability to maintain social fairness, highlighting the need for multi-dimensional evaluation frameworks for small-scale systems.
M.J.F. Valdez, Arghir-Nicolae Moldovan· Proceedings of the 15th Inte...· 0 citations
This study deploys a scalable machine learning pipeline: combining a transformer-based classifier applied to 2.01 million English-language AI-related news headlines (July 2022–July 2024) with large-language-model and human-annotator validation (three annotators, Fleiss’ κ=0.80) on stratified subsamples, to extract six interpretable, bias-linked discourse indicators computed at the AI-domain level: evaluative orientation (valence), loss salience, narrative drift, exposure-adjusted sentiment, cross-source divergence, and novelty-phase framing. Each operationalizes an established cognitive-psychology construct as a computable property of the information environment associated with biased risk–benefit reasoning. Results show systematic variation across domains: technical and methodological areas such as deep learning and natural language processing exhibit gain-salient framing, while safety-critical topics such as deepfakes (loss-to-gain headline ratio = 3.17) and facial recognition show strongly loss-salient profiles. Cross-model validation using an LLM on a stratified sample of 1000 headlines confirms that domain-level indicator rankings are robust to classifier choice (Spearman ρ=0.83; p<0.001), establishing the rank stability of pipeline outputs independently of the specific classification architecture. As a contextual application, domain-level profiles are mapped to European Union AI governance instruments, documenting parallels between discourse patterns and regulatory risk tiers. The framework provides a scalable, reproducible methodology for monitoring evaluative conditions in technology news across domains, sources, and time.
O. Topal, Inna Novalija, Joao Pita Costa et al.· Applied Informatics· 0 citations
