Skip to content

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

Sep 2026 · Conference on Empirical Methods in Natural Language Processing · Vol abs/2609.29001 · 0 citations · 37 references
Computer Science

TL;DR

This work evaluates LLM politeness judgments using two English-language datasets with complementary annotation formats, finding that inter-model agreement is stronger than model--human agreement.

Abstract

Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.

Read PDF

Similar papers

#small language model Open access Sep 2026

Measuring structural value alignment in sixteen models: LLMs have human-like moral spaces

Evaluating whether large language models (LLMs) reason about morality in human-like ways requires more than measuring whether they produce the right outputs in isolated cases. Existing approaches – including scalar agreement, distributional analysis, rationale classification, and consistency testing – compare model and...

Ye-Xiang Tang · 0 citations
Open access Sep 2026

Plausible nonsense and deliberative reasoning: Benchmarking LLMs against human judgment.

Large Language Models (LLMs) are entering democratic contexts as instruments of governance, where the challenges at hand are ill-structured, marked by ambiguity and contestation. Ill-structured democratic problems demand more than factual precision; they call for intersubjective reasoning: context-sensitive judgments t...

F. Veri, Gustavo Kreia Umbelino · 0 citations
#artificial intelligence Review Sep 2026

Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models

This study empirically confirms the presence of LLM values, accurately quantifies their shifts, and achieves more efficient and precise steering than conventional blind training, all without degrading general capabilities.

Kelvin Zhang, Jing-Yu-Gin Chen, Yu-Fan Liu et al. · 0 citations
#natural language process... Preprint Aug 2026

Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

A systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view is conducted, highlighting the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and huma...

Lin Chen, Yi-Tong Chen, Yong Li · 0 citations
#artificial intelligence Preprint Sep 2026

Human-like moral judgments conceal divergent motive attributions in large language models

Large language models (LLMs) are used to simulate human participants in psychological research. We asked whether LLMs that reproduce human evaluations of a whistleblower's moral character also reproduce the motive attributions that accompany them. Five LLMs and two human samples (N = 125 and N = 742) evaluated a physic...

Xiao-Yan Wu, J. Dreher · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.