Sep 2026· Conference on Empirical Methods in Natural Language Processing· Vol abs/2609.29001· 0 citations· 37 references
Computer Science
TL;DR
This work evaluates LLM politeness judgments using two English-language datasets with complementary annotation formats, finding that inter-model agreement is stronger than model--human agreement.
Abstract
Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.
Evaluating whether large language models (LLMs) reason about morality in human-like ways requires more than measuring whether they produce the right outputs in isolated cases. Existing approaches – including scalar agreement, distributional analysis, rationale classification, and consistency testing – compare model and...
Large Language Models (LLMs) are entering democratic contexts as instruments of governance, where the challenges at hand are ill-structured, marked by ambiguity and contestation. Ill-structured democratic problems demand more than factual precision; they call for intersubjective reasoning: context-sensitive judgments t...
F. Veri, Gustavo Kreia Umbelino· Proceedings of the National...· 0 citations
This study empirically confirms the presence of LLM values, accurately quantifies their shifts, and achieves more efficient and precise steering than conventional blind training, all without degrading general capabilities.
Kelvin Zhang, Jing-Yu-Gin Chen, Yu-Fan Liu et al.· 0 citations
A systematic comparison using a naturally occurring online persuasion corpus in which original posters explicitly verify whether a reply changed their view is conducted, highlighting the risk of treating LLM judgments as faithful proxies for human belief updating and point to structural differences in how LLMs and huma...
Large language models (LLMs) are used to simulate human participants in psychological research. We asked whether LLMs that reproduce human evaluations of a whistleblower's moral character also reproduce the motive attributions that accompany them. Five LLMs and two human samples (N = 125 and N = 742) evaluated a physic...