This work constructs the missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates.
Abstract
As search increasingly shifts toward LLM-driven answer engines, advertising is becoming embedded within the generated response itself and should therefore be evaluated for both user utility and commercial value. The key challenge is click-through intent: behavioural logs are unavailable, human annotation resists calibration, and frontier LLM judges conflate intent with linguistic fluency. These gaps compound, as principled pricing presupposes a continuous intent signal, while generating such a signal presupposes supervision that is currently unavailable. We construct the missing supervision through a psychologically grounded agent simulation framework, and distil it into a parameter-efficient evaluator that predicts click-through intent, together with the three companion dimensions of ad quality, as smooth, differentiable estimates. Validated through sign-certain behavioural perturbations, the evaluator surpasses frontier zero-shot judges on relevance sensitivity (79% versus 60-67%), tracks graded content degradation, generalises without error to 103 fictional products, and agrees with human preference in 86% of pairwise judgements across five annotators, with agreement rising in the evaluator's confidence. Upon its estimates we build the pricing layer directly, deriving the unique payment rule under which truthful bidding is optimal, demonstrating it on a best-of-k allocation, and extending the mechanism to non-monotone allocations. The same differentiable signal stands ready as a training objective for ad generation.
Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a novel simulation approach that combines categorical judgments with evaluator-specific auxiliary data--retrospective reasoning traces and interface telemetry--to enable LLM-based simulation of individual evaluators via in-context learning. We conduct a systematic empirical study of this approach using multi-facet data from 32 trained annotators across 4,200 preference judgments in a 4 x 4 x 4 factorial design. Our key findings: (1) The simulation approach achieves up to 9.9 percentage point improvements over the Base Judge; (2) Reasoning traces provide the largest gains with higher collection efforts, while interface telemetry often hurts rather than helps performance despite being cheaper to collect. (3) Simulation difficulty is systematic, predicted by an evaluator's neutral usage (most clearly on Helpfulness) and divergence from consensus; the neutral-usage tendency--rather than simulatability itself--is the cross-task-stable property (r = 0.728). These results establish both the potential and limits of evaluator-specific auxiliary data for personalized evaluation, offering methodological insights for scaling individual aware AI assessment.
Zeyu He, Xuan Qi, Subramanian Chidambaram et al.· SIGDIAL Conferences· 0 citations
A sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales is developed and applied, demonstrating that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pipeline overhead.
Alireza S. Ziabari, Kat Ellis, Colleen E. Chan et al.· 0 citations
Message search and retrieval on enterprise collaboration platforms is challenging in ''eye-off'' environments, where explicit human relevance labels cannot be collected and engagement signals such as clicks and dwell times are noisy and behaviorally biased. Large Language Models (LLMs) offer an alternative source of semantic supervision, but models trained solely on LLM-derived labels often regress on engagement-based metrics. We present LLM-Click Agreement Labeling, an industrial-scale supervision strategy that retains only those query--message pairs where click-based labels and LLM-generated labels agree. This selective filtering reduces supervision noise and maintains a balance between user interaction patterns and semantic relevance. In a human-annotated pilot, agreement-based labels improved accuracy by +18% relative to click-only supervision. A worldwide A/B deployment further showed that the approach preserves traditional search quality while delivering statistically significant gains in conversational grounding (e.g., +1.4% CiteDCG, +5.7% Good Citation Count). These results highlight that improving supervision quality, rather than modifying model architecture, is the most effective lever for advancing retrieval performance in large-scale enterprise systems.
Avinash Kumar, Hardi Rathod, Rohan Mallick et al.· Annual International ACM SIG...· 0 citations
Side-by-side comparisons that elicit human preferences are commonly used to assess the quality of large language model (LLM) output and have been applied to retrieval-augmented generation (RAG) systems. However, when applied to complex, information-seeking queries, these methods are limited by their inability to provide explanatory or diagnostic insights. As an alternative, nugget-based evaluations that decompose long-form answers into atomic facts have emerged as a promising strategy for RAG evaluation. In this work, we employ AutoNuggetizer, a nugget-based framework, to analyze ?5K Search Arena battles from LMArena by automatically generating and assigning nuggets, converting each model response into a quantitative score. We observe strong alignment between nugget-based Elo rankings and human preferences, exceeding the corresponding alignment achieved by LLM-as-a-judge with chain-of- thought (CoT) evaluation, while substantially reducing the number of preference inversions. Furthermore, we provide in-depth analyses including inversions, nugget quality, and shared-blindness effects. All our code is available at https://github.com/castorini/SxSNuggets.
Sahel Sharifymoghaddam, Shivani Upadhyay, Nandan Thakur et al.· International Conference on...· 0 citations
Generative artificial intelligence (GenAI) is rapidly reshaping the audience’s relationship with journalism, particularly through AI-generated summaries. Building on the observed patterns of limited visibility of sources, condensed summary presentation, and reduced contextual depth, we introduce the idea of news sufficiency, in which people encounter summarised content that feels enough to satisfy their immediate informational needs, thereby reducing the incentive to access full news articles. Empirically, this study examines differences between ChatGPT, Gemini, and Google Search when queried in European Portuguese, using three prompts: (a) asking for the main news of the day, (b) the latest on a specific news event, and (c) using only a basic keyword for that event. Drawing on a content analysis, we analysed 72 queries submitted by eight independent users. The results showed that platforms differ markedly in source attribution, summary structure, and contextual awareness. ChatGPT consistently provided hyperlinks and often clickable news images, while Gemini offered summaries only in text without citations. Both chatbots generated diverse and coherent news headlines capable of satisfying curiosity yet frequently failed to infer news intent from the basic keyword prompt, providing outdated or irrelevant information. The absence of Google’s AI Overview results during the observation window produced a contrast with a classic Google Search, which preserves user agency by requiring clicks on links. These findings have significant implications for journalistic authorship, editorial gatekeeping, and the economic sustainability of media, highlighting the need for AI literacy and platform governance that safeguards information pluralism.
Paulo Couraceiro, Pedro Caldeira Pais· Journalism and Media· 0 citations
A fundamental disconnect is suggested between a model's capacity for factual accuracy and its ability to maintain social fairness, highlighting the need for multi-dimensional evaluation frameworks for small-scale systems.
M.J.F. Valdez, Arghir-Nicolae Moldovan· Proceedings of the 15th Inte...· 0 citations