Skip to content
Book Open access

Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

Aug 2026 · Proceedings of the 2026 ACM Conference on Human-AI Complementarity and Alignment · pp. 187-196 · 0 citations · 73 references
Computer Science

TL;DR

An expert human evaluation is conducted, measuring summary preferences based on information satisfaction given a specific person’s background and use case, and finds that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.

Abstract

The majority of work on summarization evaluation focuses on general summary quality (e.g., ROUGE, BERTScore) or specific desired properties (e.g., readability, factuality). However, these metrics fail to measure the utility of a summary to an individual user. For example, a biomedical researcher learning about the latest vaccine research will have different informational needs from a family doctor. Query-focused summarization captures part of this need, but in practice, users rarely state everything relevant in a query: a single short query is likely inadequate to distinguish the needs of a researcher from those of a physician. By contrast, a reader’s background or persona—their role and expertise—is comparatively stable across queries and recovers much of this missing context, which makes it a practical signal for assessing whether a summary satisfies that reader’s needs. In this work, we assess how sensitive popular summarization metrics are to both informational and persona differences, and find that many popular metrics, including strong LLM-as-judge metrics, fail basic perturbation tests of informational content. We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction given a specific person’s background and use case. We find that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.

Read PDF

Similar papers

Open access Sep 2026

Evaluating Large Language Models for Biomedical Text Summarization: A Study of Cardiovascular Research

In this study, a comprehensive evaluation of abstractive and extractive summarization performance across three prominent large language models (LLMs): ChatGPT, DeepSeek, and Gemini is presented. A total of 8,000 cardiovascular-related research abstracts were collected from PubMed and summarized using two distinct promp...

Burcu Baştürk, Aytuğ Onan · 0 citations
#natural language process... Preprint Aug 2026

Improving Information Extraction with Learned Queries

This paper shows that another part of the pipeline matters at least as much: the queries used to elicit information extraction, and introduces List of Questions (LoQ), which generates document-specific question sets, and FeedQ, a feedback-driven optimization method that iteratively refines questions against extraction...

Omar Sharif, S. Vosoughi, Nikhil Singh · 0 citations
#artificial intelligence Preprint Sep 2026

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We o...

Nikhil Reddy Pottanigari, Ramin Fahimi, Noah Bolger et al. · 0 citations
#natural language process... Preprint Aug 2026

WildSEEK: Evaluating Language Models for Information-Seeking

This work introduces WildSEEK, a manually annotated dataset of 3k information-seeking queries from real user interactions, and an evaluation framework for LLM-generated responses, and finds that over a third of information-seeking queries are high-risk and more often analytical.

Tanise Ceron, Joachim Baumann, Elisa Bassignana et al. · 0 citations
#explainable ai Preprint Sep 2026

Relevance is not enough: A Communication-Oriented Retrieval System for Consequential Scientific Question Answering

This work focuses on consequential scientific questions whose results directly shape people's lives and study them through a public water-quality communication system, where residents and community leaders interpret the findings and choose actions.

Avinab Nakarmi, Naga Datha Saikiran Battula, Anthony Diaz et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.