Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation
An expert human evaluation is conducted, measuring summary preferences based on information satisfaction given a specific person’s background and use case, and finds that both traditional and LLM-based metrics are insufficient measures of information satisfaction and agree poorly with human judgment.