Jiang-Qiang Li

Same Winners, Different Success Rates: Evaluating How LLM Agents Recover from Failures

Evaluating how LLM agents recover from mid-task failures is central to deploying reliable agentic systems. Existing checkpoint-based benchmarks measure recovery by comparing which action is selected as best across independent runs, a quantity known as set agreement. However, set agreement is a purely ordinal measure th...