When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agent...
Kaiser Sun, Bernal Jiménez Gutiérrez, Hong-Jun Liu et al.· 0 citations
This work introduces PrimeScientist, which jointly determines research direction and resource investment across successive research attempts, and proposes an adaptive MCTS-based allocation policy that balances exploration and exploitation using experimental feedback and remaining resources.
The Last Translation Benchmark is introduced, a collection of human-authored and peer-reviewed examples that break leading machine translation models and a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and a...
Vilém Zouhar, Niyati Bafna, Mukund Choudhary et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.