This work introduces Selection-Aware Semantic Stress Testing (\SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks.
Abstract
Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selected from the same data. We introduce Selection-Aware Semantic Stress Testing (\SASST{}), which learns a task reweighting from pre-execution features on discovery tasks and evaluates the same paired comparison on separate confirmation tasks. The protocol checks support and stability, uses joint bounds for all planned claims, and can return no claim. We prove conditional asymptotic validity under stated cluster assumptions. A forty-cluster audit finds Gaussian undercoverage and conservative Bonferroni $t$ bounds. In one 480-episode $\tau$-bench study, a $3.75$ point discovery gain vanished on confirmation. A second-model study likewise confirmed neither a workflow benefit nor a stable stress rule.
SelfSuite is presented, in which the agent's own base model, given only the world's public materials, designs a small evaluation suite of weighted judges and grounded per-task briefs, freezes it, and uses it to gate a keep-best retry and to label a typed, outcome-tracked memory.
Saeid Asgari, Emre Kıcıman, Leonardo de Oliveira Nunes et al.· 0 citations
The study contributes a traceable evaluation that connects aggregate reward to contract-level evidence and judge sensitivity, together with concrete review checks for migration advice, together with concrete review checks for migration advice.
Bei-Ming Liu, Hai-Hao Li, Min-Jie Chen et al.· 0 citations
This work introduces RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots, and reduces source-task degradation after a shift, whereas random retention reaches a higher destination endpoint.
Zi-Bo Zhao, Ji-Jun Shi, Mo-Qing Zhou et al.· 1 citation
This work reconstructs how evidence is reused in a 400-task NeuroGolf campaign, with selected wellbore-prediction records from the same operator as cross-domain comparisons, to preserve not just experimental results, but their limits on subsequent action.
Evaluation of agentic information retrieval remains limited to scripted interactions with uniform users, missing both natural personality diversity and adversarial brittleness. We present AgentWorld, a simulation framework combining (i)Big Five (OCEAN) personality-driven user populations with stateful tool-use environm...
Gunjan Agarwal, Arup Das, Arun Menon et al.· 1 citation
Joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost supports joint reporting of priority responsiveness, presentation-dependent defaults, and task success and cost.
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
