Results distinguish functional success from constraint compliance, with failures concentrated in workflow ordering, tool-use discipline, and reporting, and substantial interactions between models and scaffolds.
Wen-Ya Zhu, Xin-Ping Lei, Zhi-Yi Lai et al.· 0 citations
While test-time scaling enhances Large Language Model (LLM) agents in long-horizon software engineering (SWE), sparse binary rewards (Pass/Fail) create a severe credit assignment crisis and waste failed exploratory trajectories. Current trajectory optimization and scaling methods are costly and structurally limited, re...
Jing-Xuan Xu, Gang Wu, Ya-Nan Wu et al.· 0 citations
This work introduces GameLogicBench, a benchmark of 72 gameplay-logic tasks in Godot projects, and compared versions of the benchmark evaluator built with and without validation using mutants.
Xin-Yu Che, Yun-Fei Ge, Shi-Hao Li et al.· 2 citations
It is found that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost.
Yu-Hao Wu, Jing-Yuan Zhang, Jia-Jun Shi et al.· 12 citations· ⚡1
This work introduces ASPIRE, a benchmark for vague-goal-driven self-evolution and shows that vague goals redirect search effort toward goal interpretation, and evaluates the resulting systems on a hidden, expert-authored set of 520 items spanning six goals.
Yu-Hao Wu, Jingyuan Zhang, Jia-Jun Shi et al.· 0 citations
These findings show that recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies, and provide a unified framework for diagnosing this process and identifying the bottlenecks that prevent agents from translating interaction experience into reliable se...
Jia-Jun Shi, Siyang Tao, Yu-Hao Wu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.