Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harnes...
Yuyao Ge, Sheng-Hua Liu, Pei-Zhong Gao et al.· 0 citations
Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this c...
Kun Chen, Hao-Rong Hong, Pei-Zhong Gao et al.· 6 citations
WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective, is proposed and experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.
ExplorationBench turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds, and finds that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can s...
Ming Zhang, Zhen Xiang, Pei-Zhong Gao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.