SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
SpecBench is introduced, a benchmark comprising 30 systems-level programming tasks ranging from short horizon tasks like building a JSON parser to ultra long horizon tasks like building an entire OS kernel from scratch, which offers a principled testbed for measuring whether coding agents build genuine working systems...
Bing-Chen Zhao, Dhruv Srikanth, Yuxiang Wu et al.
· arXiv.org · 25 citations
· ⚡1