Preprint
Jul 2026
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The ORCA-bench benchmark is introduced, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting and is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability.
Albert Gong, Kyuseong Choi, Abhineet Agarwal et al.
· 0 citations