DistDebug-Bench, a benchmark for evaluating agents'ability to diagnose complex-system failures at the code level, curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns.
Abstract
LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents'ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope. The benchmark and evaluator are available at https://github.com/OrderLab/distdebug-bench, with agent traces and a leaderboard at https://distdebug.com.
AgentScript-Eval is introduced, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents.
Dipin Khati, Shubham Mehrotra, Bin Bi et al.· 0 citations
Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deplo...
Hao Sun, Ze-Nan Li, Cong Li et al.· Proceedings of the 32nd ACM...· 0 citations
This paper presents MetaPI, an extensible toolkit that automatically identifies plausible injection points, generates realistic attack objectives, optimizes adaptive payloads, and validates attacks in the real environment, and enables scalable and consistent security evaluation across diverse and evolving agent ecosyst...
Pei-Ran Wang, Chong Xiang, Wen-Jie Qu et al.· 0 citations
TomasuLLM is presented, a runtime that executes agent tool calls out of trajectory order while preserving task-execution correctness, and improves the reported benchmark means and scales with tool latency.
Jiang-Nan Yu, Ce-Yu Xu, Meng-Ming Li et al.· arXiv.org· 0 citations
E2E-SWE is introduced, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end and finds substantial variation in end-to-end repository generation ability, providing strong model differentiation while leaving considerable headroom for future progress.
SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levels in a gamified manner, provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Álvaro F. Silva, Martim Afonso, Nuno Saavedra et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026