Skip to content

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

Nov 2026 · 0 citations · 32 references
Computer Science

TL;DR

DistDebug-Bench, a benchmark for evaluating agents'ability to diagnose complex-system failures at the code level, curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns.

Abstract

LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents'ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope. The benchmark and evaluator are available at https://github.com/OrderLab/distdebug-bench, with agent traces and a leaderboard at https://distdebug.com.

View source

Similar papers

Book Open access 2027

From Specification to Kernel Commit: Verified Code Generation on Real-World Systems

Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deplo...

Hao Sun, Ze-Nan Li, Cong Li et al. · 0 citations
#artificial intelligence Preprint Nov 2026

MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark

This paper presents MetaPI, an extensible toolkit that automatically identifies plausible injection points, generates realistic attack objectives, optimizes adaptive payloads, and validates attacks in the real environment, and enables scalable and consistent security evaluation across diverse and evolving agent ecosyst...

Pei-Ran Wang, Chong Xiang, Wen-Jie Qu et al. · 0 citations

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

E2E-SWE is introduced, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end and finds substantial variation in end-to-end repository generation ability, providing strong model differentiation while leaving considerable headroom for future progress.

Hantian Ding, Chloe Bi, Jia-Cheng Zhu et al. · 0 citations
#artificial intelligence Preprint Nov 2026

SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification

SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levels in a gamified manner, provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.

Álvaro F. Silva, Martim Afonso, Nuno Saavedra et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.