GitHub - OrderLab/distdebug-bench: DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? (NeurIPS 2026 Evaluations & Datasets)

Can LLM agents diagnose bugs in distributed systems?

Website · Leaderboard · Paper · Dataset · Traces · Submit

DistDebug-Bench is a benchmark for evaluating whether LLM agents can diagnose real-world failures in distributed systems. Each task gives the agent a bug report from the system's issue tracker, the logs and stack traces attached to it, and the source code as it was before the fix. The agent has to find the code that caused the failure. We score its diagnosis against human-labeled root-cause code blocks, rather than by whether a patch passes tests.

Bugs in distributed systems are hard to diagnose. They depend on timing, message ordering, and partial failures across nodes, are often difficult to reproduce, and rarely come with a failing test. In our evaluation of 17 agent configurations, the best agents find the right files 84% of the time but cover at most 68% of the root-cause code blocks. Results also vary between runs: on 12–29% of bugs, block coverage differs by more than 20 points across three attempts.

✨ Highlights

120 real bugs from the issue trackers of 11 distributed systems, including ZooKeeper, Kafka, HDFS, etcd, Kubernetes, Ray, and TiKV. They cover 5 languages and 10 failure patterns, such as concurrency races, leader-election faults, replication inconsistencies, and resource leaks.
Human-labeled root causes. Each bug comes with the code blocks that caused it, so a diagnosis is scored at the file, function, line, and block level.
No access to the fix. The agent's workspace contains only the pre-fix snapshot (no later commits, branches, tags, or remotes), and its commands run offline in a bubblewrap sandbox. A leakage check flags attempts that mention the original issue.
Works with any agent. We include Mini-SWE-Agent configs for 13 models in configs/. Other agents only need to write a diagnosis.json.
Reproducible results. All 6,120 scored attempts from the paper are on Hugging Face, and paper/reproduce.py rebuilds the paper's main table and figures from them.
A Lite subset of 34 bugs for cheaper runs.

🏆 Leaderboard

Top configurations on the full benchmark (120 bugs, Mini-SWE-Agent, mean of three attempts per bug). Block coverage is the primary metric.

#	Model	Block coverage	File coverage	Tokens / run
1	GPT-6 Astra	67.7 _{[60.4, 74.9]}	83.9	132K
2	Claude Opus 5.5	66.4 _{[59.2, 73.6]}	82.0	193K
3	GPT-5.5	65.1 _{[57.8, 72.3]}	84.0	468K
4	GPT-5.4	63.1 _{[56.0, 70.2]}	79.8	690K
5	GPT-5.3-Codex	59.7 _{[52.2, 67.2]}	79.1	416K

_{Brackets are 95% bug-bootstrap confidence intervals.}

The leaderboard lists all 17 configurations. To add yours, see Submitting to the leaderboard.

🧩 The task

Input. The issue title and description, its attachments (logs, stack traces, analyses), and the system's source code at the pre-fix snapshot. The agent works in a sandboxed shell with no network access.

Output. A diagnosis.json that describes the cause of the failure and lists the code blocks responsible:

{
  "bug_cause_description": "Leader election retains a stale ZKDatabase ...",
  "buggy_blocks": [
    {
      "file": "src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
      "function": "ZooKeeperServer.shutdown",
      "start_line": 555,
      "end_line": 558,
      "bug_contribution": "...",
      "patch": "..."
    }
  ]
}

The scorer compares the predicted blocks with the labeled ones and reports precision and coverage at four levels (see metrics).

Systems

System	Language	Bugs	System	Language	Bugs
Ray	Python, C++	21	Kafka	Java	8
Elasticsearch	Java	19	etcd	Go	8
ZooKeeper	Java	18	Ignite 3	Java	7
TiKV	Rust	11	HBase	Java	5
HDFS	Java	10	Ignite	Java	3
Kubernetes	Go	10

🚀 Quick start

You need Linux with bubblewrap, Python 3.10 or later, and an API key for your model provider. The commands below run Mini-SWE-Agent with GPT-5.4 on one bug, ZooKeeper-2845, and score the result:

git clone https://github.com/OrderLab/distdebug-bench.git
cd distdebug-bench
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[agent]"
export OPENAI_API_KEY=...        # or the provider key the config needs

python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml \
    --bug zookeeper-2845 --runs 1
python -m distdebug_bench.score results/mini-swe-agent-gpt-5.4 --attempts 1

The attempt is saved under results/mini-swe-agent-gpt-5.4/zookeeper-2845/. configs/mini-swe-agent/ has a config for each model in the paper.

To load the data without running an agent, pip install -e . is enough. It has no dependencies:

from distdebug_bench import get_bug

bug = get_bug(".", "zookeeper-2845")
print(bug.title)                # Data inconsistency issue due to retain database in leader election
print(bug.ground_truth_blocks)  # the labeled root-cause code blocks

🧪 Running the full benchmark

The runner reproduces the paper's Mini-SWE-Agent evaluation. One config per model in the paper's Table 1 is in configs/mini-swe-agent/.

pip install -e ".[agent]"
export OPENAI_API_KEY=...        # or the provider key the config needs

# Run every bug until it has three successful attempts (resumable).
python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml --parallel 8

# Flag attempts that named the original issue, set them aside, and rerun them.
python -m distdebug_bench.leakage results/mini-swe-agent-gpt-5.4 --quarantine
python -m distdebug_bench.run --config configs/mini-swe-agent/gpt-5.4.yaml --parallel 8

# Score and aggregate as in the paper.
python -m distdebug_bench.score results/mini-swe-agent-gpt-5.4

Use --subset lite, --bug zookeeper-2845, or --system kafka to run fewer bugs, and --runs 1 for a single attempt per bug. Commands run inside bubblewrap, which must be installed (Linux only).

The protocol, as in the paper:

Agent. Mini-SWE-Agent 2.2.8 with the prompt in distdebug_bench/mini_swe_agent_prompt.yaml, a $3 cost limit per attempt, and a 60-second limit per command.
Input. The issue title and description, the issue's attachments (copied to .artifacts/ in the workspace), and the system's source at the pre-fix snapshot. The agent writes diagnosis.json and checks it with .validate_diagnosis.py.
Leakage controls. Each workspace has a random name and contains only the snapshot and its ancestors: no branches, tags, remotes, reflog, or later commits, so the fix is unreachable. chafa and img2txt are disabled in the sandbox. distdebug_bench.leakage flags attempts whose model-visible text names the original issue; the paper discards and reruns them.
Attempts. Each bug is attempted until it has three schema-valid diagnoses. Scoring takes the first three in run order, averages them within the bug, then averages equally over bugs.

Each attempt is saved under results/<config>/<bug>/run-NN/ as trajectory.json, diagnosis.json, session_stats.json (API calls, tokens, cost), and, after scoring, diagnosis_scores.json.

🛠️ Evaluating your own agent

Any agent or harness can be evaluated. Give it what the runner gives Mini-SWE-Agent: the issue title and description, the issue's attachments, and the system's source at the pre-fix snapshot, with no internet access. For each attempt, have it write <bug>/run-NN/diagnosis.json in the benchmark schema (checked by distdebug_bench/validate_diagnosis.py) and a human-readable <bug>/run-NN/trajectory.json. Then run the leakage guard and the scorer on the results directory, as above.

The Python API loads a bug and its task prompt, checks out the pre-fix snapshot so that the fix is unreachable, and scores a single diagnosis.

📤 Submitting to the leaderboard

The leaderboard lists every evaluated configuration, and the submission guide summarizes the steps below.

Run every bug in the full benchmark (or in Lite) until it has three valid attempts. Use distdebug_bench.run, or your own agent writing the same layout: <bug>/run-NN/diagnosis.json (checked by distdebug_bench/validate_diagnosis.py) and a human-readable <bug>/run-NN/trajectory.json.
Run python -m distdebug_bench.leakage <results> --quarantine and rerun the flagged attempts.
Copy submissions/metadata.template.yaml, fill it in, and package the run:
```
python -m distdebug_bench.submit package results/<run> --metadata metadata.yaml
```
This scores the run and writes submissions/<date>_<agent>__<model>/ (metadata, scores, and diagnoses) plus a trajectories archive.
Upload the archive anywhere public (Google Drive, Hugging Face, Zenodo, ...), set trajectories_url in the submission's metadata.yaml, and check it:
```
python -m distdebug_bench.submit check submissions/<date>_<agent>__<model>
```
Open a pull request adding the submission directory, or email jyuxuan@umich.edu with the subject [DistDebug-Bench-Submit] <agent> + <model> and links to the submission directory and the trajectories.

Rules:

Every bug in the chosen subset needs three valid attempts; scores are aggregated by distdebug_bench.score. Report Lite results separately; the leaderboard currently lists full-benchmark runs.
The agent must not access the original issues, their fixes, or this benchmark's repository or website. We rerun the leakage guard and review trajectories.
Describe the system in a paper, technical report, or blog post (report_url).
We rescore every submission from its diagnoses before listing it.

🔬 Reproducing the paper's results

The traces behind the paper's Table 1 are on Hugging Face at OrderLab/distdebug-bench-traces: 17 configurations, 120 bugs, and the three scored attempts per bug, for 6,120 attempts. paper/reproduce.py rescores every diagnosis with distdebug_bench.score and aggregates the scores with the paper's scripts. It rebuilds Table 1 and Figures 2 and 3 in paper/out/ and checks them against paper/reference/:

pip install -e ".[paper]"
hf download OrderLab/distdebug-bench-traces --repo-type dataset --local-dir traces
python paper/reproduce.py traces

It prints whether each configuration's scores, token use, and confidence intervals match the paper. Pass --use-released-scores to aggregate the stored diagnosis_scores.json files instead of rescoring.

📚 Reference

Repository structure

.
├── benchmark/
│   ├── systems.json
│   ├── partition.json
│   ├── bug_classifications.json
│   ├── bug_difficulty_characterizations.json
│   ├── lite.txt                           # DistDebug-Bench Lite: 34-bug subset
│   └── bugs/<system>/<bug-id>/
│       ├── report.json
│       ├── blocks.json
│       └── artifact_dir/                  # optional supporting logs and traces
│
├── distdebug_bench/                       # Loader, evaluator, and agent runner
├── assets/logo/                           # Logo files (SVG and PNG)
├── configs/mini-swe-agent/                # The paper's model configurations
├── paper/                                 # Rebuilds the paper's Table 1 and Figures 2-3 from the traces
├── submissions/                           # Leaderboard submissions
├── release/                               # Hugging Face export and Croissant metadata
├── tests/
├── collection-pipeline/                   # Pipeline used to construct the benchmark
└── utils/

Per-bug file layout under `benchmark/bugs/`

Each bug directory at benchmark/bugs/<system>/<id>/ contains two JSON files and an optional artifacts directory.

`report.json`

Bug metadata and natural-language description.

{
  "id": "2845",
  "system": "zookeeper",
  "branch": "release-3.5.3",
  "title": "Data inconsistency issue due to retain database in leader election",
  "description": "In ZOOKEEPER-2678, the ZKDatabase is retained ...",
  "issue_link": "...",
  "fix_link": "...",
  "severity": "Critical",
  "root_cause_type": "Concurrency",
  "bug_complexity": "Hard",
  "distributed": "Yes",
  "contains_analysis": "Yes",
  "contains_logs": "No",
  "contains_stack_traces": "No",
  "artifact_dir": "artifact_dir"
}

branch is the upstream branch to check out before running an agent. artifact_dir, when present, is a relative path to supporting logs and traces.

`blocks.json`

The ground-truth root-cause code blocks. A bug may have one or more blocks spanning multiple files.

[
  {
    "file": "src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
    "function": "ZooKeeperServer.shutdown",
    "start_line": 555,
    "end_line": 558
  }
]

Line numbers may appear as strings or integers depending on the curator's tooling; the loader coerces both to int.

Python API

Python 3.10 or later is required. We recommend installing into a virtual environment so that subsequent python invocations resolve to the project's interpreter:

git clone https://github.com/OrderLab/distdebug-bench.git
cd distdebug-bench
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

Loading a bug from the benchmark

from distdebug_bench import load_benchmark, get_bug

bench = load_benchmark(".")
print(len(bench.systems), "systems,", len(bench.lite_bug_ids), "Lite bugs")

bug = get_bug(".", "zookeeper-2845")
print(bug.title)
print(bug.branch)
print(bug.ground_truth_blocks)
print(bug.task_prompt)

for bug in bench.iter_bugs("all"):  # or "lite"
    print(bug.name, len(bug.ground_truth_blocks))

Subsets

Subset	Bugs	Use
`all`	120	The full benchmark, as evaluated in the paper.
`lite`	34	DistDebug-Bench Lite: a subset selected for report and label quality and lower evaluation cost. Report Lite results separately.

Checking out the buggy revision of the upstream system

from distdebug_bench import ensure_system_repo, temporary_checkout

system = bench.systems[bug.system]
cache = ensure_system_repo(system, "./.cache/repos")
with temporary_checkout(bug, cache, "./.cache/workspaces") as workdir:
    # `workdir` has a random name and holds only `bug.branch` and its
    # ancestors, so later commits (including the fix) are unreachable.
    # The directory is removed automatically on exit.
    ...

Scoring an agent's predicted root-cause blocks

from distdebug_bench import compute_correctness_report
from distdebug_bench.models import BuggyBlock

predicted = [BuggyBlock(
    file="src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java",
    function="ZooKeeperServer.shutdown",
    start_line=555, end_line=558,
)]
report = compute_correctness_report(bug.ground_truth_blocks, predicted)
# To also score mention coverage as in the paper, pass the raw text of the
# agent's diagnosis.json as a third argument.
print(report)

Evaluation metrics produced by `compute_correctness_report`

This is the scorer behind the paper's results: rescoring every attempt in the paper reproduces its stored scores and Table 1 exactly. Line ranges are half-open, [start_line, end_line).

Metric	Unit	A prediction matches the ground truth when
`file_precision`, `file_coverage`	distinct files	the paths are equal after normalization, or one contains the other
`function_precision`, `function_coverage`	distinct functions	the names agree after canonicalization to lowercase `class.method` (argument lists, Go receivers, and `::`/`#`/`$` delimiters are handled; a bare method matches its qualified name)
`line_precision`, `line_coverage`	lines	overlapping line numbers in a matching file
`block_precision`, `block_coverage`	blocks	the ranges overlap within 5 lines in a matching file (function names are not used)
`strict_block_coverage`	blocks	the ranges overlap without tolerance in a matching file
`mention_coverage`	blocks	the ground-truth function appears in the diagnosis text (generic names such as `run` must appear as `class.method`)

Precision divides by predicted files, functions, lines, or blocks; coverage divides by ground-truth ones. block_coverage is the paper's main metric.

🤝 Contributing

To suggest a bug for the benchmark, open an issue with the Bug Entry template. We also welcome pull requests, including scorer fixes and configs for new agents.

📄 License

The benchmark annotations and code are released under the MIT license. Issue reports and attachments come from public issue trackers and remain subject to their terms; upstream source code referenced by report.json and blocks.json is governed by each project's own license.

📝 Citation

If you use DistDebug-Bench in your research, please cite:

@inproceedings{jiang2026distdebugbench,
  title     = {{DistDebug-Bench}: Can {LLM} Agents Diagnose Bugs in Distributed Systems?},
  author    = {Jiang, Yuxuan and Liu, Chenglin and Rose, John and Miller, Mason and Thavamani, Ruthesh and Wu, Juntao and Huang, Yinwei and Xu, Jiahao and Huang, Peng},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

_{If DistDebug-Bench is useful to you, a ⭐ helps others find it.}

Navigation Menu

Repository navigation

About

Repository files navigation

Can LLM agents diagnose bugs in distributed systems?

✨ Highlights

🏆 Leaderboard

🧩 The task

Systems

🚀 Quick start

🧪 Running the full benchmark

🛠️ Evaluating your own agent

📤 Submitting to the leaderboard

🔬 Reproducing the paper's results

📚 Reference

Repository structure

Per-bug file layout under `benchmark/bugs/`

`report.json`

`blocks.json`

Python API

Loading a bug from the benchmark

Subsets

Checking out the buggy revision of the upstream system

Scoring an agent's predicted root-cause blocks

Evaluation metrics produced by `compute_correctness_report`

🤝 Contributing

📄 License

📝 Citation

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
assets/logo		assets/logo
benchmark		benchmark
collection-pipeline		collection-pipeline
configs/mini-swe-agent		configs/mini-swe-agent
distdebug_bench		distdebug_bench
paper		paper
release		release
submissions		submissions
tests		tests
utils		utils
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation

Can LLM agents diagnose bugs in distributed systems?

✨ Highlights

🏆 Leaderboard

🧩 The task

Systems

🚀 Quick start

🧪 Running the full benchmark

🛠️ Evaluating your own agent

📤 Submitting to the leaderboard

🔬 Reproducing the paper's results

📚 Reference

Repository structure

Per-bug file layout under benchmark/bugs/

report.json

blocks.json

Python API

Loading a bug from the benchmark

Subsets

Checking out the buggy revision of the upstream system

Scoring an agent's predicted root-cause blocks

Evaluation metrics produced by compute_correctness_report

🤝 Contributing

📄 License

📝 Citation

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Per-bug file layout under `benchmark/bugs/`

`report.json`

`blocks.json`

Evaluation metrics produced by `compute_correctness_report`