MigBench: An Execution Certified Benchmark for Large Language Model Review and Repair of MongoDB Data Migrations
TL;DR
MigBench is a benchmark of 300 MongoDB migration scripts in which 100 are correct and 200 each contain exactly one defect from eight operationally defined categories, and every label is certified by execution.
Abstract
Production data migrations run with write credentials, often while the application they serve continues to handle traffic, and their worst failure modes concern how they change data rather than whether the code runs. Language model reviewers are increasingly asked to gate such scripts, with little evidence about their reliability in this setting. This paper presents MigBench, a benchmark of 300 MongoDB migration scripts in which 100 are correct and 200 each contain exactly one defect from eight operationally defined categories. The dataset is generated deterministically from a single seed, and every label is certified by execution: each script runs against a disposable MongoDB replica set under five behavioral probes covering expected state and scope, repeated execution, a counter race against simulated live traffic, crash injection with an invariant across collections, and crash injection followed by resume. All 300 labels were confirmed by behavior before any reviewer ran.