Mutation Testing in the Wild: Challenges and Lessons from Bitcoin Core

Abstract

Mutation testing is a technique for evaluating test suite effectiveness, yet its adoption in industrial settings remains limited due to scalability challenges. In this paper, we report our experience applying mutation testing to Bitcoin Core, the reference implementation of the Bitcoin protocol - a safety-critical, C++ codebase with hundreds of thousands of lines of code and a heterogeneous test suite. We present bcore-mutation, an open-source tool written in Rust and tailored to Bitcoin Core’s specific constraints, and describe the challenges we faced: test selection for incremental analysis, unproductive mutant filtering, and surfacing results to pull request authors without disrupting the review workflow.We further discuss how mutation testing was applied to evaluate the quality of fuzz harnesses, and report an experimental use of Large Language Models to automatically generate tests that kill surviving mutants. Our experience shows that mutation testing is tractable even on large, complex codebases when the tooling is designed around the realities of the project, and offers lessons for practitioners considering similar approaches on safety-critical systems.