LLM-Pyro: LLM-Based Mutation Testing with Equivalence Triage for Python

Abstract

Mutation testing assesses test suite quality by injecting controlled faults into production code. Recent studies show that large language models (LLMs) can generate more diverse mutants, closer to real faults, than traditional syntactic operators, though at the cost of more non-compilable, duplicate, and equivalent mutants. This evidence, however, is concentrated in Java, leaving Python’s dynamic typing largely unexplored, and existing tools either rely on fixed operators or apply LLMs without addressing equivalent mutants. We present LLM-Pyro, a Python adaptation of LLMorpheus that couples mutant generation with an LLM-based oracle classifying survivors as equivalent, non-equivalent, or unknown. We evaluate LLM-Pyro in a pilot study on HumanEval against mutmut and Mutahunter, and on real faults from PyBugHive. On HumanEval, LLM-Pyro produced more syntactically diverse mutants (average edit distance 2.96), with 72.2% compilability, 15.3% duplication, and 10.4% equivalence, generating 1.85× more mutants than mutmut and 4.53× more than Mutahunter (58.4% non-equivalent). On PyBugHive, within the 44 evaluated bugs from four projects, LLM-Pyro reached an 84.9% mutation score (vs. 77.9% for Mutahunter), suggesting its mutants explore code regions relevant to real faults, though only a fraction match the exact fault taxonomy. These results provide initial evidence, within the scope of the subjects evaluated, that LLM-based mutation with equivalence triage is feasible for Python.