Behavioral Consistency in Machine Learning

Abstract

A trained model is usually described by its weights, as if the weights fixed its behavior. In practice the same weights are run by different programs. Training uses high-precision accumulation and fixed tensor shapes, while serving uses low-precision arithmetic, fused attention kernels, a paged key-value cache, and dynamic batching. These programs produce different next-token distributions from identical weights. We make this dependence explicit by indexing a policy by the kernel that executes it, and we propose kernel contracts, which state how far a serving kernel may drift from a reference kernel on each slice of traffic and what the serving system should do when a limit is exceeded. We prove non-asymptotic bounds that link the logit difference between two kernels to total-variation and KL divergence, to the change in expected reward, to the bias of policy gradients computed from rollouts generated by another kernel, and to whether the greedy token can change. We test the bounds and the contracts in GPU experiments on Qwen2.5-7B-Instruct with Hugging Face and vLLM. The bounds hold on every prompt we measured. Lower precision, a different serving engine, and a different batch size each change the generated token on some prompts, and these changes concentrate on prompts where the model is nearly undecided between two tokens. A kernel can therefore look fine on aggregate metrics while behaving differently on exactly these prompts. Checking a contract separately on each slice of traffic catches these cases.