From Dyadic Derivatives to Pareto Circuit Surgery: Resolving the Quantization Wall, Static-Vector Dilution, and Low-Rank Attention Editing in Qwen2.5-7B

Abstract

Prior work on Qwen2.5-7B-Instruct showed that the raw size of a residual-stream displacement does not predict its effect, while a simple pre-intervention measure of local sensitivity does rank which soft patches will move the model’s decision margin. Three problems remained. A second-order curvature assay failed, in part because bfloat16 quantization swallowed the small shifts it was meant to detect. Computing the sensitivity itself required an expensive extra forward and backward pass. And the earlier demonstrations of causal potency only showed that interventions could degrade correct answers, not that they could restore them. This paper reports five locked experimental stages that address those gaps. On a simple non-gradient stochastic system, a drift-filtering method recovered the rotational parameter more accurately than raw finite differences, and a geometric path optimizer reduced the transition cost by 48% relative to a straight-line path. On the language model itself, a paired precision audit at the final layer showed that native bfloat16 collapses a 16-step micro-dose ramp to a single flat zero, while recomputing the same steps in float32 restores an essentially linear response. Replacing global curvature estimates with a local analysis of late-layer GeGLU neurons—identified only on geography and chemistry prompts—allowed selective knockout of 64 neurons per layer that improved held-out physics, arithmetic, and taxonomy margins by roughly 0.19 to 0.25 nats, while count-matched random knockout did nothing. The same sequence also explains why a static cross-domain steering vector fails to beat matched random noise: unnormalized addition is diluted by later RMSNorm layers, and spherical perturbations let downstream attention re-read the clean prefix cache. Compiling the intervention into permanent Rank-1 edits of the attention output matrices, at a carefully chosen strength, recovered 0.116 nats on held-out items while keeping reference-text drift at 0.0027 nats—well inside a 0.05-nat budget, and far below the drift produced by a stronger Rank-2 edit.