Diffusion-based image style transfer via parameter-free saliency recalibration

Abstract

Image style transfer focuses on rendering a content image with the visual appearance of a reference artwork while keeping its main semantic layout unchanged. StyleID enables image-guided stylization without additional training by recording attention features during DDIM inversion, yet its results may still suffer from weak local texture activation, amplified unrelated patterns, and mild boundary distortion. To alleviate these problems, we develop a diffusion stylization framework that recalibrates injected features using a parameter-free saliency mechanism. Specifically, SimAM is inserted into the residual branch after style injection in the U-Net Spatial Transformer of StyleID. Without introducing learnable parameters, SimAM estimates neuron importance from the internal energy distribution of the injected 4D feature map, enhancing key texture and brushstroke responses while suppressing redundant style perturbations. Experiments on MS-COCO and WikiArt demonstrate improved trade-offs between stylization strength and content fidelity.