Research and Analysis of Cross-Modal Attention Based on Pareto Optimal Fusion of Sketch Image Generation

Abstract

Conditional Sketch Generation aims to convert abstract sketches into realistic images. This task is challenging due to the modality gap between sketches and photos, weak pixel-level correspondence, and the one-to-many mapping problem. This paper introduce two paradigm shifts within a single framework, replacing ControlNet's zero-convolution with deep cross-modal attention and noise prediction with L1 reconstruction loss. On this basis, a Dual-Path Non-Shared Encoder is presented. During the experiments on SketchCOCO, this paper uncovers three empirical phenomena with universal significance. Firstly, all evaluation metrics exhibit a synchronous improvement phase when w < 0.7 . Secondly, the guidance weight exhibits a stable Pareto interval, with model performance sustaining high stability for w ∈ [ 0.6,0.9 ] . Finally, through dense sampling with a step size of 0.1, w = 0.7 achieves the best trade-off across all metrics (FID = 177.90, Corr = 0.7160, SSIM = 0.6419). Meanwhile w = 0.9 yields the lowest LPIPS (0.1949) and w = 0.8 achieves the highest SSIM (0.6471). This work not only provides a lightweight yet powerful model for sketch generation, but also offers actionable insights for designing fusion mechanisms in cross-modal tasks.