PRISM-IR: Fine-Grained Expert Selection for All-in-One Image Restoration
Abstract
Recent All-in-One Image Restoration (AIR) methods leveraging pre-trained diffusion models have demonstrated great potential. However, they struggle with two fundamental challenges: the inherent conflict between optimizing for perceptual realism and pixel-wise fidelity, and the inability to model the relationships and differences among degradations, resulting in compromised restoration performance. To address these limitations, we introduce PRISM-IR, a novel Low-Rank Adaptation (LoRA) allocation method for AIR. Our method first performs a coarse-grained allocation by disentangling the adaptation into two distinct pathways: a reconstruction LoRA to reduce distortion and a perceptual LoRA to enhance visual quality. This dual-pathway design allows for explicit control over the trade-off between realism and fidelity during inference. Specifically, the perceptual pathway can be continuously modulated by an inference-time strength coefficient, enabling users to move along the fidelity-realism spectrum without retraining the model. For the reconstruction pathway, we propose a fine-grained, degradation-aware selection mechanism. By considering the sub-matrices in LoRA as experts, this mechanism dynamically selects the most appropriate experts based on degradation-related features. This enables a fine-grained adaptation to capture both the unique characteristics of individual tasks and the underlying similarities among them. The proposed method establishes a new state-of-the-art across several challenging image restoration benchmarks, demonstrating superior performance and efficiency.