Aug 2026· IEEE Transactions on Pattern Analysis and Machine Intelligence· Vol PP, pp. 1-16· 0 citations
Medicine
TL;DR
A large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to teach models to recognize and match fundamental structures across images, which generalizes effectively across more than eight unseen cross-modality registration tasks.
Abstract
Image matching, which aims to identify corresponding pixel locations between images, is crucial in a wide range of scientific disciplines, aiding in image registration, fusion, and analysis. However, when dealing with images captured under different imaging modalities that result in significant appearance changes, the performance of learning-based image matching algorithms often deteriorates due to the scarcity of annotated cross-modal training data. This limitation hinders applications in various fields that rely on multiple image modalities to obtain complementary information. To address this challenge, we propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to teach models to recognize and match fundamental structures across images. This capability is transferable to real-world, unseen cross-modality image matching tasks. Our key finding is that the matching model trained with our framework generalizes effectively across more than eight unseen cross-modality registration tasks using the same set of network weights, substantially outperforming existing generalizable methods and achieving competitive or superior performance compared to specialized models on several tasks.
Retinal image registration is essential for ophthalmic diagnosis, longitudinal disease monitoring, and multimodal retinal image analysis. Existing retinal registration methods are typically modality-dependent: they are designed or optimized either for a single imaging modality in mono-modal registration or for a fixed pair of modalities in cross-modal registration. This limits their flexibility and applicability in practical scenarios involving diverse retinal imaging modalities and different combinations of them. In this work, we propose a generalizable two-stage, modality-invariant framework for retinal image registration. First, we introduce a sparse feature-matching model driven by a universal retinal vessel segmentation to achieve robust coarse global alignment across modalities. Second, we develop a modality-invariant optical flow estimation network, termed MI-RAFT, to refine the alignment through dense local registration. Extensive experiments demonstrate that the proposed method can handle diverse combinations of commonly used retinal imaging modalities, exhibiting strong modality invariance while outperforming state-of-the-art modality-dependent registration methods.
Bo Wen, N. Mehta, Melanie Tran et al.· 0 citations
Multi-focus image fusion combines partially focused images into a single all-in-focus composite. Existing object-based methods assume precise spatial and scale alignment across source images, an assumption that frequently fails in Misaligned Multi-Focus Dataset scenarios due to camera displacement and focal length variation. This paper proposes a novel training-free, object-aware fusion framework to address this limitation through a five-stage pipeline: YOLOv8x detection, SAM2-L segmentation, LoFTR correspondence matching, a novel Scale-Aware Area Resize Algorithm, and GLCM-guided MSB/LSB bit-level fusion. The framework was evaluated on the EDMF benchmark (20 image pairs, synthetically modified to simulate Misaligned Multi-Focus Dataset shifts) and a Misaligned Multi-Focus Dataset (10 image pairs). Results show consistent improvements over the best source in Average Gradient by 5.02% on the EDMF dataset and 6.31% on the Misaligned Multi-Focus Dataset, Mutual Information by 2.17% and 2.52%, and Gradient-based Quality by 3.10% and 3.53%, respectively. Ablation study confirms the MSB/LSB fusion strategy contributes +7.41% (AG) and +1.03% (MI) for mismatched objects. Qualitative results validate visual coherence and boundary fidelity of the fused outputs.
Baneen Al-Kalabi, Tawfiq A. Al-assadi· Journal of Intelligent Infor...· 0 citations
Dense image matching is crucial in applications such as 3D reconstruction, autonomous driving, and remote sensing mapping; however, weak textures, occlusions, and large-disparity scenes remain challenging. To address these issues, this paper proposes a dense matching network based on a Transformer and multi-scale feature fusion, called Task-aware Multi-Scale Matching Network (TMSMNet). First, Swin Transformer is used to model global context in feature maps, enhancing the feature discriminability in weak texture regions. Then, a multi-scale cost volume is constructed, and adaptive fusion is achieved through deformable convolution to accommodate disparity variations of different ranges. Finally, an attention- guided iterative optimization module is introduced to improve the matching accuracy in occluded regions. Experimental results on the Scene Flow, KITTI-2015, and Middlebury datasets show that TMSMNet outperforms mainstream methods such as RAFT-Stereo on the D1-all metric of KITTI- 2015 and demonstrates good generalization and robustness. Ablation studies also confirm the effectiveness of each module. In summary, the method in this paper provides a feasible approach for dense matching. Future work will explore model lightweighting to support real-time applications and attempt to combine generative models to handle completely textureless regions, further enhancing its performance in complex scenes.
Multi-modal image matching (MIM) remains a challenging problem due to nonlinear radiometric variations and geometric distortions across heterogeneous sensors. This paper proposes a robust feature-based matching framework that reduces reliance on intensity information while enhancing structural representation. The filter with local normalization is applied to transform the input images into a common intermediate domain. A block-based strategy is then employed to enforce a uniform spatial distribution of keypoints using the ORB (Oriented FAST and Rotated BRIEF) detector. To further suppress intensity variations and improve discriminability, a novel Max-Index-based HOG (MIHOG) is developed. This descriptor integrates multi-scale feature representations and encodes dominant structural information through discrete max-index mapping. Finally, correspondences are established using a brute-force matching strategy. Extensive experiments are conducted on two multi-modal datasets covering eight diverse scenarios. The proposed method achieves an average NCM of 224.52, RMSE of 3.4788, and SR of 92%. MIHOG obtains the highest NCM on 4/8 test scenarios and improves the average NCM by 18.3% compared with the second-best method. Meanwhile, it maintains competitive computational efficiency, with an average running time of 10.20s. These results demonstrate that MIHOG can provide dense and reliable correspondences under complex cross-modal radiometric and geometric variations.
Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language to provide additional information beyond RGB images, showing great potential in improving tracking stabilization in complex scenarios. Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data. Constrained by the limited multi-modal training data, the performance of these methods is unsatisfactory. To alleviate this limitation, this work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model. Diff-MM leverages the denoising network of pre-trained Stable Diffusion as a tracking feature extractor through the proposed parallel feature extraction pipeline, which enables pairwise image inputs for object tracking. We further introduce a multi-modal sub-module tuning method that learns to gain complementary information between different modalities. By harnessing the extensive prior knowledge in the generation model, we achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking. Extensive experiments are conducted on the mainstream diffusion model architectures, e.g., UNet and MMDiT. Experimental results demonstrate the promising performance of our method compared with recently proposed trackers, e.g., its AUC outperforms OneTracker by 10.4% on TNL2K. Our code will be released.
Shiyu Xuan, Zechao Li, Jinhui Tang et al.· IEEE Transactions on Pattern...· 0 citations
Deep learning has achieved remarkable success in deformable image registration, yet the visual information that drives deformation estimation remains poorly understood. Rather than pursuing incremental performance improvements, this work investigates the fundamental source of robustness in deep registration models. Using diverse, domain-agnostic synthetic datasets, we decouple deformation learning from application-specific appearance and show that domain-shift immunity is an inherent, largely architecture-agnostic property of deep de-formable registration when trained with a robust pipeline. To identify the mechanism underlying this immunity, we compare models trained directly on raw image intensities with models operating exclusively on local feature representations extracted by a fixed, pre-defined feature extractor. The comparable performance of these models provides strong empirical evidence that deformation estimation is governed primarily by local structural features, rather than global, domain-specific appearance cues. These findings offer a principled explanation for the cross-domain generalizability of deep registration networks and point toward feature-centric designs for domain-independent registration.
Mingzhen Shao, Sarang C. Joshi· International Conference on...· 0 citations