Multimodal Deep Learning For Eye Disease Recognition From Fundus And OCT Images With Synthetic OCT Generation
TL;DR
This study synthesizes 3D OCT from fundus images using three generative paradigms using three generative paradigms-a 3D GAN (DCGAN/WGAN-GP style), a 3D VAEGAN, and a fundus-conditioned latent diffusion model-and integrates the generated volumes into an EyeMoST- based uncertainty-aware fusion framework.
Abstract
Multimodal learning that integrates Color Fundus Photography (CFP) and Optical Coherence Tomography (OCT) can enhance retinal disease recognition by combining surface appearance with depth-resolved structural cues. However, the limited availability of synchronized 2D-3D CFP-OCT datasets hinders the deployment of multimodal AI in real-world screening settings where OCT devices are often unavailable. This study investigates whether synthetic 3D OCT volumes can serve as a practical auxiliary modality to support multimodal classification in fundus-only scenarios. We synthesize 3D OCT from fundus images using three generative paradigms-a 3D GAN (DCGAN/WGAN-GP style), a 3D VAEGAN, and a fundus-conditioned latent diffusion model (3D-DDPM with classifier-free guidance)-and integrate the generated volumes into an EyeMoST- based uncertainty-aware fusion framework. We evaluate the pipeline on RFMiD and ODIR-5K across multiple backbone configurations. Empirically, diffusion-generated OCT yields the most stable fusion performance. In contrast, synthetic OCT does not consistently outperform strong fundus-only models due to domain gap and imperfect biomarker preservation.