Multimodal Beam Prediction With Missing Modality Adaptability: A Contrastive Learning Approach

Abstract

Multimodal beam prediction based on deep learning can significantly reduce the beam training overhead in millimeter-wave multiple-input multiple-output (MIMO) systems. However, the performance of existing multimodal approaches is often degraded by missing modalities, and their intricate modality fusion mechanisms typically incur high inference complexity. To address these challenges, a novel contrastive learning approach that inherently accommodates arbitrary patterns of missing modalities and enables low-complexity inference is proposed in this paper. First, a unified latent space is constructed through contrastive learning, which aligns the unimodal embeddings with a set of learnable candidate beam embeddings. This alignment facilitates simple yet effective modality fusion by averaging the embeddings from available modalities, as well as low-complexity beam selection via similarity search. Then, to establish a semantically meaningful initial structure within the latent space, we leverage the inter-beam correlations to initialize the candidate beam embeddings, thereby guiding the pretraining of unimodal encoders. Finally, the standard information noise contrastive estimation loss is generalized by incorporating the relevance among training samples and candidate beams to provide richer supervisory signals for model training. Simulation results demonstrate that the proposed approach can establish a semantically meaningful latent space, and achieve superior performance and lower complexity compared to existing approaches in multimodal beam prediction with missing modalities.