This work introduces a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model, offering a practical latent-variable lens for characterizing continuous multimodal interactions.
Abstract
Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.
Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework, is proposed, which consistently improves over strong baselines and remains robust under noisy and missing-modality settings.
Chun-Lei Meng, Peng-Bin Feng, Jacqueline J. Pang et al.· 2 citations
The introduction of Trimodal Structured-Data Benchmark (TSDBench), an entity-centric benchmark for evaluating multimodal structured-data learning, highlights the need for more adaptive multimodal learning methods capable of selectively leveraging heterogeneous structured evidence.
Kai-Jie Cheng, Nuo-Yue Xu, Xing-Xuan Zhang et al.· AI Plus· 0 citations
InfluenceField, an intervention-aware latent field inserted between the visual encoder and language decoder, is introduced, an intervention-aware latent field inserted between the visual encoder and language decoder that lifts patch features into a continuous spatial representation, propagates directed influence over m...
Multimodal prediction from images and structured metadata requires integrating complementary evidence whose reliability can vary across samples and latent directions. A single confidence weight per modality cannot capture this directional variation. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fus...
MUNITE achieves the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.
Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes int...
Yavuz Yarici, Ghassan AlRegib· arXiv.org· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026