Skip to content

Structured Latent Modeling for Supervised Multimodal Information Decomposition

Sep 2026 · arXiv.org · Vol abs/2609.35502 · 0 citations · 55 references
Computer Science

TL;DR

This work introduces a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model, offering a practical latent-variable lens for characterizing continuous multimodal interactions.

Abstract

Multimodal prediction relies on diverse forms of evidence: information repeated across modalities, cues specific to a single source, and complex cross-modal dependencies that emerge only when inputs are considered together. While recent methods promote richer interactions, they lack a principled way to isolate these target-relative contributions within learned continuous representations. We introduce a framework that applies contrastive or masked objectives at intermediate layers, coupled with source-wise invertible normalizing flows and a supervised, low-rank latent variable model. This architecture explicitly factorizes the joint distribution into shared task-relevant variation, modality-specific predictive variation, and task-irrelevant dependence. Drawing connections to prior multimodal learning assumptions, our approach evaluates how modalities independently and jointly contribute to the target. Ultimately, this framework unites intermediate representation learning with structured likelihood-based guidance, offering a practical latent-variable lens for characterizing continuous multimodal interactions. Empirically, we demonstrate the effectiveness of our approach across diverse multimodal benchmarks, showing robust improvements in predictive performance.

View source

Similar papers

Preprint Aug 2026

Adaptive Hierarchical Representation Alliance for Multimodal Learning

Adaptive Hierarchical Representation Alliance (AHRA), a hierarchical shared--private expert framework, is proposed, which consistently improves over strong baselines and remains robust under noisy and missing-modality settings.

Chun-Lei Meng, Peng-Bin Feng, Jacqueline J. Pang et al. · 2 citations
Open access 2026

When does heterogeneous structured evidence help? A unified benchmark for structured-data modeling

The introduction of Trimodal Structured-Data Benchmark (TSDBench), an entity-centric benchmark for evaluating multimodal structured-data learning, highlights the need for more adaptive multimodal learning methods capable of selectively leveraging heterogeneous structured evidence.

Kai-Jie Cheng, Nuo-Yue Xu, Xing-Xuan Zhang et al. · 0 citations
#machine learning Preprint Sep 2026

InfluenceField: A Differentiable Field with Interventionally Identifiable Causal Structure for Multimodal World Modeling

InfluenceField, an intervention-aware latent field inserted between the visual encoder and language decoder, is introduced, an intervention-aware latent field inserted between the visual encoder and language decoder that lifts patch features into a continuous spatial representation, propagates directed influence over m...

Zi-Hao Yang, Zi-Jian Wang, Zhi-Qiu Huang · 0 citations
Preprint Sep 2026

RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertainty

Multimodal prediction from images and structured metadata requires integrating complementary evidence whose reliability can vary across samples and latent directions. A single confidence weight per modality cannot capture this directional variation. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fus...

Yin Xu, Tie-Ming Liu, Ye Liang · 2 citations

SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals

Multimodal contrastive learning is a dominant paradigm for learning transferable representations from unlabeled data, but standard objectives primarily capture information that is redundant between modalities. Partial Information Decomposition (PID) shows that task-relevant information in multimodal data decomposes int...

Yavuz Yarici, Ghassan AlRegib · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.