Skip to content

Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders

Aug 2026 · 0 citations · 23 references
Computer Science

TL;DR

The results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions as separate questions in evaluating multimodal physical representations.

Abstract

Multimodal world models are often evaluated by whether different sensors produce similar representations. However, similar representations do not necessarily imply that the models make the same physical predictions, or that those representations can be reused when actions are combined in a new order. We study both questions through the physical responses predicted by a model. We first use the Cluster Haptic dataset to ask whether audio and acceleration can independently recover the behavior of the same surface from different observations. Predictions from the two sensors are substantially closer for the same surface than for different surfaces, with a $4.5\times$ gap on average, while both also outperform an average-surface prediction. We then show that this agreement alone does not determine how familiar actions should compose. In a controlled elastoplastic system, shared step dynamics fit observed programs less accurately than a whole-program predictor but generalize better to unseen action orders, with the ranking reversing on both held-out transitions across three independent initializations. Fusing free-decay and hysteresis observations further improves prediction, with diagonal Gaussian beliefs yielding the lowest errors. Together, these results distinguish cross-sensor consistency, multimodal fusion, and generalization to new action orders as separate questions in evaluating multimodal physical representations.

View source

Similar papers

Open access Aug 2026

Explainable Decoding of Sensorimotor Communication in Joint Object Manipulation

Humans often collaborate under asymmetric information, for example when two people carry a table and only one knows the destination. They coordinate without speech using cues from movement kinematics, interaction forces, and object states. Characterizing this sensorimotor communication is difficult because these signal...

Yiming Liu, D. Verdel, Raz Leib et al. · 0 citations
Sep 2026

The Planning Limits of Latent World Models

This work studies action-conditioned predictors built on five frozen self-supervised visual backbones, finding that a world model guides action selection reliably only when the goal lies within, or slightly beyond, the trajectory it imagines during planning.

A. Alrasheed, Basim Azam, Naveed Akhtar · 3 citations
Preprint Aug 2026

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

RididBench is introduced, a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description, with per-frame masks, depth, 6-DoF trajectories, and contacts available for scoring.

Swarnim Jain, Shang-Zhe Wu · 5 citations

Dexterous Tactile World Model

World models for manipulation are typically trained from video, yet the events that determine how manipulation unfolds, such as making and releasing contact, are difficult to observe visually and are often easier to sense through touch. We present the Dexterous Tactile World Model (DTWM), a video world model for future...

Zi-Yao Zeng, Xiatao Sun, Hao Wang et al. · 1 citation

Learning Expressive and Compositional Motion Representation via Spectral Skills

Robotic foundation models offer a promising path toward general-purpose humanoid robot control, often through hierarchical architectures. However, their effectiveness depends on the command interface between the planner and the controller, which must support accurate execution while remaining easy to predict, and ideal...

Fei-Yang Wu, Chen-Xiao Gao, Chen Yang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.