Skip to content

DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.24976 · 1 citation · 31 references
Computer Science

TL;DR

A visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling is presented.

Abstract

Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26x faster training and 1.29x faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.

View source

Similar papers

Dexterous Tactile World Model

World models for manipulation are typically trained from video, yet the events that determine how manipulation unfolds, such as making and releasing contact, are difficult to observe visually and are often easier to sense through touch. We present the Dexterous Tactile World Model (DTWM), a video world model for future...

Zi-Yao Zeng, Xiatao Sun, Hao Wang et al. · 1 citation

TacSushi: Tactile-Grounded World-Action Modeling for Dexterous Sushi Manipulation

Comparisons support complementary benefits of feature-wise gated tactile fusion and training-only predictive supervision of TacSushi, a tactile-grounded, Cosmos3-based world-action policy that learns from recorded future consequences while acting on current observations.

Hao-Di Hu, Kaen Kogashi, T. Koike-Akino · 0 citations
Preprint Sep 2026

STAR: Sparse Tactile Representation Learning in Vision-Tactile-Language-Action Models for Dexterous Manipulation

Dexterous manipulation requires coordinated multi-finger control and effective tactile feedback, yet learning these capabilities remains challenging due to the lack of large-scale real-world data and the difficulty of extracting effective representations from sparse tactile signals. We build a robot platform and teleop...

Xiang-Cheng Liu, Tian-Hao Wu, Le Zheng et al. · 1 citation
#artificial intelligence Preprint Sep 2026

DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination

DeCAL is presented, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation and introduces Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy.

Yan-Kai Fu, Ning Chen, Jun-Kai Zhao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation

A Unified Visual-Tactile-Action Model is proposed that maps both embodiments into aligned tactile and action representations and jointly predicts future action and tactile trajectories and achieves an average success rate of 70%, outperforming the strongest visual-tactile baseline, which achieves 29%, and an architectu...

Wen-Qiao Li, Qian-You Zhao, Jia-Wen Hao et al. · 0 citations
Preprint Sep 2026

DexTouch-WM: Learning Action-Conditioned Tactile World Models from Human Touch for Dexterous Robot Manipulation

Learning predictive models of contact-rich dexterous manipulation requires dense tactile interaction, but such data are costly to scale on real robots and remain tied to embodiment-specific sensors. We introduce DexTouch-WM, an action-conditioned world model that learns from scalable human touch to jointly predict futu...

Yan-Jiao Qin, Yue Chen, Wen-Wei Lin et al. · 3 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.