Skip to content
Open access

SIRModel: Learning Spatial Intermediate Representation to Parameter-Efficiently Fine-Tune a Vision Language Model for Manipulation

Aug 2026 · Applied Informatics · Vol 7, pp. 334 · 0 citations · 48 references

TL;DR

A spatial Gaussian-guided hierarchical framework that uses ordered 3D Gaussian interaction regions as an explicit planning interface between vision–language reasoning and action generation and demonstrates the effectiveness of explicit 3D interaction representations for spatial reasoning and long-horizon manipulation.

Abstract

Long-horizon robotic manipulation requires a policy to bridge task-level semantic reasoning with metric three-dimensional interaction geometry. Existing vision–language–action policies usually acquire geometry implicitly from visual tokens or introduce deterministic intermediate variables only in the image plane, which rely on expensive human annotations and training cost. This article presents a spatial Gaussian-guided hierarchical framework that uses ordered 3D Gaussian interaction regions as an explicit planning interface between vision–language reasoning and action generation. The proposed framework enables efficient adaptation of a pretrained vision–language model for robotic manipulation tasks. First, an automatic geometric enhancement pipeline converts raw robot demonstration videos into near-, mid-, and late-stage Gaussian supervision through foreground extraction, metric depth estimation, stable camera aggregation, end-effector localization, 3D lifting, and temporal grouping, without requiring manual 3D interaction annotation. The generated Gaussian representations provide structured spatial guidance, where their covariance characterizes interaction-region extent and variability rather than fully calibrated physical uncertainty. Second, a shared vision–language backbone predicts structured subtasks and Gaussian interaction regions, while a conditional diffusion executor generates future action chunks under these semantic and geometric conditions. A trajectory-to-Gaussian likelihood objective explicitly encourages consistency between generated motions and the predicted spatial interaction plan. Experiments on a mixed real-robot dataset derived from LHManip and RH20T show that our method improves trajectory tracking success from 55.7% to 70.8% over a same-backbone direct VLA baseline. Closed-loop simulation evaluation on LIBERO with 80% backbone parameter frozen achieves 85.3% average task success, demonstrating the effectiveness of explicit 3D interaction representations for spatial reasoning and long-horizon manipulation.

Read PDF

Similar papers

Preprint Nov 2026

SLA: Scaling Spatial-Language-Action Model via Continual Pre-training

Reliable robotic manipulation requires both semantic understanding and precise control in three-dimensional space. However, vision-language-action (VLA) foundations largely inherit visual representations from 2D vision-language pre-training, while spatial supervision is often introduced only during task-specific adapta...

Fu-Hao Li, Wen-Xuan Song, Qi-Zhou Huang et al. · 0 citations
Sep 2026

"Dear LLaVA, Please Drive": A Depth-Aware Vision-Language Agent for Closed-Loop Robotic Control

This work proposes a parameter-efficient approach to fine-tune a pretrained VLM for autonomous navigation using an Imperative Learning paradigm, and introduces a unified end-to-end navigation pipeline for natural-language-driven robotic control.

Sebastian Berger, Katharina Winter, Fabian B. Flohr · 0 citations
Preprint Oct 2026

StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation

Vision-and-Language Navigation (VLN) policies increasingly benefit from strong semantic priors provided by large vision-language models (VLMs). However, standard action supervision does not explicitly encourage intermediate representations to preserve scene geometry, relative orientation, or global episode progress. In...

Anh Dao, Q. Phạm, Le Danh Vinh et al. · 0 citations
Sep 2026

MachEmbodied-U0: Unified Understanding and Generation Model for Embodied Intelligence

MachEmbodied-U0, a unified embodied foundation model connecting understanding and generation experts through a Mixture-of-Transformers architecture, is presented, a unified embodied foundation model connecting understanding and generation experts through a Mixture-of-Transformers architecture that combines competitive...

Hao-Ran Wen, Wen-Fu Wang, Kun-Song Shi et al. · 1 citation
Preprint Oct 2026

Relational Spatial Alignment for Vision-Language-Action Models

This work proposes a representation-alignment method that enables the injection of 3D spatial awareness without compromising the model's original expressivity, and effectively incorporates 3D spatial representations into VLA reasoning, achieving robust performance and enhanced spatial awareness across diverse manipulat...

Chao-Ran Zhu, Hengyi Wang, Cheng-Lin Cui et al. · 0 citations

Acting in Meters: Learning Metric Interactions for Precise Robotic Manipulation

A metric interaction framework that models object-level and scene-level interactions in physical Cartesian space at a shared metric scale and improves diverse VLA and WAM baselines with a small number of additional parameters and training steps is introduced.

Li-Jie Wang, Zheng Lu, Yi-Ming Wang et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.