Skip to content
Preprint

Exploring 2D backbone effects for indoor semantic occupancy prediction

Sep 2026 · 0 citations · 20 references
Computer Science

TL;DR

The results indicate that the image backbone is not a secondary engineering detail in embodied semantic occupancy, but a major source of variation in the final 3D prediction.

Abstract

Semantic occupancy prediction gives an embodied agent a voxel-level account of where space is free, occupied, and semantically meaningful. In RGB-D pipelines such as EmbodiedScan, the image encoder is often left as a default module, even though its features are the visual evidence later sampled into the 3D grid. We study this design choice directly. A central finding is that changing the 2D backbone improves occupancy accuracy more than several carefully designed occupancy architectures or modules. We keep the main RGB-D projection, depth branch, and occupancy head fixed, and replace only the image backbone. The compared encoders are CLIP-ResNet, CLIP-ViT, BLIP2, and DINOv2. Under the controlled setting, the measured mIoU changes substantially: DINOv2 obtains 30.55\%, BLIP2 obtains 29.49\%, CLIP-ViT obtains 24.33\%, and CLIP-ResNet obtains 17.41\%. The stronger encoders also exceed the original EmbodiedScan ResNet-50 baseline without modifying the downstream 3D fusion pipeline. Class-level results give a more detailed picture: DINOv2 is stronger on many layout and structural categories, whereas BLIP2 remains close on several object-centered classes. CLIP-ViT improves clearly over CLIP-ResNet, showing that the way CLIP features are exposed as dense tokens matters for voxel lifting. These results indicate that the image backbone is not a secondary engineering detail in embodied semantic occupancy, but a major source of variation in the final 3D prediction.

View source

Similar papers

Preprint Sep 2026

AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks

Embodied tasks demand accurate, flexible, and semantically rich 3D scene representations. 3D semantic occupancy is well suited to this requirement, as it can model holistic 3D spaces by encoding geometric occupancy along with semantic categories. However, existing occupancy prediction methods struggle to meet practical...

Jing-Long Wang, Yun-Jie Wang, Zhi-Yang Zhang et al. · 0 citations
Preprint Aug 2026

OccAnyScene: Towards Unified Indoor-Outdoor 3D Occupancy Prediction

OccAnyScene is proposed, a pixel-frustum-centered Gaussian framework built upon a pretrained depth foundation model which employs Pixel-Aligned Frustum Feature Aggregation to construct a camera-aware frustum query for each feature pixel, and Frustum-Parameterized Gaussian Construction to decode each query into multiple...

Junjie Liu, Wan-Shui Gan, Zi-Tong Dai et al. · 0 citations
Conference Aug 2026

Enhancing 3D semantic scene completion via efficient attention and feature augmentation

A 3D Local- Global Linear Attention Mechanism (LG-LAM) is devised that efficiently captures long-range contextual information with linear complexity, enabling a comprehensive understanding of the 3D scene without heavy computational burdens.

Jie Li, Jia-Heng Xu, Laiyan Ding et al. · 0 citations
Open access Aug 2026

Efficient 3D Semantic Occupancy Prediction via Integrated 2D-3D Feature Fusion

This paper presents a novel approach to 3D semantic occupancy prediction that leverages the integration of 2D and 3D features. Traditional 3D voxel representations, while detailed, are computationally intensive. Our method addresses this challenge by encoding 3D voxel features into a Bird’s-Eye View (BEV) representatio...

Sang-Min Park, Jong-Eun Ha · 0 citations
#artificial intelligence Preprint Sep 2026

VOIM: Training-Free Open-Vocabulary 3D Instance Mapping for RGB-D and Monocular SLAM

We present Voxel-Grounded Online Instance Manager (VOIM), a training-free voxel-grounded instance manager that builds open-vocabulary 3D instance maps from RGB-D or from monocular RGB alone, a regime no prior training-free system addresses. Online systems typically segment object instances and label them at first detec...

Sangmim Song, S. Kodagoda, Marc G. Carmichael et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.