Bi-SGL is proposed, a Bidirectional, Spatially Grounded, and Language-informed framework for point cloud semantic scene completion that improves semantic completion accuracy among evaluated point cloud SSC baselines while using substantially fewer parameters than cascaded dense-fusion models.
Abstract
Semantic scene completion (SSC) requires a model to infer both the geometry and semantic labels of a complete 3D scene from partial point cloud observations. Recent point cloud SSC methods improve efficiency over dense volumetric formulations, yet progressive point cloud decoders must still resolve three coupled sources of ambiguity: local spatial context around each generated point, information exchange between geometric refinement and semantic prediction, and semantic relations among visually or structurally similar scene classes. We propose Bi-SGL, a Bidirectional, Spatially Grounded, and Language-informed framework for point cloud semantic scene completion. Bi-SGL integrates three complementary signals within a stagewise semantic-geometric decoder. Spatially-Aware Hierarchical Geometric Grounding (SAHGG) aggregates multi-resolution encoder features into a pointwise spatial grounding signal. Bidirectional Branch Coupling (BBC) enables geometry and semantics to exchange information during progressive refinement. Class-Similarity Adaptive Bias (CSAB) introduces language-derived class priors from frozen OpenCLIP text embeddings: a text-seed prior preserves task-specific class identities while injecting class-level semantics, and a class-relation prior biases semantic decoding with inter-class similarities. Experiments on SSC-PC and NYUCAD-PC show that Bi-SGL improves semantic completion accuracy among evaluated point cloud SSC baselines while using substantially fewer parameters than cascaded dense-fusion models. On SSC-PC, Bi-SGL improves mIoU from 90.71% to 92.92% over ProtoFormer while reducing CD from 8.917 to 8.672. On NYUCAD-PC, it achieves 51.46% mIoU, outperforming ProtoFormer by 2.27 percentage points and CasFusionNet by 2.13 percentage points. Ablation and class-level analyses indicate that spatial grounding, branch coupling, and language-derived class priors make complementary contributions to point cloud SSC.
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Xianhao Chen, Jiarui Hu, Yuanbo Yang et al.· 0 citations
Scene graph generation (SGG) aims to detect objects and recognize their semantic relationships from images, serving as an important bridge between visual perception and high-level semantic understanding. Existing debiasing methods primarily address inter-class long-tailed bias, while paying insufficient attention to intra-class semantic heterogeneity of predicates. High-frequency predicates typically encompass multiple semantic sub-patterns, causing instances of the same predicate to scatter widely in the visual feature space and resulting in a fundamental misalignment between feature distributions and classification objectives. To address these issues, we propose AMCA, an unbiased SGG framework integrating adaptive multi-prototype learning with cross-modal alignment. Specifically, the Multi-Prototype Construction (MPC) module employs a distribution-aware prototype allocation strategy to quantify the semantic distributional complexity of each predicate and dynamically assign prototype counts, explicitly modeling intra-class semantic heterogeneity. Building on this, the Hybrid Relation Refinement Module (HRRM) embeds semantic prototypes into the visual relation feature refinement process via a prototype-augmented attention mechanism, and further employs Mamba to model global contextual dependencies among different relation representations, achieving deep fusion of semantic priors and visual features. Furthermore, the Visual-Text Alignment (VTA) module aligns the fused visual relation representations with CLIP text embeddings, significantly enhancing the model’s ability to distinguish fine-grained semantics. Extensive experiments on Visual Genome and GQA demonstrate that AMCA achieves consistently competitive performance across multiple SGG tasks, with particularly strong improvements on the unbiased mR@K metric. Our code is publicly available at https://github.com/SJVision/AMCA.
Jinhao Fan, Yuanhao Xi, Chuanping Hu et al.· Journal of King Saud Univers...· 0 citations
Semantically-guided progressive network (SGP-Net) is proposed, a semantically guided progressive refinement framework for MDE based on multi-task learning that improves key relative-error and accuracy metrics over the DCDepth baseline and remains competitive with recent methods.
Henan Hu, Xu Cheng, Ronghua Li et al.· Measurement science and tech...· 0 citations
Driving World Models (DWMs) have recently advanced rapidly with generative models, yet most existing methods mainly focus on conditional scene generation and lack explicit 3D scene understanding, language-grounded reasoning, and controllable 4D editing capabilities. Moreover, commonly used point cloud, occupancy, or BEV representations make it difficult to achieve fine-grained alignment between textual information and the underlying 3D scene structure. To address these limitations, we propose a foundation-feature Gaussian driving world model that unifies scene understanding, language-grounded reasoning, controllable 4D editing, and multi-modal generation within a single framework. Specifically, we introduce a foundation-feature Gaussian tokenizer that directly distills Qwen/SigLIP visual-language features into 3D Gaussian primitives, building a compact open-vocabulary Gaussian semantic field. We further design a geometry-aware Gaussian adapter that combines importance-aware hierarchical selection with text-conditioned Perceiver-style cross-attention to aggregate dense Gaussian primitives into compact world tokens. To improve representation compatibility, we introduce a KL-based Gaussian--image distribution alignment objective that aligns Gaussian world tokens with foundation image tokens. Based on the aligned Gaussian representation, our framework further supports instruction-controllable scene editing, including weather-conditioned generation and dynamic vehicle manipulation. Extensive experiments on broader driving benchmarks demonstrate that our method achieves state-of-the-art performance across scene understanding, visual grounding, planning-oriented reasoning, and controllable 4D generation tasks. We will release the code and datasets publicly on Github.
Tianchen Deng, Xuefeng Chen, Shuang Wu et al.· 0 citations
Achieving unified scene parsing that simultaneously outputs cross-domain semantic segmentation and depth estimation without scene-specific retraining is crucial for robust perception in complex real-world environments, yet remains a challenging goal. While recent monocular depth estimation models such as DepthAnything V2 exhibit strong domain generalization, semantic segmentation still suffers from severe structural degradation under domain and viewpoint shifts. We observed a persistent hierarchical calibration gap, where Euclidean representations exhibit larger calibration gaps between child and parent categories under domain shifts, suggesting limitations of existing Euclidean-based methods in preserving semantic hierarchies. To address this issue, we propose HyperMapper, a hyperbolic structure-aware mapping framework that bridges semantic understanding and geometric priors through hyperbolic token-to-feature interactions. By exploiting the negative curvature of hyperbolic space, HyperMapper helps capture hierarchical relationships and maintains geometric consistency across domains. Furthermore, by combining the expressive priors of vision foundation models (VFMs) with parameter-efficient fine-tuning (PEFT), HyperMapper achieves cross-domain adaptation with minimal trainable parameters in backbone while retaining the strong depth estimation capability of DepthAnythingV2 without retraining. Extensive experiments on multiple cross-domain and cross-viewpoint benchmarks demonstrate that HyperMapper achieves a higher mIoU for both parent and child categories while consistently improving segmentation accuracy over strong baselines. Our approach establishes a promising direction for task-preserving dual-task adaptation, bridging semantic and geometric learning and paving the way toward unified, cross-domain scene parsing.
Jiawei Liu, Da Yang, Tingwei Feng· IEEE Transactions on Neural...· 0 citations
Outdoor LiDAR semantic scene completion (SSC) recovers a dense semantic voxel grid from a scan observing 1% of the target volume, under class imbalance beyond 7,000x. We recast SSC as generative semantic scene completion (GSSC): a single discrete-diffusion formulation in three roles. First, paired sparse-dense scene synthesis (PS$^3$) generates matched sparse LiDAR observations with their dense semantic completions, addressing the long tail at its source and yielding the PS$^3$-SemanticKITTI corpus we train on alongside SemanticKITTI. Second, semantic-guided generative scene completion (SGSC) generates the scene from noise with multinomial discrete diffusion, conditioned on the sparse scan through a bird's-eye-view semantic map and a sparse 3D feature stream. Third, the same framework instead refines an existing completion in one flow-matching step: structured source discrete diffusion (S$^2$D$^2$). S$^2$D$^2$ improves the mIoU of SGSC's own output and every external SSC base tested, without base retraining or test-time adaptation. On the strongest base, one step without test-time augmentation reaches 38.8% mIoU on the SemanticKITTI hidden test. To our knowledge that is the best causal, single-sweep, single-sample result on that leaderboard, +2.1 pp over the previous best published score under the same restriction. Four correction steps with eight-view test-time augmentation reach 39.2%, outside that restriction.