Modeling Scientific Experiment Scenes: Dataset and Model

SPOT: Structured Prompting with Object-centric Tokens for open-world scene graphs

Foundation Models based Scene Graph Generation.

Scene Graph Generation (SGG) is an essential task at the interface of computer vision and natural language processing, aimed at creating structured semantic representations of visual scenes. However, traditional paradigms that train from scratch with SGG datasets face persistent issues, including model bias due to long-tail distributions and difficulties with zero-shot triplet recognition due to sparse annotations. Recently, Foundation Models (FMs), pretrained on extensive datasets, have demonstrated remarkable success in various downstream tasks, showcasing their potential in enhancing SGG task. Consequently, we propose a FMs based SGG (FMSGG) framework, leveraging the broad pretraining of FMs to benefit tail relationships and zero-shot triplets. FMSGG addresses three key challenges when incorporating FMs into SGG task: inefficient training and inference, difficulty in distinguishing similar relationships, and dual-bias. Specifically, our method involves isolating relationship embeddings from image features obtained from FMs via relationship prompts, and augmenting text prompts to enhance the distinctiveness of relationship description embeddings. Additionally, we introduce the triplet-aware logit adjustment method to mitigate long-tail bias, and propose adaptive evidential smoothing to specifically alleviate inherent biases in FMs. Experimental results across various evaluation modes and metrics indicate that FMSGG achieves state-of-the-art mean recall rate and zero-shot recall rate, while also providing an optimal balance between mean recall rate and recall rate.

AMCA-SGG: Adaptive multi-prototype learning and cross-modal alignment for unbiased scene graph generation

Scene graph generation (SGG) aims to detect objects and recognize their semantic relationships from images, serving as an important bridge between visual perception and high-level semantic understanding. Existing debiasing methods primarily address inter-class long-tailed bias, while paying insufficient attention to intra-class semantic heterogeneity of predicates. High-frequency predicates typically encompass multiple semantic sub-patterns, causing instances of the same predicate to scatter widely in the visual feature space and resulting in a fundamental misalignment between feature distributions and classification objectives. To address these issues, we propose AMCA, an unbiased SGG framework integrating adaptive multi-prototype learning with cross-modal alignment. Specifically, the Multi-Prototype Construction (MPC) module employs a distribution-aware prototype allocation strategy to quantify the semantic distributional complexity of each predicate and dynamically assign prototype counts, explicitly modeling intra-class semantic heterogeneity. Building on this, the Hybrid Relation Refinement Module (HRRM) embeds semantic prototypes into the visual relation feature refinement process via a prototype-augmented attention mechanism, and further employs Mamba to model global contextual dependencies among different relation representations, achieving deep fusion of semantic priors and visual features. Furthermore, the Visual-Text Alignment (VTA) module aligns the fused visual relation representations with CLIP text embeddings, significantly enhancing the model’s ability to distinguish fine-grained semantics. Extensive experiments on Visual Genome and GQA demonstrate that AMCA achieves consistently competitive performance across multiple SGG tasks, with particularly strong improvements on the unbiased mR@K metric. Our code is publicly available at https://github.com/SJVision/AMCA.

SG-Layout: Structured Scene Graph-Guided Layout Generation with LLMs

Understanding and generating spatially coherent layouts from natural language remains a fundamental yet challenging task for large language models (LLMs). Existing LLMs often struggle to capture explicit geometric relationships and structural dependencies between objects. To address this issue, we propose SG-Layout, a graph-guided layout generation framework that explicitly incorporates structured spatial knowledge into LLMs. SG-Layout follows a two-stage training paradigm: (1) a graph-language feature alignment stage, where a relational graph encoder and a projector are trained to map scene-graph embeddings into the LLM's linguistic space; and (2) an instruction tuning stage, where LoRA-based adapters enable efficient fine-tuning for instruction-driven layout generation while keeping the backbone frozen. We evaluate SG-Layout on image layout generation, indoor scene synthesis and robotic object rearrangement tasks. Experimental results show that SG-Layout improves spatial reasoning accuracy and geometric consistency over the compact open-source backbone, with particularly clear advantages in relation-dense and compositionally complex scenes. These results highlight the effectiveness of graph-structured feature alignment for enhancing controllable layout generation.

Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

This work introduces Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations and proposes node-as-proxy graph rewards to consolidate efficient graph exploration.

LEGO: Leveled Language Gaussian Splatting

We introduce LEGO for advanced open-vocabulary scene understanding. Beyond basic concept recognition, its core innovation lies in capturing the intrinsic semantic hierarchies within the scene, such as the"flowerpot ->bouquet ->bud ->petal"lineage. While foundation models like SAM can identify multi-granular structures in 2D, their partitions are strictly perspective-bound and lack cross-view consensus. LEGO self-adaptively re-grades volatile multi-view SAM granularities into a unified, 3D-consistent hierarchy. This provides precise supervision for the structurally coherent, multi-level segmentation of 3D scenes. By grounding these segments with CLIP embeddings, LEGO recovers open-vocabulary semantic logic across hierarchical levels. Furthermore, by incorporating spatial relationships, we elevate these segments into level-wise language scene graphs, effectively empowering Large Language Models to perform complex, context-aware spatial reasoning and precise visual grounding. Experimental results demonstrate that LEGO establishes new state-of-the-art performance across both promptable and open-vocabulary 3D segmentation benchmarks, exhibiting advanced hierarchical scene decomposition and context-aware spatial reasoning.