vToP: VLM-Guided Transparent Object Perception for Robotic Manipulation

Abstract

Transparent objects, such as beakers, flasks, and other laboratory glassware, are commonly used in both laboratory and industrial settings. When light passes through these materials, it is refracted, which degrades image quality and impairs depth estimation of object surfaces. This often leads to failures in robotic perception of transparent targets, reducing task efficiency and potentially creating safety hazards. To address this problem, we propose a vision–language-guided Transparent Object Perception (vToP) method that combines cognitive augmentation with confidence-based depth completion. First, using structural priors and cognitive feedback from vision–language models, transparent objects are redesigned as deformable models, allowing them to adaptively deform and align with observed object contours. Next, regional confidence scores are designed to evaluate depth quality for each pixel and are then used as weighting factors to dynamically fuse depth estimation with cognitive 3D modeling, recovering missing depth information of transparent objects. Finally, we evaluate vToP on the public TransCG dataset and a newly constructed synthetic dataset, CSSG, designed for transparent object perception. Experimental results demonstrate that vToP significantly outperforms existing methods in transparent object depth recovery. Furthermore, vToP can be seamlessly integrated with grasp detection algorithms, and real-world experiments on transparent objects using a UR5 robotic arm validate its effectiveness. Note to Practitioners—Transparent objects are commonly encountered in laboratory and industrial settings. However, conventional RGB-D cameras often produce inaccurate depth estimates when capturing transparent or highly reflective objects. This limitation leads to failures in object recognition, grasping, and manipulation, as well as potential safety hazards. Therefore, a reliable solution for transparent object perception is critically important. The proposed method, vision–language-guided Transparent Object Perception (vToP), is inspired by biological visual cognition. It identifies and completes uncertain depth regions by integrating structural priors with cognitive feedback from vision–language models, thereby significantly improving the accuracy of transparent object perception. Experimental results demonstrate that vToP not only enhances depth recovery but also achieves high success rates in real-world grasping tasks, highlighting its strong potential for applications in intelligent laboratories and other automated environments.