Jul 2026· Frontiers in Robotics and AI· Vol 13· 0 citations· 39 references
Medicine
TL;DR
This work presents a novel integration of detector-based visual representations with conditional diffusion modeling (DINO + CDP) for real-world robotic imitation learning and demonstrates that object-query-conditioned diffusion significantly improves task success rates, produces smoother trajectories, and exhibits superior robustness to high-entropy visual inputs, establishing a scalable pathway for imitation learning in challenging agricultural domains.
Abstract
Imitation learning in complex, unstructured environments remains challenging due to the difficulty of grounding perception in physically meaningful representations and the need to model multimodal action distributions. Existing approaches often rely on unstructured pixel-level feature encodings or stochastic latent-variable decoders, which can lead to brittle attention in cluttered scenes. In this work, we present a novel integration of detector-based visual representations with conditional diffusion modeling (DINO + CDP) for real-world robotic imitation learning. Our framework utilizes a DINO object detection transformer to extract spatially-grounded object-query embeddings that serve as the conditioning signal for a diffusion-based policy. A primary contribution of this work is the systematic quantification of how scene complexity—measured via image entropy—affects robotic policy performance. By comparing rigid-object baselines with complex biological plant scenes, we demonstrate that organic morphology induces a measurable increase in pixel-level uncertainty that degrades standard pixel-centric models. Our results show that DINO + CDP mitigates this degradation by grounding action generation in stable object-level features. We evaluate our approach using a fully real-world manipulation dataset collected without simulation or synthetic pre-training. To isolate the impact of our architectural choices, we conduct a comparative study within a unified framework against convolutional (CNN-MLP), transformer-patch (ViT), and latent-variable (DETR + CVAE) variants. Experimental results in a robotic-arm biocell setup demonstrate that object-query-conditioned diffusion significantly improves task success rates, produces smoother trajectories, and exhibits superior robustness to high-entropy visual inputs, establishing a scalable pathway for imitation learning in challenging agricultural domains.
This work proposes a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations, and renders scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations.
A. Tekden, Yasemin Bekiroglu· IEEE Robotics and Automation...· 0 citations
Diffusion Policy has emerged as a powerful approach for imitation learning in robotic manipulation. However, policies trained solely on RGB observations often fail to capture the spatial structure of a scene, which limits performance on tasks requiring precise distance-aware motions, such as approaching an object along the camera axis. A natural remedy is to provide depth as an additional input, but two practical challenges remain: designing a conditioning mechanism that encourages the policy to use depth effectively and obtaining reliable depth when hardware measurements are unavailable or noisy. This paper presents Depth-Augmented Diffusion Policy, a framework that addresses both issues. The method employs separate visual encoders for RGB images and depth maps and fuses their representations through stochastic feature-level addition with modality dropout, reducing modality dominance and promoting the use of geometric cues during action generation. In addition, the results demonstrate that pseudo-depth generated by the pre-trained Depth Anything V2 estimator can replace sensor depth, removing the dependence on an RGB-D camera. The proposed approach is evaluated on a real-world robotic manipulation task performed using the Somatic Jinx robot, in which the robot must close a toilet-seat lid. The proposed policy achieves a 95% success rate with sensor depth and 90% with pseudo-depth, compared with 50% for the RGB-only baseline and 75% for 3D Diffusion Policy. These results show that explicit depth conditioning substantially improves performance on this distance-sensitive manipulation task and that pseudo-depth can serve as an effective alternative to hardware depth in this setting. Additional evaluation is conducted on the LIBERO simulation benchmark. Pseudo-depth augmentation yields a modest but consistent improvement across three task suites. The results indicate that the benefit extends beyond the single real-world task but remains most pronounced in distance-sensitive manipulation.
Lesia Hentosh, Yaroslav Hladun, I. Tsmots et al.· Scientific Reports· 0 citations
Imitation Learning aims to learn skills from extensive observations and demonstrations for robots, so it suffers from data scarcity and environment generalization. The existing methods predominantly focus on imitation from in-domain tasks and consequently struggle with generalization to unseen tasks. To bridge this generalization gap, we propose the \textbf{D}ynamics-\textbf{A}ware \textbf{M}eta-\textbf{I}mitation (DAMI) framework. By integrating meta-learning to construct a shared skill space, DAMI equips agents for rapid adaptation to novel tasks. We introduce the Visual-Motor Trajectory (VMT) module to capture complex spatio-temporal dynamics within the task latent space. Furthermore, we propose the Unpaired Unified Task (U2T) block to fuse unstructured multimodal observations. To coordinate these representations, we integrate a Task-Conditioned Feature Modulation (TCFM) mechanism customized for modulating low-level 3D features. By capturing intrinsic dynamics from a random complete reference demonstration, our framework learns the underlying task logic rather than memorizing static cues, ensuring effective generalization. Extensive experiments in both simulation and real-world settings demonstrate that our approach outperforms state-of-the-art baselines regarding direct inference on seen tasks and adaptation to unseen tasks via few-shot fine-tuning.
Zhenduo Shang, Xiyao Liu, Bohan Li et al.· 0 citations
Learning long-horizon manipulation skills with reinforcement learning remains challenging due to the complexity of reward design, the limited guidance of sparse rewards, and the high cost of manual subtask annotation. Visual demonstrations can provide supervision for reward learning, but rewards learned from raw pixels can be brittle and sensitive to visual variation, background appearance, and robot motion. In this work, we propose GORDON, a graph-based object-centric reward learning framework that learns dense rewards from action-free video demonstrations. Each visual scene is represented as a graph of detected objects and spatial relations, and a graph neural network is trained in a self-supervised manner to embed these graphs into a task-aligned latent space. To align the representation with semantic task progress, we introduce an activity-aware weighted pooling mechanism that emphasizes task-relevant objects while masking robot-dominated motion. The dense reward is then computed as distances in the learned latent space of the current state to demonstrated goal configurations, providing a measure of task progress. In long-horizon tasks, the temporal profile of this reward reveals stage-wise object-state transitions, enabling automatic subtask discovery without manual segmentation. The discovered segments are then used to train subtask-specific rewards and specialized policies that are composed sequentially. Experiments on seven manipulation tasks on MAGICAL and ManiSkill3 benchmarks show that our object-centric reward improves reinforcement learning in short-horizon settings and enables successful policy learning in complex long-horizon tasks through automatic decomposition, achieving an average success rate of 74.4% across the long-horizon tasks (on average approximately +35 p.p. vs. best learned baseline and approximately +25 p.p. vs. oracle).
Andrea Protopapa, Davide Buoso, Francesca Pistilli et al.· 0 citations
Flow2Act is presented, a unified framework that integrates agglomerative perception with a deterministic one-step generative policy and devise a curriculum region-aware mechanism via a Spatial-Grounded State Space Duality architecture, demonstrating significant gains in policy performance, robustness to environmental perturbations, and cross-task real-world applicability.
Sen Wang, Le Wang, Hongcheng Huo et al.· IEEE Transactions on Pattern...· 0 citations
Artificial Foveated Perception is proposed, a lightweight, policy-agnostic module that takes the same vision and language inputs as Vision-Language-Action and World Action Model pipelines and predicts task-conditioned masks over relevant objects, the robot, and other action-critical regions and reduces fine-tuning time, suppresses overfitting, and improves generalization under environmental perturbations.