Aug 2026· Frontiers in Neurorobotics· Vol 20· 0 citations· 34 references
TL;DR
A VLM-guided retrieval-augmented generation (RAG) framework that crops task-relevant image regions, retrieves examples from a structured local database, re-ranks candidates, and expands the search when needed enables accurate, data-efficient adaptation to new device types without fine-tuning, at the cost of increased and variable latency.
Abstract
Reliable action prediction is essential for robotic electronic-waste disassembly, where device diversity and damage make preplanned sequences impractical. Vision-language models (VLMs) offer broad visual and semantic knowledge, but zero-shot predictions can be unreliable, while fine-tuning requires substantial task-specific data. We propose a VLM-guided retrieval-augmented generation (RAG) framework that crops task-relevant image regions, retrieves examples from a structured local database, re-ranks candidates, and expands the search when needed. We evaluated the framework on six types of smoke detectors and heat-cost allocators and compared the performance of three VLMs using zero-shot inference, single-example RAG, VLM-guided RAG, and task-specific fine-tuning. Cropping improved retrieval across all evaluated models, while VLM-guided re-ranking increased top-1 retrieval accuracy from 68.47% to 84.57-89.29%. Next action prediction accuracy reached 90.82-95.92% and significantly outperformed single-example RAG. RAG-based methods also outperformed the fine-tuned models. Median prediction latency was 8.95 s, although expanded searches were slower. Overall, VLM-guided RAG enables accurate, data-efficient adaptation to new device types without fine-tuning, at the cost of increased and variable latency.
While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performance can degrade when adapted to new tasks with limited task-specific demonstrations. Retrieval offers a practical way to reuse existing demonstrations for data-efficient adaptation, but existing methods often rely on visual similarity, state-action representations, or task-level language matching. These approaches may overlook the hierarchical structure of long-horizon manipulation tasks, where complete task matches are rare but reusable skills are often abundant. To address this challenge, we propose Hierarchical Skill Retrieval (HSR), a retrieval framework for data-efficient VLA adaptation. Specifically, HSR first decomposes a target task into candidate skill sequences. It evaluates each plan based on both semantic plausibility and skill reliability estimated from the prior dataset. The selected decomposition is then used for hybrid retrieval. This combines subtask-level language retrieval with behavior-feature reranking to identify demonstrations that are both semantically relevant and compatible with the target task. Finally, we adapt the policy through a two-stage pretraining and finetuning pipeline, which separates general skill acquisition from task-specific adaptation. Experiments on the LIBERO benchmark and several real-world robot manipulation tasks show that HSR improves the average success rate by 10.3% and 21.3% over the strongest baseline, respectively. These results demonstrate the effectiveness of structured skill-level retrieval for data-efficient VLA adaptation. Videos and code are available at https://hoar012.github.io/HSR-Project.
Haoran Hao, Shahram Najam Syed, Jeff Schneider et al.· 0 citations
Multimodal Large Language Models (MLLMs) based Embodied navigation faces a severe challenge where key cues are easily overwhelmed by complex environmental noise, leading to inefficient decision-making. To address this, we propose a Semantic Relevance Guided grounding enhanced navigation framework(SRG-Nav). The core idea of our approach lies in utilizing semantic relevance to guide visual and language attention. By evaluating the correlation between scene entities and the navigation goal, SRG-Nav adds ranked high-value cues to system prompts and maps them back into the visual space to generate explicit bounding boxes. This mechanism explicitly directs the MLLM to focus on task-relevant entities and regions while effectively suppressing environmental noise. Experiments on the AI2Thor platform demonstrate that SRG-Nav outperforms baseline methods in both success rate and path efficiency, validating that structured semantic-visual prompts significantly improve the robustness of embodied navigation.
Shuai Chen, Hao Chen, Beiyu Guo et al.· International Conference on...· 0 citations
Service robots often receive natural language instructions in changing workspaces where multiple visible objects may match one description. Relying on detector confidence, random selection, or direct vision–language model (VLM) prediction can lead to a wrong action. This paper presents ActivAsk, a zero-shot framework for resolving referential ambiguity before robotic grasping. ActivAsk constructs open-vocabulary candidates from red-green-blue-depth (RGB-D) input, asks candidate-grounded yes/no questions when needed, updates the candidate state from the user’s answer, and grasps after target resolution. It selects among VLM-proposed candidate partitions using an expected free energy (EFE) criterion motivated by active inference; with neutral response preferences, this reduces to information gain over candidate partitions. Offline experiments showed that interactive clarification improved target accuracy from about 53–54% for noninteractive baselines to about 90–92%. ActivAsk matched the best interactive accuracy (92.13%) while asking 15.47–19.71% fewer questions on asked trials and 21.43–23.88% fewer for ambiguous instructions. In online real robot experiments, ActivAsk achieved 92.98% target selection accuracy and 87.72% full correct object grasp success; unresolved or wrong targets were not physically executed after operator-controlled verification and were counted as task failures.
Haoandong Yang, Gabriel W. Haddon-Hill, Teresa Zielińska et al.· Robotics· 0 citations
Retrieval-Augmented Generation (RAG) has emerged as the dominant strategy to ground LLM outputs in retrieved evidence to reduce hallucinations and increase factuality, yet RAG itself introduces new challenges: noisy retrieval, knowledge conflicts, and wasted computation from unnecessary retrieval. This tutorial provides an in-depth treatment of modern RAG based on AI-facilitated systematic analysis of ~2000 recent papers (2020--2026). We trace the RAG pipeline from its modular foundations through graph-enhanced reasoning to the latest RL-driven agentic architectures, covering each stage---triggering, query rewriting, retrieval, post-processing, generation, and knowledge-graph integration---with running examples, method comparisons, and benchmark results. Attendees will leave with a structured roadmap, evidence-backed practical recommendations, and a clear map of open research opportunities.
X. Dong, Sanat Sharma, Kai Sun et al.· Proceedings of the 32nd ACM...· 0 citations
Assembly action understanding is a key enabler for effective human-robot collaborative assembly, yet it remains challenging due to subtle motions and fine-grained hand-object interactions. We adapt vision-language models (VLMs) to this challenging domain with Compositional Context Fine-Tuning (CCFT), a method that decomposes assembly actions into semantic elements (Verb, Object, Tool) and fine-tunes VLMs to recognize each action element using templated question-answering pairs. This approach ensures near-deterministic outputs. To enable efficient and effective multi-task learning under limited data, a Layer-Partitioned Alternating Training (LP-AT) method is presented, which assigns distinct model layers to recognize specific action elements through element-specific low-rank adapters. LP-AT alternates weight updates across element-specific adapters, reducing cross-task interference while enabling per-adapter hyperparameter optimization. Furthermore, we create HA-ViD-VQA and IKEA-ASM-VQA datasets from existing assembly video datasets. Extensive experiments on these datasets demonstrate that our method consistently outperforms strong action recognition baselines while providing interpretable element-level predictions that can support diverse downstream applications.
Hao Zheng, Jinyi Huang, Tiantian Zheng et al.· 0 citations
This work proposes ARIES-Mission2, a zero-shot Vision-Language-Action (VLA) framework that decouples visual-semantic perception from physical route optimization in low-altitude Unmanned Aerial Vehicle (UAV) mission generation and indicates that the TSP module maintains lower growth in computation time as the number of targets increases.
Junhao Wei, Yanxiao Li, Haochen Li et al.· 0 citations