MEKARD: Multimodal Embodied Knowledge-Augmented Robot Decision-Making System

Abstract

This paper presents MEKARD, a robot system that provides context-aware, proactive assistance in environments shared with humans by querying an external database of past experiences. The database stores multimodal records, including text, images, numerical state values, and embodiment information describing the physical states and actions of both humans and robots; this embodiment information is treated as a core component. Because human states, robot states, and environmental context can combine to create diverse situations that are difficult to handle using hand-crafted rules alone, we design a retrieval-augmented generation (RAG) framework that retrieves relevant past situations to support decision making. The system integrates the database and similarity-based retrieval with a multi-stage pipeline based on vision-language models and large language models. Experimental results show that the proposed method produces the largest number of correct outputs among the evaluated methods. A user study further suggests that embodiment-aware actions are generally rated more positively than the alternatives in the tested scenarios.