Towards Intelligent Virtual Agents for Visual Monitoring of Assembly Tasks

Abstract

Manual assembly is essential in domains of high tactile complexity and product variability such as aircraft manufacturing, automotive assembly lines, or at-home furniture construction. Here, agentic artificial intelligence (AI) is deployed to visually monitor the assembly process and offer varying levels of support. At the same time, assembly tasks may benefit from the application of Mixed Reality (MR) which seamlessly integrates virtual elements in the physical space. Combining AI and MR allows a fundamental transformation of visual monitoring systems in assembly tasks and opens a vast, underexplored design space for intelligent virtual agents (IVAs). In this paper, we begin to systematically navigate and structure this space. To this end, we developed a MR prototype for the Apple Vision Pro featuring two contrasting virtual agent representations, one abstract and one humanoid, which provide assistance in two assembly tasks. Semi-structured expert interviews reveal two opposing preferences: while some experts favored the abstract agent’s ability to transparently signal its machine-based logic, others perceived the humanoid agent as more natural. These preliminary findings suggest that a one-size-fits-all approach to agent design – which is often pursued – may be insufficient. Moving forward, we aim to investigate adaptable agent representations that can be dynamically tailored to user preferences and special task requirements.