Action Recognition in Human-robot Teaming for Assembly Tasks

Abstract

This paper presents a study on Human-Robot Teaming (HRT), focusing on state-of-the-art methodologies and real-world implementation of a collaborative system for industrial assembly tasks. The study explores the integration of real-time human action recognition using deep learning, specifically LSTM (Long Short-Term Memory) networks fed by upper-body skeletal data (limbs and torso) extracted via MediaPipe, in combination with object detection using the YOLO (You Only Look Once) method. The system allows a UR5e collaborative robot to dynamically adjust its assistance in response to the operator’s actions, specifically anticipating the tool needed based on the context rather than a fixed schedule. Unlike traditional automation, which relies on rigid, pre-programmed sequences, the proposed approach is sequence-independent. This flexibility allows the robot to support variable assembly workflows, with the potential to mitigate operator cognitive workload. The system enables the robot to autonomously select and offer the required tool to the operator at the right moment, further enhancing collaboration.