LLM‐Based Multimodal Robotic Endoscope Control Framework for Enhancing Human–Robot Interaction in Minimally Invasive Surgery

Abstract

Minimally invasive surgery (MIS) has gained widespread adoption over open surgery due to its numerous benefits for patient outcomes. In laparoscopic surgical procedures, maintaining a stable and optimal camera view is crucial, making human endoscope holders indispensable members of the MIS team, as they ensure consistent visualization during operations. However, their performance can vary due to factors such as experience level and fatigue. As a result, robotic systems have emerged as promising alternatives, offering precision and stability. Nevertheless, existing solutions often suffer from nonintuitive interfaces or focus solely on isolated sub‐tasks within the control workflow, limiting their overall effectiveness. To address these challenges and advance the development of robotic endoscopes, we present a novel robotic endoscope control framework powered by a large language model (LLM). Our system integrates both manual and autonomous functionalities, employing a real‐time deep learning‐based instrument tip detection model for accurate visual tracking and an automatic speech recognition module for natural language command interpretation. The LLM‐based central logic unit processes multimodal inputs through tool use capability to trigger and control the endoscope's movements. Experimental results validate the system's practicality, achieving an 89.47% success rate with a response latency of ≈1 s. Related codes and LLM prompts will be made available in the GitHub repository.