Priority-Driven Hierarchical Multi-Agent Systems with Fine-Tuned LLMs

Abstract

Ambient Assisted Living (AAL) environments aim to enable older people to remain active and lead an autonomous and independent life for as long as possible. Among the technologies that can be incorporated into these settings, socially assistive robots (SAR) seek to establish a more natural and intuitive means of interaction with people, whilst helping them to carry out everyday tasks. One of the main challenges facing the design of these robots is how to enable them to undertake more complex tasks. Recent advances in Large Language Models (LLMs) have opened new avenues for flexible robot deliberation, yet their integration into real-time robotic systems remains challenging due to latency constraints, reasoning reliability, and the complexity of coordinating multi-step tasks. This paper proposes a hierarchical multi-agent architecture for robot deliberation that addresses these challenges by combining LLM-based planning with structured execution mechanisms within the ROS 2 ecosystem. The proposed architecture employs a supervisor agent that decomposes high-level natural language instructions into prioritised subtasks, enabling a priority-driven execution model that dynamically adapts to task relevance, temporal constraints, and environmental feedback. Subtasks are delegated to a set of Single-Purpose Agents (SPAs), orchestrated via LangGraph state machines and coordinated through a priority-aware scheduling mechanism. A key design principle is the use of Behaviour Trees (BTs) as high-level callable tools through the Model Context Protocol (MCP), encapsulating closed-loop control strategies while enabling preemptive and priority-consistent execution. This reduces the number of LLM inference steps required per task and improves robustness under dynamic conditions. A further contribution concerns the deployment of fine-tuned, lightweight LLMs—on the order of 0.6 billion parameters—specifically adapted for both the supervisor and the individual SPA roles through parameter-efficient low-rank adaptation (LoRA). These models are trained on role-specific tool-calling datasets to specialise in constrained reasoning patterns and task-specific decision-making, enabling efficient, low-latency inference directly on edge hardware. The combination of fine-tuning and hierarchical priority control enhances both the determinism and responsiveness of the system while mitigating error propagation across agent interactions. The paper presents the full software architecture, a formal characterisation of the system as a priority-aware hierarchical policy over a graph of agent workflows, and an experimental evaluation in an Ambient Assisted Living scenario assessing task success rate, inference efficiency, responsiveness under competing priorities, and overall user experience. Because SPA execution is decoupled from the supervisor’s own reasoning loop, the architecture is designed to keep accepting, processing, and queuing new user queries while previously dispatched SPAs are still executing their tasks.