Talk-to-Fly: An Agentic LLM Runtime for Natural-Language UAV Task Control

Abstract

The current integration of large language models (LLMs) with unmanned aerial vehicles (UAVs) offers a promising approach to more accessible UAV task control. However, existing methods utilise LLMs as one-shot planners, which are vulnerable to changes in vehicle state and dynamic operating conditions. This paper introduces the Talk-to-Fly system, an agentic runtime for natural-language guided UAV task control. The runtime combines four core technical components: context-aware prompt assembly; a domain-specific language for flight plan generation; a flight plan review and clarification mechanism for resolving ambiguity; and closed-loop execution with postcondition monitoring and replanning. This allows the LLM to operate in a sense-think-act control loop. Talk-to-Fly is evaluated across 375 simulated tasks using the ArduPilot software-in-the-loop simulator, and physical deployment with outdoor flights on a real UAV. Results show a 100% task success-rate and median time-to-first-movement of 1.90 s, compared with the existing one-shot LLM planner which reduces overall success rate to 70%, showing the robustness of our Talk-to-Fly LLM system on natural-language-based UAV task planning and control.