Knowledge-Supervised LLM Decision Making for Intelligent Driving Systems in Long-Tail Scenarios

Abstract

While intelligent driving systems have achieved notable advances in perception, prediction, and motion planning, robust high-level decision-making remains a persistent challenge in long-tail scenarios that demand semantic scene understanding, contextual reasoning, and experience-based judgment. Rulebased policies tend to become brittle outside their manually specified operating envelopes, whereas purely data-driven policies may exhibit excessive sensitivity to rare maneuvers that are underrepresented in training logs. This paper presents a knowledge-supervised Large Language Model (LLM) decision module for intelligent driving systems. The module first converts publicly available driving baseline scenarios into structured linguistic representations that encode map topology, traffic rules, navigation intent, and surrounding environment information. An LLM-based driver then decomposes the decision-making process into scenario-type recognition, lateral lane selection, and longitudinal speed control, and passes discrete decisions to downstream planning and control. To enhance reliability, an offline supervisor distills reusable contextual experience knowledge from publicly available driving logs and closed-loop exploratory tests, organizing the extracted insights into an evolving knowledge graph. During inference, an online supervisor retrieves relevant knowledge to enhance the reasoning process and provides decision evaluation through feedback-driven reconsideration. Experimental evaluations on Highway-env, focusing on lane-level interaction, passage efficiency, and supervisor ablations, show that the proposed module improves success rate and weighted passage score over a reinforcement-learning baseline, a memorybased LLM driver, and a driver-only variant. Further ablations indicate that compact situation-lesson knowledge, graph-aware retrieval, and supervisor feedback contribute to more stable LLM-based decision reasoning. The module is designed as a plugin high-level decision interface for a modular intelligent driving stack rather than as an end-to-end trajectory generator.