Collaborative Knowledge Distillation and Reinforcement Learning for Automated Ticket Triage in Large-Scale Production Systems

Abstract

In large-scale enterprise environments, growing system complexity makes failures inevitable, threatening business continuity and customer satisfaction. To maintain system stability, efficient ticket triage is crucial for timely incident resolution. However, it remains a knowledge-intensive task requiring substantial domain expertise and detailed analysis, revealing the limitations of traditional rule-based and text-based methods. Recent advances in large language models (LLMs) offer new possibilities for automating triage through their remarkable reasoning and language capabilities. Yet, in industrial settings, LLMs often struggle to leverage domain-knowledge essential for accurate triage. We present CoTriage, a practical and scalable end-to-end automated ticket triage system, designed and deployed for real-world industrial scenarios. CoTriage leverages novel collaboration between large and small models: a small set of labeled tickets is used to distill high-quality LLM reasoning into a lightweight model, which is further optimized through a self-reinforcement mechanism. The refined triager then acts as a reward model to fine-tune a ticket summarizer using reinforcement learning. Comprehensive experiments conducted in the production environment of ByteDance, a leading global online video service provider, demonstrate the effectiveness of CoTriage, reducing the average triage time to 15.0 seconds and significantly enhancing operational efficiency and accelerating incident resolution in practice.