Adaptive Ticket Routing through Retrieval-Augmented In-Context Learning from Analyst Behaviour at CERT-EU

Abstract

Ticket routing is a critical operational task: organisations must triage incoming requests to specialised teams, typically via queues, but routing policies are organisation-specific and evolve continuously. Rule-based systems demand constant maintenance, while supervised classifiers suffer from concept drift and require periodic retraining. These challenges intensify in regulated environments where sensitive data prevents cloud-based AI services, annotation capacity is limited, and misrouting carries significant cost. We present an adaptive ticket routing system that frames routing as retrieval-augmented in-context learning with temporal feedback learning. Rather than training a static model, the system retrieves similar successfully routed tickets as demonstrations, grounding predictions in accumulated organisational expertise. It learns continuously from routine analyst behaviour: corrections are captured as error signals, while predictions unchanged beyond a calibrated time threshold are interpreted as implicit confirmation---transforming everyday workflow into supervision at zero incremental annotation cost. Predictions are constrained to queues present in retrieved demonstrations; an on-premises LLM resolves conflicts only when demonstrations disagree. Deployed at CERT-EU (cybersecurity service provider for European Union entities), the system operates in two modes: assisted routing, where analysts receive recommendations based on similar historical cases (2× speedup); and selective autonomy, where tickets assigned to high-confidence queues are routed automatically (22× speedup). Over six months of production deployment (1900 tickets), it achieves 86.3% weighted F1, outperforming supervised baselines by 4.5 points and zero-shot LLM inference by 15 points. The system runs entirely on-premises and adapts continuously without retraining or dedicated annotation, offering a practical blueprint for organisations operating under strict data-sovereignty constraints.