An Agentic Supervisory Layer for Online Parameter Regulation in Active PTZ Tracking

Abstract

Active pan-tilt-zoom (PTZ) tracking of small aerial targets is governed by operational parameters—control gains, the admissible zoom band, loss-detection and recovery threshold, and estimator noise—that are conventionally tuned offline against one condition and then frozen. Because counter-UAV conditions are strongly non-stationary, a single static configuration is detuned almost everywhere it is used. We instead regulate these parameters online, casting the problem as a contextual meta-decision posed above the controller: a supervisory agent reads a compact summary of recent tracking telemetry and periodically reconfigures an otherwise unchanged low-level controller, without entering the per-frame control path. Acting on a slow cadence, the supervisor is controller-agnostic and can afford a deliberation budget far larger than any per-frame policy. We ask whether effective regulation must be learned online from reward or can rely on a pretrained model's semantic prior, and instantiate the supervisor at both ends of that axis: A-Lite, a reward-driven contextual bandit, and A-LLM, a training-free large language supervisor that regulates zero-shot and can be steered by a natural-language mission. On a programmable PTZ simulator, A-Lite matches the strongest offline-tuned configuration on tracking success at no exploration cost, while A-LLM trades a marginal (1.2-2.5 pp) success-rate change for a substantial recovery gain-raising loss-recovery rate by 3.6-5.9 pp and shortening mean time-to-recovery by 20-26%. To our knowledge, this is the first formulation of active-tracking parameter regulation as an agentic decision and the first use of an LLM as a zero-shot supervisor for an active visual tracking loop.