Learning Adaptive Search with Reinforcement Learning for Small and Fast Object Tracking

Abstract

The persistent challenge in visual object tracking, particularly for small and fast-moving targets, lies in the trade-off between effective resolution and contextual information. Fixed search regions cannot adapt to variations in target scale and motion, often resulting in degraded target representation and tracking failures. In this paper, we introduce AdaSAM2, a framework that formulates adaptive search-region selection as a sequential decision-making problem. Unlike conventional per-frame heuristics, our approach employs an event-driven reinforcement learning policy that selects the cropping scale only during initialization and unreliable tracking states, while reusing the previous configuration during stable tracking. To handle target disappearance, we further introduce a lost-aware recovery mechanism that combines progressive search-region enlargement with constrained policy re-selection. Extensive experiments on TSFMO, LaTOT, UAV123, and UAVDT demonstrate consistent improvements over the SAMITE baseline. For example, AdaSAM2 improves Success and Precision on TSFMO from 42.6% and 74.0% to 43.9% and 75.3%, respectively, while improving UAV123 Success and Precision from 69.6% and 92.7% to 71.1% and 94.8%. Moreover, the RL policy is activated on only 0.74% of processed frames on TSFMO, resulting in an average overhead of only 0.0085 ms per frame. These results demonstrate that adaptive input-space optimization can improve tracking accuracy while introducing negligible computational overhead.