3D CNN and Study of Attention Mechanisms to Relative Camera Position
TL;DR
A spatio-temporal regression architecture combining 3D CNNs and attention mechanisms that regresses these fused representations into a relative position estimate and demonstrates the effectiveness of the method on the TII Drone Racing and UZH-FPV datasets.
Abstract
Calculating the relative position between images in Unmanned Aerial Vehicles (UAVs) is a core component in tasks such as visual odometry, SLAM, and state estimation. It enables the UAV to estimate its movement between frames using onboard sensors. In this paper, we present a spatio-temporal regression architecture combining 3D CNNs and attention mechanisms. The model takes a sequence of image frames and corresponding Inertial Measurement Unit (IMU) data for each frame as input and outputs the estimated relative position in meters between the images. The inclusion of IMU measurements is critical for improving robustness to motion blur, low-texture environments, and rapid maneuvers, as it provides complementary information about the UAV’s linear acceleration and angular velocity. The CNN layers extract compact spatio-temporal features from the video input, while the multihead attention layer captures temporal dependencies and contextual relations across time. The final Multi-Layer Perceptron (MLP) regresses these fused representations into a relative position estimate. We systematically evaluate different attention mechanisms: Masked, Causal, Local-window, and Linformer, to analyze their impact on accuracy and efficiency in relative position estimation. We demonstrate the effectiveness of our method on the TII Drone Racing and UZH-FPV datasets.