Efficient High-Frequency Event-Based Optical Flow With Temporal Iterative Refinement and Spiking Neural Networks

Abstract

Existing event-based optical flow approaches often build on frame-based counterparts, failing to deliver high-frequency flow estimation. Methods that specifically address this issue fail to achieve comparable performance or the desired computational efficiency. In this work, we introduce a novel temporal iterative refinement (TIR) framework to obtain low-latency flow updates at high frequency. The TIR module incorporates the previous flow estimate along with the updated feature maps to simultaneously update and refine the flow estimate at each time step, thereby predicting accurate nonlinear pixel trajectories. However, updating the feature space at high frequency with conventional CNNs may lead to the temporal aperture problem, as the small temporal receptive field may not be enough to capture the necessary spatial context. We introduce SNN-based feature encoders to efficiently address this problem. The temporal dynamics of the SNNs provide an increased temporal receptive field, while their deployment on neuromorphic hardware offers a promising path toward additional energy efficiency. The results obtained on the real-world MVSEC dataset show that our network achieves $17 \times $ and $33 \times $ lower computations than the state-of-the-art E-RAFT and TMA, respectively, while maintaining similar accuracy performance. Compared to other supervised learning-based approaches, our network exhibits better cross-domain generalization, hinting toward the strong inductive biases of the network. To demonstrate the remarkable potential of our approach, we also provide results in extremely challenging scenarios with highly nonlinear pixel trajectories from the MultiFlow dataset, which also features high-frequency ground truth. Our code will be available at https://github.com/AhmedHumais/STIRFlow