Unrolling the shutter in 4D: exploiting sub-scan dynamics with event cameras for efficient dynamic measurements

Abstract

We report an event-camera-based optical 4D (3D+time) sensor and compare it with a classical frame-based counterpart while measuring dynamic scenes. Frame-based active stereo 4D sensors project a series of patterns per single 3D scan into a scene for parallel temporal coding of object surface points. This is comparable to the parallel measurement of a global shutter camera in 3D. At the same time, each of the cameras measures the scene's responses. Phase retrieval, temporal correlation, or other methods identify pixels in both cameras which have likely seen the same object point. This approach forbids significant object motion during a single scan for spatio-temporal consistency. Consequently, high projection rates, high frame rates, and high illumination power are necessary for measuring fast motion. We developed an event-based stereo 4D sensor to break this limitation. Event cameras don’t measure complete frames, but instead, each pixel monitors its incoming brightness independently. When a positive or negative change crosses a threshold, an event is reported, i.e. a small data package with the pixel’s coordinates, a high-resolution timestamp, and photometric information, most commonly the polarity of the change. Coupled with a specialized projector, our event cameras sequentially measure sparse slices of the scene, which can be thought of as a rolling shutter. Temporally corresponding events are used to triangulate and reconstruct 3D points with timestamps from these events. We show that our event-based approach coupled with motion compensation can scan slower than a frame-based approach to recover the same moving object, reducing redundancy in measurements.