Title: EBBINNOT: A Hardware Efficient Hybrid Event-Frame Tracker for Stationary Neuromorphic Vision Sensors

Abstract: In this paper, we present a hybrid event-frame approach for detecting and tracking objects recorded by a stationary neuromorphic vision sensor (NVS) used in the application of traffic monitoring with a hardware efficient processing pipeline that optimizes memory and computational needs. The usage of NVS gives the advantage of rejecting background while it has a unique disadvantage of fragmented objects due to lack of events generated by smooth areas such as glass windows. To exploit the background removal, we propose an event based binary image (EBBI) creation that signals presence or absence of events in a frame duration. This reduces memory requirement and enables usage of simple algorithms like median filtering and connected component labeling (CCL) for denoise and region proposal (RP) respectively. To overcome the fragmentation issue, a YOLO inspired neural network based detector and classifier (NNDC) to merge fragmented region proposals has been proposed. Finally, a simplified version of Kalman filter, termed overlap based tracker (OT), exploiting overlap between detections and tracks is proposed with heuristics to overcome occlusion.
The proposed pipeline is evaluated using more than 5 hours of traffic recordings. Our proposed hybrid architecture outperformed (AUC = $0.45$) Deep learning (DL) based tracker SiamMask (AUC = $0.33$) operating on simultaneously recorded RGB frames while requiring $2200\times$ less computations. Compared to pure event based mean shift (AUC = $0.31$), our approach requires $68\times$ more computations but provides much better performance. Finally, we also evaluated our performance on two different NVS: DAVIS and CeleX and demonstrated similar gains. To the best of our knowledge, this is the first report where an NVS based solution is directly compared to other simultaneously recorded frame based method and shows tremendous promise.