References & Citations
Computer Science > Computer Vision and Pattern Recognition
Title: Continual 3D Convolutional Neural Networks for Real-time Processing of Videos
(Submitted on 31 May 2021 (this version), latest version 5 Jul 2022 (v3))
Abstract: This paper introduces Continual 3D Convolutional Neural Networks (Co3D CNNs), a new computational formulation of spatio-temporal 3D CNNs, in which videos are processed frame-by-frame rather than by clip. In online processing tasks demanding frame-wise predictions, Co3D CNNs dispense with the computational redundancies of regular 3D CNNs, namely the repeated convolutions over frames, which appear in multiple clips. While yielding an order of magnitude in computational savings, Co3D CNNs have memory requirements comparable with that of corresponding regular 3D CNNs and are less affected by changes in the size of the temporal receptive field. We show that Continual 3D CNNs initialised on the weights from preexisting state-of-the-art video recognition models reduce the floating point operations for frame-wise computations by 10.0-12.4x while improving accuracy on Kinetics-400 by 2.3-3.8. Moreover, we investigate the transient start-up response of Co3D CNNs and perform an extensive benchmark of online processing speed as well as accuracy for publicly available state-of-the-art 3D CNNs on modern hardware.
Submission history
From: Lukas Hedegaard [view email][v1] Mon, 31 May 2021 18:30:52 GMT (938kb,D)
[v2] Tue, 1 Feb 2022 08:51:21 GMT (1052kb,D)
[v3] Tue, 5 Jul 2022 07:42:04 GMT (1174kb,D)
Link back to: arXiv, form interface, contact.