We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CV

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computer Vision and Pattern Recognition

Title: TAda! Temporally-Adaptive Convolutions for Video Understanding

Abstract: Spatial convolutions are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to previous temporal modelling operations, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, the kernel calibration brings an increased model capacity. We construct TAda2D networks by replacing the 2D convolutions in ResNet with TAdaConv, which leads to at least on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin.
Comments: Accepted to ICLR 2022. Project page: this https URL
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2110.06178 [cs.CV]
  (or arXiv:2110.06178v3 [cs.CV] for this version)

Submission history

From: Ziyuan Huang [view email]
[v1] Tue, 12 Oct 2021 17:25:07 GMT (2522kb,D)
[v2] Wed, 24 Nov 2021 04:27:28 GMT (2566kb,D)
[v3] Wed, 26 Jan 2022 15:49:35 GMT (1279kb,D)
[v4] Thu, 17 Mar 2022 15:14:37 GMT (1281kb,D)

Link back to: arXiv, form interface, contact.